This post from Ian Ferguson, Associate Engineering Manager at The Movement Cooperative (TMC), was originally published on Medium and is reposted here with permission. We’ve lightly edited the language for clarity and consistency with our blog.

Across the movement, teams are building and managing their own systems alongside shared tools. At TMC, a core part of our work is supporting organizations as they connect their data and systems—so teams spend less time dealing with fragmented tools and more time focusing on their programs. The Terraform-based approach outlined below shows how teams can create flexible, scalable foundations that support that work over time.

Imagine you are the sole infrastructure engineer on a team with multiple stakeholders. Each of these stakeholders may have (see: definitely have) similar but discrete requirements—separate storage buckets, service accounts, varying degrees of API and IAM access, etc.

Pointing and clicking in the UI is a recipe for failure in this scenario—we’re going to quickly lose track of which pieces of infrastructure have been provisioned and which ones haven’t.

Enter: Terraform.

About Terraform

HashiCorp Terraform is the premier Infrastructure-as-Code (IaC) tool—meaning it allows you to define and manage your cloud infrastructure in code instead of configuring it manually. It’s open-source, widely used, and well-documented.

You can think of Terraform as a wrapper for all the API calls you might otherwise write. Instead of clicking “Create Instance” ad infinitum in the UI or running aws or gcloud commands in the terminal, you can simply run terraform apply, review your changes, and let Terraform do its work.

Example Project

Let’s use Terraform to manage infrastructure for multiple fictitious clients. By the end of this article, we’ll provision the following:

A production BigQuery dataset
A scratch BigQuery dataset (with default lifecycle rules)
Optionally, a Cloud Storage bucket (if the client requests it)
A Google service account with access to the client’s resources (and ONLY their resources)

You can follow along with the code in this repository: https://github.com/IanRFerguson/modular-infrastructure

Designing a Terraform Module

Typically, a Terraform module will include the following:

main.tf — This is the main interface that will create our infrastructure. We’ll define all of our resources here, and they’ll be created based on the inputs in variables.tf.
variables.tf — Think of these like Python function arguments. We’ll supply them to our module, and the relevant infrastructure defined in main.tf will be created accordingly.
output.tf — If we wanted our module instance to be accessible to other components of our infrastructure, we would define those attributes here. This isn’t included in this example project, but HashiCorp’s docs on this topic are excellent.

Because these three files are bundled together in a subdirectory—a module—they are accessible to one another (i.e., main.tf can access the inputs defined in variables.tf, etc.)

Our example project is structured such that ./main.tf defines client modules, which are defined in our ./modules/client/main.tffile.

Ian @ infra $ tree .
.
├── main.tf
├── modules
│   └── client
│       ├── main.tf
│       └── variables.tf

3 directories, 5 files

// In ./main.tf
module "client_bar" {
  source           = "./modules/client"
  client_name      = "bar"
  provision_bucket = true
}

The variables used here—client_name and provision_bucket—are defined in ./modules/client/variables.tf like so:

variable "client_name" {
  type = string
}

variable "provision_bucket" {
  type    = bool
  default = false
}

These variables are fed into the main.tf file in the module, where they’ll be used to uniquely name and permission the resources we’ll need for our hypothetical clients:

// Create client-specific service account
resource "google_service_account" "service_account" {
  account_id   = "sa-$"
  display_name = "Client service account for $"
}
// BigQuery datasets
resource "google_bigquery_dataset" "prod" {
  dataset_id    = "$__prod"
  friendly_name = "$__prod"
  description   = "Production BigQuery dataset for $"
  location      = var.bigquery-region
}
resource "google_bigquery_dataset" "scratch" {
  dataset_id                  = "$__scratch"
  friendly_name               = "$__scratch"
  description                 = "Scratch BigQuery dataset for $"
  location                    = var.bigquery-region
  default_table_expiration_ms = var.default_table_expiration
}
// BigQuery IAM
resource "google_bigquery_dataset_iam_member" "prod-editor" {
  dataset_id = google_bigquery_dataset.prod.friendly_name
  role       = "roles/bigquery.dataEditor"
  member     = "serviceAccount:$"
}
resource "google_bigquery_dataset_iam_member" "scratch-editor" {
  dataset_id = google_bigquery_dataset.scratch.friendly_name
  role       = "roles/bigquery.dataEditor"
  member     = "serviceAccount:$"
}
// Cloud storage
resource "google_storage_bucket" "default" {
  count    = var.provision_bucket ? 1 : 0
  name     = "bkt-$"
  location = var.gcs-region
}
resource "google_storage_bucket_iam_member" "member" {
  count  = var.provision_bucket ? 1 : 0
  bucket = google_storage_bucket.default[count.index].name
  role   = "roles/storage.admin"
  member = "serviceAccount:$"
}

Leveraging Modules

As a refresher, in our hypothetical scenario we were intending to provision BigQuery and Cloud Storage resources for multiple clients in the same project.

We’ve defined two client modules in this project, like so:

module "client_foo" {
  source      = "./modules/client"
  client_name = "foo"
}

module "client_bar" {
  source           = "./modules/client"
  client_name      = "bar"
  provision_bucket = true
}

After running terraform apply, we’ll have:

A production dataset
A scratch dataset with lifecycle rules

Data editor access for the client-specific service account

A bucket provisioned for the bar client (but NOT the foo client)

Extending These Ideas

This is a fairly high-level overview of modular Terraform, but it hopefully illustrates what’s possible. In addition to BigQuery and Cloud Storage, you could use Terraform to:

Create and manage Postgres users in a CloudSQL instance
Manage a VPC with custom ingress rules
Set custom IAM roles for different groups

The goal of this article is to get the wheels turning as you start thinking about your next infrastructure project. Thanks so much for reading!

About the author

Ian Ferguson

Associate Engineering Manager, The Movement Cooperative

Ian has been with TMC since 2023 and currently serves as the Associate Engineering Manager on the Data Engineering team. When he’s not building pipelines for the Cooperative, Ian can be found roaming the streets of Brooklyn with a camera in hand or cheering for the Knicks.