DEV Community

Cover image for Terraform for Data Engineers: Provisioning GCS, BigQuery, S3, and Lambda Without Clicking Through Consoles
De' Clerke
De' Clerke

Posted on

Terraform for Data Engineers: Provisioning GCS, BigQuery, S3, and Lambda Without Clicking Through Consoles

#devops#dataengineering#terraform#aws

Every data pipeline eventually needs a bucket. Then a second bucket. Then a BigQuery dataset, a service account with the right permissions, and a Lambda function to handle alerts. If you set all of that up through the GCP and AWS consoles, you get something that works once, is impossible to reproduce exactly, and will be misconfigured in the next project because you forgot which checkboxes you ticked. Terraform solves this by treating infrastructure as code: version-controlled, reviewable, and repeatable.

This article covers the patterns a data engineer actually needs. Not VPCs and Kubernetes clusters. GCS buckets, BigQuery tables with partitioning, S3 data lakes with lifecycle rules, Lambda functions for lightweight processing, and the IAM wiring that makes service accounts work without over-permissioning.

All provider versions in this article are current as of June 2026: Terraform 1.15.5, Google provider 7.34.0, AWS provider 6.47.0.

The Mental Model: State, Plan, Apply

Terraform works by comparing three things: what you wrote in your .tf files, what it last recorded in the state file, and what actually exists in the cloud. The core workflow is three commands:

terraform init # download providers and modules
terraform plan # show what will change without touching anything
terraform apply # make the changes
Enter fullscreen modeExit fullscreen mode

terraform plan is the command you run the most. It shows exactly what will be created, modified, or destroyed before anything happens. A plan that shows a resource being replaced (-/+) when you expected it to be modified (~) is a signal to stop and read the plan carefully. Replacement destroys and recreates the resource, which means downtime for anything depending on it.

The state file (terraform.tfstate) is how Terraform knows what it manages. It contains the IDs, attributes, and dependencies of every resource it has created. Never edit it manually and never delete it. If the state file is lost, Terraform loses track of what it owns and will try to create everything from scratch.

File Structure

Split your Terraform config across five files. Every file has a specific responsibility:

project/
├── main.tf # resource definitions
├── variables.tf # input variable declarations
├── outputs.tf # output values
├── providers.tf # provider and version config
└── terraform.tfvars # actual variable values (gitignore this if it has secrets)
Enter fullscreen modeExit fullscreen mode

providers.tf is where you pin versions. This is not optional.

terraform{required_version=">= 1.9.0"required_providers{google={source="hashicorp/google"version="~> 7.0"}aws={source="hashicorp/aws"version="~> 6.0"}}}provider"google"{project=var.project_idregion=var.region}provider"aws"{region="us-east-1"}
Enter fullscreen modeExit fullscreen mode

The ~> 7.0 constraint allows patch and minor updates (7.1, 7.34) but blocks major version upgrades (8.0). Major version bumps in both providers have historically included breaking changes. Pinning to a major version means terraform init -upgrade will not silently change provider behavior.

variables.tf declares inputs with types and validation:

variable"project_id"{description="GCP project ID"type=string}variable"region"{description="Default region"type=stringdefault="africa-south1"# Johannesburg; BigQuery and GCS available}variable"environment"{type=stringvalidation{condition=contains(["dev","staging","prod"],var.environment)error_message="Must be dev, staging, or prod."}}variable"labels"{type=map(string)default={}}
Enter fullscreen modeExit fullscreen mode

terraform.tfvars provides the actual values. Add it to .gitignore if it contains credentials or account IDs you do not want public:

project_id="my-gcp-project-123"region="africa-south1"environment="dev"labels={project="kenya-data-pipeline"managed_by="terraform"}
Enter fullscreen modeExit fullscreen mode

Remote State: Stop Storing State Locally

By default, Terraform writes terraform.tfstate to your local working directory. This works for solo projects and breaks the moment anyone else touches the infrastructure. Remote state keeps the file in a shared location with locking so two people cannot run terraform apply simultaneously and corrupt the state.

For GCP projects, use a GCS bucket as the backend:

# backend.tfterraform{backend"gcs"{bucket="my-project-terraform-state"prefix="terraform/state/pipeline"}}
Enter fullscreen modeExit fullscreen mode

For AWS projects, use S3 with a DynamoDB table for locking:

terraform{backend"s3"{bucket="my-terraform-state-bucket"key="state/pipeline/terraform.tfstate"region="us-east-1"dynamodb_table="terraform-lock"encrypt=true}}
Enter fullscreen modeExit fullscreen mode

The DynamoDB table needs a LockID string partition key. Create it manually once before initializing:

aws dynamodb create-table \--table-name terraform-lock \--attribute-definitionsAttributeName=LockID,AttributeType=S \--key-schemaAttributeName=LockID,KeyType=HASH \--billing-mode PAY_PER_REQUEST
Enter fullscreen modeExit fullscreen mode

After adding a backend, run terraform init again. It will ask whether to migrate the existing local state to the remote backend.

GCS: The Data Lake Bucket


A data lake GCS bucket with versioning, lifecycle rules, and uniform access control:

resource"google_storage_bucket""data_lake"{name="${var.project_id}-data-lake-${var.environment}"location="US"storage_class="STANDARD"force_destroy=falseversioning{enabled=true}lifecycle_rule{condition{age=90}action{type="SetStorageClass"storage_class="NEARLINE"}}lifecycle_rule{condition{age=365}action{type="Delete"}}uniform_bucket_level_access=truelabels=var.labels}
Enter fullscreen modeExit fullscreen mode

force_destroy = false prevents Terraform from deleting the bucket if it contains objects. If terraform destroy encounters a non-empty bucket, it fails with an error instead of silently deleting your data. Leave this as false on anything that contains data you care about.

GCS bucket names are globally unique across all GCP accounts. Including the project ID in the name (${var.project_id}-data-lake) avoids the Error 409: The requested bucket name is not available error, which you will hit if you try to create a bucket with a generic name like data-lake.

BigQuery: Datasets and Partitioned Tables 

resource"google_bigquery_dataset""raw"{dataset_id="raw"friendly_name="Raw Layer"location="US"labels=var.labelsdelete_contents_on_destroy=falseaccess{role="OWNER"special_group="projectOwners"}}resource"google_bigquery_table""flights"{dataset_id=google_bigquery_dataset.raw.dataset_idtable_id="flights"project=var.project_iddeletion_protection=falsetime_partitioning{type="DAY"field="departure_time"}clustering=["airline","origin"]schema=file("${path.module}/schemas/flights.json")labels=var.labels}
Enter fullscreen modeExit fullscreen mode

Two things worth explaining here.

First, deletion_protection = false on the table resource. As of Google provider 6.0, many resources have deletion_protection defaulting to true, which prevents terraform destroy from deleting them. For BigQuery tables in a data pipeline project you plan to rebuild frequently, set it to false explicitly or terraform destroy will error out on the table.

Second, the combination of time_partitioning and clustering. Partitioning by day on departure_time means BigQuery scans only the relevant day partitions when you filter by date, reducing bytes processed and cost. Clustering by airline and origin within each partition further reduces scan size when you filter by those columns. For a table that receives daily appends and is queried by date and airline, this setup can reduce query cost by 80% or more compared to an unpartitioned table.

The schema file is a standard BigQuery JSON schema:

[{"name":"flight_id","type":"STRING","mode":"REQUIRED"},{"name":"airline","type":"STRING","mode":"NULLABLE"},{"name":"origin","type":"STRING","mode":"NULLABLE"},{"name":"departure_time","type":"TIMESTAMP","mode":"NULLABLE"}]
Enter fullscreen modeExit fullscreen mode

GCP IAM: Service Accounts for Pipelines

Never run a pipeline with personal credentials or a broad role like roles/editor. Create a service account with exactly the permissions needed.

resource"google_service_account""pipeline"{account_id="data-pipeline-sa"display_name="Data Pipeline Service Account"project=var.project_id}resource"google_project_iam_member""pipeline_bq_editor"{project=var.project_idrole="roles/bigquery.dataEditor"member="serviceAccount:${google_service_account.pipeline.email}"}resource"google_project_iam_member""pipeline_bq_job"{project=var.project_idrole="roles/bigquery.jobUser"member="serviceAccount:${google_service_account.pipeline.email}"}resource"google_storage_bucket_iam_member""pipeline_gcs"{bucket=google_storage_bucket.data_lake.namerole="roles/storage.objectAdmin"member="serviceAccount:${google_service_account.pipeline.email}"}
Enter fullscreen modeExit fullscreen mode

Note the two separate BigQuery roles. roles/bigquery.dataEditor lets the service account read and write table data. roles/bigquery.jobUser lets it run query jobs. You need both for a pipeline that reads from and writes to BigQuery. Without jobUser, queries fail with a 403 even though the service account has data access.

For local development, generate a key file and set the environment variable:

resource"google_service_account_key""pipeline_key"{service_account_id=google_service_account.pipeline.name}output"sa_key"{value=base64decode(google_service_account_key.pipeline_key.private_key)sensitive=true}
Enter fullscreen modeExit fullscreen mode
terraform output -raw sa_key > sa-key.json
export GOOGLE_APPLICATION_CREDENTIALS=$(pwd)/sa-key.json
Enter fullscreen modeExit fullscreen mode

Add sa-key.json to .gitignore immediately. For production and CI/CD, use Workload Identity instead of key files.

S3: Data Lake with Encryption and Lifecycle

S3 bucket resources in provider 6.x are split into separate resources for each concern, unlike the older monolithic aws_s3_bucket with nested blocks. Each setting is its own resource:

resource"aws_s3_bucket""data_lake"{bucket="${var.project_name}-data-lake-${var.environment}"tags=var.tags}resource"aws_s3_bucket_versioning""data_lake"{bucket=aws_s3_bucket.data_lake.idversioning_configuration{status="Enabled"}}resource"aws_s3_bucket_server_side_encryption_configuration""data_lake"{bucket=aws_s3_bucket.data_lake.idrule{apply_server_side_encryption_by_default{sse_algorithm="AES256"}}}resource"aws_s3_bucket_public_access_block""data_lake"{bucket=aws_s3_bucket.data_lake.idblock_public_acls=trueblock_public_policy=trueignore_public_acls=truerestrict_public_buckets=true}resource"aws_s3_bucket_lifecycle_configuration""data_lake"{bucket=aws_s3_bucket.data_lake.idrule{id="archive-and-expire"status="Enabled"transition{days=90storage_class="STANDARD_IA"}expiration{days=365}}}
Enter fullscreen modeExit fullscreen mode

If you have existing Terraform code using the old aws_s3_bucket_object resource, it was renamed to aws_s3_object in AWS provider 4.x. Use the moved block to update the state reference without destroying and recreating the object:

moved{from=aws_s3_bucket_object.schema_fileto=aws_s3_object.schema_file}
Enter fullscreen modeExit fullscreen mode

Lambda: Lightweight Processing and Alerts

Lambda is useful in data pipelines for things that do not belong inside the main DAG: webhook receivers, lightweight event-driven transforms, and alert dispatchers. Here is the full pattern for a scheduled Python Lambda:

data"archive_file""lambda_zip"{type="zip"source_dir="${path.module}/lambda"output_path="${path.module}/lambda.zip"}resource"aws_lambda_function""alert"{filename=data.archive_file.lambda_zip.output_pathfunction_name="pipeline-alert-${var.environment}"role=aws_iam_role.lambda.arnhandler="handler.lambda_handler"runtime="python3.12"source_code_hash=data.archive_file.lambda_zip.output_base64sha256timeout=30memory_size=256environment{variables={SNS_TOPIC_ARN=aws_sns_topic.alerts.arnENVIRONMENT=var.environment}}tags=var.tags}
Enter fullscreen modeExit fullscreen mode

source_code_hash is what tells Terraform the code changed. Without it, Terraform only updates the function when the .tf file changes, not when the Python code in /lambda changes. With output_base64sha256, a new zip hash triggers a redeployment automatically on terraform apply.

The supported Python runtimes as of June 2026 are python3.12, python3.13, and python3.14. python3.12 is a stable, widely tested choice for production. python3.9 reached Python EOL in October 2025 and Lambda deprecated it in early 2026. Do not use it for new functions.

The Lambda needs an IAM role:

resource"aws_iam_role""lambda"{name="pipeline-lambda-role-${var.environment}"assume_role_policy=jsonencode({Version="2012-10-17"Statement=[{Action="sts:AssumeRole"Effect="Allow"Principal={Service="lambda.amazonaws.com"}}]})}resource"aws_iam_role_policy_attachment""lambda_basic"{role=aws_iam_role.lambda.namepolicy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"}resource"aws_iam_role_policy""lambda_sns"{name="lambda-sns-publish"role=aws_iam_role.lambda.idpolicy=jsonencode({Version="2012-10-17"Statement=[{Effect="Allow"Action=["sns:Publish"]Resource=aws_sns_topic.alerts.arn}]})}
Enter fullscreen modeExit fullscreen mode

AWSLambdaBasicExecutionRole grants CloudWatch Logs write access, which is the minimum a Lambda needs to emit logs. Everything else (SNS, S3, DynamoDB) needs explicit policy attachments.

To schedule the Lambda, use EventBridge:

resource"aws_cloudwatch_event_rule""hourly"{name="hourly-pipeline-check"schedule_expression="rate(1 hour)"}resource"aws_cloudwatch_event_target""lambda"{rule=aws_cloudwatch_event_rule.hourly.nametarget_id="PipelineAlertLambda"arn=aws_lambda_function.alert.arn}resource"aws_lambda_permission""cloudwatch"{statement_id="AllowCloudWatchInvoke"action="lambda:InvokeFunction"function_name=aws_lambda_function.alert.function_nameprincipal="events.amazonaws.com"source_arn=aws_cloudwatch_event_rule.hourly.arn}
Enter fullscreen modeExit fullscreen mode

The aws_lambda_permission resource is easy to miss. Without it, EventBridge will attempt to invoke the Lambda and get an access denied error, even though the EventBridge rule and target are configured correctly. Lambda requires explicit permission grants for each invoking service.

Modules: Reusing Patterns Across Projects

Once you write a GCS bucket with lifecycle rules and IAM correctly once, you do not want to rewrite it for every project. Extract it into a module:

modules/
└── gcs_data_lake/
├── main.tf
├── variables.tf
└── outputs.tf
Enter fullscreen modeExit fullscreen mode
# modules/gcs_data_lake/main.tfresource"google_storage_bucket""this"{name=var.bucket_namelocation=var.locationproject=var.project_idstorage_class="STANDARD"uniform_bucket_level_access=trueforce_destroy=falselabels=var.labelsversioning{enabled=true}lifecycle_rule{condition{age=var.nearline_days}action{type="SetStorageClass";storage_class="NEARLINE"}}}# modules/gcs_data_lake/variables.tfvariable"bucket_name"{type=string}variable"project_id"{type=string}variable"location"{type=string;default="US"}variable"labels"{type=map(string);default={}}variable"nearline_days"{type=number;default=90}# modules/gcs_data_lake/outputs.tfoutput"bucket_name"{value=google_storage_bucket.this.name}output"bucket_url"{value=google_storage_bucket.this.url}
Enter fullscreen modeExit fullscreen mode

Use it from the root module:

module"landing_zone"{source="./modules/gcs_data_lake"project_id=var.project_idbucket_name="${var.project_id}-landing-${var.environment}"location="US"labels=var.labelsnearline_days=60}output"landing_bucket"{value=module.landing_zone.bucket_name}
Enter fullscreen modeExit fullscreen mode

Run terraform init after adding a module reference. Without it, Terraform does not know the module exists.

Common Errors and Actual Fixes

State lock error after a crashed run:

Error: Error acquiring the state lock
Lock Info:
ID: abc-123-def
Enter fullscreen modeExit fullscreen mode

A previous Terraform run exited without releasing the lock. Fix it with the lock ID from the error message:

terraform force-unlock abc-123-def
Enter fullscreen modeExit fullscreen mode

GCP 403 permission error:

Error: googleapi: Error 403: The caller does not have permission
Enter fullscreen modeExit fullscreen mode

The service account running Terraform is missing an IAM role. Fix it in the Terraform config with google_project_iam_member, or temporarily with:

gcloud projects add-iam-policy-binding MY_PROJECT \--member="serviceAccount:sa@project.iam.gserviceaccount.com"\--role="roles/bigquery.dataEditor"
Enter fullscreen modeExit fullscreen mode

GCS bucket name conflict:

Error: Error creating Bucket: googleapi: Error 409: The requested bucket name is not available
Enter fullscreen modeExit fullscreen mode

GCS bucket names are globally unique. Another account (or a previous version of your own project) already has that name. Add var.project_id or a random suffix to the bucket name.

GCP credentials not found:

Error: No valid credential sources found
Enter fullscreen modeExit fullscreen mode

Terraform cannot find GCP credentials. Fix with one of:

gcloud auth application-default login
# orexport GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-key.json
Enter fullscreen modeExit fullscreen mode

AWS Lambda default timeout:

Lambda's default timeout is 3 seconds. Any function doing API calls, database writes, or anything with network latency will time out. Set it explicitly in the resource:

resource"aws_lambda_function""alert"{timeout=30memory_size=256}
Enter fullscreen modeExit fullscreen mode

Maximum timeout is 900 seconds (15 minutes).

AWS provider 6.x boolean values:

If you have existing configs that use "0" or "1" for boolean attributes, provider 6.x rejects them. Update to true or false:

# Old (fails in provider 6.x)versioning_enabled="1"# Correctversioning_enabled=true
Enter fullscreen modeExit fullscreen mode

CI/CD: Running Terraform in GitHub Actions 

# .github/workflows/terraform.ymlname:Terraformon:push:branches:[main]paths:['terraform/**']pull_request:branches:[main]paths:['terraform/**']jobs:terraform:runs-on:ubuntu-lateststeps:-uses:actions/checkout@v4-uses:hashicorp/setup-terraform@v3with:terraform_version:"~>1.9"-name:Terraform Initworking-directory:terraform/run:terraform initenv:GOOGLE_CREDENTIALS:${{ secrets.GCP_SA_KEY }}-name:Terraform Validateworking-directory:terraform/run:terraform validate-name:Terraform Planworking-directory:terraform/run:terraform plan -out=tfplanenv:GOOGLE_CREDENTIALS:${{ secrets.GCP_SA_KEY }}TF_VAR_project_id:${{ secrets.GCP_PROJECT_ID }}TF_VAR_environment:prod-name:Terraform Applyif:github.ref == 'refs/heads/main' && github.event_name == 'push'working-directory:terraform/run:terraform apply -auto-approve tfplanenv:GOOGLE_CREDENTIALS:${{ secrets.GCP_SA_KEY }}
Enter fullscreen modeExit fullscreen mode

Pass sensitive variables through environment variables prefixed with TF_VAR_. Terraform picks them up automatically, mapping TF_VAR_project_id to var.project_id. This avoids putting credentials or project IDs in .tfvars files that might be committed.

The if: condition on Apply means the plan runs on every pull request but apply only runs when merged to main. Pull request authors see the plan output in the workflow logs before anything changes.

Terraform vs OpenTofu

In August 2023, HashiCorp changed Terraform's license from MPL 2.0 to the Business Source License (BUSL). The BSL prohibits using Terraform directly in competing products. OpenTofu is an open-source fork under the Linux Foundation that continued under MPL 2.0.

As of June 2026, both tools use the same HCL syntax and are largely compatible. OpenTofu 1.11 introduced ephemeral values (temporary credentials that never land in state), and its state encryption feature from 1.7 has no direct Terraform equivalent. A January 2026 survey found 31% of platform engineering teams had migrated at least one environment to OpenTofu.

For a data engineer building pipelines, the practical difference is minimal today. If you are using Terraform Cloud or HCP Terraform for remote state and collaboration, stay on Terraform. If you want open-source-only tooling or are concerned about the license, OpenTofu is a direct drop-in replacement: rename the binary and nothing else in your workflow changes.

The Three Files to Start With

Every new pipeline project gets three Terraform files from the start. They are the minimum needed to provision a data lake bucket and keep the state in a remote backend:

  1. providers.tf: provider versions pinned to major version ranges, remote backend configured
  2. variables.tf: project ID, region, environment, labels
  3. main.tf: GCS bucket or S3 bucket with versioning, encryption, lifecycle, and public access block

Run terraform plan before every terraform apply. Read the plan. A plan that shows destruction where you expected modification is telling you something about how the resource handles updates. Trust the plan more than you trust your memory of what you configured.

The Terraform patterns in this article are drawn from multiple data engineering projects using GCP and AWS. Infrastructure code for the Kenya Economic Pulse and BizPulse Kenya pipelines is on my GitHub.

Follow me on dev.to for more on data engineering, dbt, and Airflow.

Top comments (0)