Terraform State Management: Remote State, Locking & Team Collaboration

Terraform state is your infrastructure's source of truth. Store it locally, and you'll lose it. Manage it poorly in teams, and everyone's changes conflict. This guide covers production-grade state management.

Why State Matters

Terraform's .tfstate file tracks resource IDs, properties, and dependencies. Without it, Terraform can't:

Know which resources it created
Detect drift (changes made outside Terraform)
Destroy resources safely

# Local state: DANGEROUS in teams
terraform apply
# Both engineers modify the same state = conflicts, lost changes, corruption

Remote State: The Solution

Store state in a shared backend that supports locking:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

Setup: S3 + DynamoDB

# bootstrap/main.tf - Run once to create state infrastructure
resource "aws_s3_bucket" "terraform_state" {
  bucket = "my-terraform-state-${data.aws_caller_identity.current.account_id}"
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# State lock table
resource "aws_dynamodb_table" "terraform_locks" {
  name           = "terraform-locks"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

data "aws_caller_identity" "current" {}

Why DynamoDB?

Consistent locks (prevents concurrent modifications)
Fast: Lock/unlock in milliseconds
Reliable: AWS-managed

State Locking in Action

$ terraform plan
Acquiring state lock. This may take a few moments...

# While locked, other engineers see:
$ terraform plan
Error: Error acquiring the backend lock
Code: DynamoDBUpdateConditionCheck
Reason: The conditional request failed

Both engineers can't modify state simultaneously. First one wins, second waits.

Team Workflow

# main.tf
terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
  }

  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

Entire team now shares the same state:

# Engineer A
$ terraform init
$ terraform plan
$ terraform apply
# State locked during apply (~30 seconds)

# Engineer B (during A's apply)
$ terraform plan
Error: Error acquiring the backend lock
# Waits politely

# After A finishes
$ terraform plan
# Includes A's changes automatically
$ terraform apply

State Isolation: Workspaces & Modules

Workspaces (Per-Environment)

terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    key    = "terraform.tfstate"
    region = "us-east-1"
  }
}

$ terraform workspace list
  default
  dev
  staging
  prod

$ terraform workspace select prod
$ terraform apply
# Modifies prod/.../terraform.tfstate

$ terraform workspace select dev
$ terraform apply
# Modifies dev/.../terraform.tfstate (separate!)

Workspaces provide logical separation without managing multiple backends.

Directory Structure (Recommended)

terraform/
├── bootstrap/           # Creates state bucket (run once)
├── env/
│   ├── dev/
│   │   ├── terraform.tfvars
│   │   ├── main.tf      # Backend config
│   │   └── variables.tf
│   ├── staging/
│   └── prod/
├── modules/
│   ├── vpc/
│   ├── eks/
│   └── rds/
└── shared/
    └── outputs.tf

Each env has its own state:

# env/dev/main.tf
terraform {
  backend "s3" {
    bucket = "terraform-state"
    key    = "dev/terraform.tfstate"
  }
}

# env/prod/main.tf
terraform {
  backend "s3" {
    bucket = "terraform-state"
    key    = "prod/terraform.tfstate"
  }
}

State Management Best Practices

1. Never Commit .tfstate to Git

# .gitignore
*.tfstate
*.tfstate.*
.terraform.lock.hcl
.terraform/

State contains secrets (RDS passwords, API keys). Keep it private.

2. Backup State Regularly

resource "aws_s3_bucket" "terraform_state" {
  bucket = "my-terraform-state"
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"  # Keep all versions
  }
}

With versioning, recover any previous state:

$ aws s3api list-object-versions \
  --bucket my-terraform-state \
  --prefix prod/

# Restore to specific version
$ aws s3api get-object \
  --bucket my-terraform-state \
  --key prod/terraform.tfstate \
  --version-id XXXXXXXXX \
  terraform.tfstate

3. Encrypt State at Rest

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
      kms_master_key_id = aws_kms_key.terraform.arn
    }
  }
}

4. Restrict Access

resource "aws_s3_bucket_policy" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Deny"
      Principal = "*"
      Action = "s3:*"
      Resource = [
        aws_s3_bucket.terraform_state.arn,
        "${aws_s3_bucket.terraform_state.arn}/*"
      ]
      Condition = {
        Bool = { "aws:SecureTransport" = "false" }
      }
    }]
  })
}

Only allows HTTPS access.

State Commands for Debugging

# View current state
$ terraform state list
aws_instance.web
aws_security_group.web

# Show specific resource
$ terraform state show aws_instance.web
# id = i-0123456789abcdef0

# Manual refresh (detect drift)
$ terraform refresh
# Updates state with current AWS values

# Move resource (refactoring)
$ terraform state mv aws_instance.web aws_instance.app
# Updates resource references in state

# Remove from state (stop managing)
$ terraform state rm aws_instance.deprecated
# Resource still exists in AWS, Terraform stops tracking

Common State Issues

Lost State File

# State corrupted? Restore from S3 versioning
$ aws s3api get-object \
  --bucket terraform-state \
  --key prod/terraform.tfstate \
  --version-id ABC123 \
  terraform.tfstate

Lock Stuck

# Lock is dead? Force unlock (dangerous!)
$ terraform force-unlock <LOCK_ID>
# Check DynamoDB for lock entry first:
$ aws dynamodb scan \
  --table-name terraform-locks

State Drift

# Detect changes made outside Terraform
$ terraform plan -refresh-only
# Shows what changed in AWS

# Sync state with reality
$ terraform apply -refresh-only

Checklist

Conclusion

Remote state with locking is non-negotiable for teams. S3 + DynamoDB is the standard. Set it up once, forget about it, and focus on building infrastructure safely.

The 30 minutes you spend configuring this saves hours of debugging state conflicts later.

SUBID