IaC Environment Lifecycle Patterns - Architecture Insights

The Core Challenge

When recreating environments (especially dev/test), many resources generate dynamic identifiers that other resources depend on.

The solution to managing dynamic infrastructure identifiers is indirection. Use stable abstractions that point to changing resources.

The Problem

Resources generate unique identifiers each time:

RDS endpoint: mydb.abc123.us-east-1.rds.amazonaws.com → mydb.xyz789.us-east-1.rds.amazonaws.com
ElastiCache configuration endpoints change
Load balancer DNS names change
Resource ARNs include unique identifiers

Consumers need stable references:

Application configuration
IAM policies
Parameter Store values
Security group rules
DNS records

The Solution: Indirection

Three patterns for stable references to changing resources:

1. DNS Abstraction

Stable DNS name points to dynamic endpoint:

# DNS record updates automatically when resource recreated
resource "aws_route53_record" "db" {
  zone_id = aws_route53_zone.internal.zone_id
  name    = "db.${var.environment}.internal.example.com"
  type    = "CNAME"
  ttl     = 60
  records = [aws_db_instance.main.address]
}

# Application always uses: db.dev.internal.example.com

2. Parameter Store

Store dynamic values in centralized configuration:

resource "aws_ssm_parameter" "db_endpoint" {
  name  = "/${var.environment}/database/endpoint"
  type  = "String"
  value = aws_db_instance.main.endpoint
}

Application reads at startup:

var ssmClient = new AmazonSimpleSystemsManagementClient();
var dbEndpoint = await ssmClient.GetParameterAsync(
    new GetParameterRequest { Name = $"/{environment}/database/endpoint" }
);

3. Service Discovery

AWS Cloud Map for microservices:

resource "aws_service_discovery_instance" "db" {
  instance_id = aws_db_instance.main.id
  service_id  = aws_service_discovery_service.database.id

  attributes = {
    AWS_INSTANCE_CNAME = aws_db_instance.main.endpoint
  }
}

Resource Discovery Patterns

DNS Abstraction (Recommended for Most Cases)

Setup once per environment:

# Private hosted zone
resource "aws_route53_zone" "internal" {
  name = "${var.environment}.internal.example.com"
  vpc { vpc_id = aws_vpc.main.id }
}

# Stable DNS for database
resource "aws_route53_record" "database" {
  zone_id = aws_route53_zone.internal.zone_id
  name    = "db.${var.environment}.internal.example.com"
  type    = "CNAME"
  ttl     = 60
  records = [aws_db_instance.main.address]
}

# Stable DNS for cache
resource "aws_route53_record" "cache" {
  zone_id = aws_route53_zone.internal.zone_id
  name    = "cache.${var.environment}.internal.example.com"
  type    = "CNAME"
  ttl     = 60
  records = [aws_elasticache_cluster.main.configuration_endpoint]
}

Application configuration:

{
  "ConnectionStrings": {
    "Database": "Server=db.dev.internal.example.com;Database=app;...",
    "Cache": "cache.dev.internal.example.com:6379"
  }
}

Benefits:

Universal compatibility
Low TTL enables fast updates
No code changes when infrastructure recreated

Parameter Store for Complex Configuration

Hierarchical organization:

/{environment}/{service}/{key}

/dev/database/endpoint
/dev/database/port
/dev/cache/endpoint
/global/region

Write from infrastructure:

resource "aws_ssm_parameter" "db_endpoint" {
  name  = "/${var.environment}/database/endpoint"
  type  = "String"
  value = aws_db_instance.main.endpoint
}

resource "aws_ssm_parameter" "cache_endpoint" {
  name  = "/${var.environment}/cache/config-endpoint"
  type  = "String"
  value = aws_elasticache_replication_group.main.configuration_endpoint_address
}

Read from application:

public async Task<Dictionary<string, string>> GetConfigAsync(string service)
{
    var request = new GetParametersByPathRequest
    {
        Path = $"/{_environment}/{service}/",
        Recursive = true,
        WithDecryption = true
    };

    var response = await _ssmClient.GetParametersByPathAsync(request);

    return response.Parameters.ToDictionary(
        p => p.Name.Split('/').Last(),
        p => p.Value
    );
}

Combine with Secrets Manager:

# Secret in Secrets Manager
resource "aws_secretsmanager_secret" "db_password" {
  name = "${var.environment}/database/password"
}

# Store ARN in Parameter Store for discovery
resource "aws_ssm_parameter" "db_password_arn" {
  name  = "/${var.environment}/database/password-secret-arn"
  type  = "String"
  value = aws_secretsmanager_secret.db_password.arn
}

Choosing a Pattern

Use Case	Pattern
Database endpoints	DNS (simple) or RDS Proxy (production)
Cache clusters	DNS + Parameter Store
Complex config (multiple values)	Parameter Store
Microservices	Service Discovery
Simple references	DNS
Mix of static/dynamic config	Parameter Store hierarchy

Layered Lifecycle Management

Different infrastructure layers have different recreation frequencies.

Separation Principle

Separate infrastructure by change frequency. Fast-changing application code should not live in the same state file as slow-changing networking infrastructure. This reduces blast radius and speeds up deployments.

Layer Definitions

Foundation Layer (Weeks to Months)

VPCs, subnets, routing
NAT gateways, VPN connections
Base security groups
Route53 zones

Data Layer (Days to Weeks)

RDS instances
ElastiCache clusters
S3 buckets
Message queues

Application Layer (Hours to Days)

ECS services, Lambda functions
Application load balancers
Auto-scaling groups
IAM roles for applications

Implementation

Separate state files:

infrastructure/
├── foundation/
│   ├── backend.tf     # State: foundation/terraform.tfstate
│   ├── vpc.tf
│   └── dns.tf
├── data/
│   ├── backend.tf     # State: data/terraform.tfstate
│   ├── rds.tf
│   └── elasticache.tf
└── application/
    ├── backend.tf     # State: application/terraform.tfstate
    ├── ecs-service.tf
    └── lambda.tf

Cross-layer references:

# data/main.tf - reads from foundation
data "terraform_remote_state" "foundation" {
  backend = "s3"
  config = {
    bucket = "terraform-state"
    key    = "${var.environment}/foundation/terraform.tfstate"
  }
}

resource "aws_db_subnet_group" "main" {
  subnet_ids = data.terraform_remote_state.foundation.outputs.database_subnet_ids
}

# Write endpoint to Parameter Store for application discovery
resource "aws_ssm_parameter" "db_endpoint" {
  name  = "/${var.environment}/database/endpoint"
  value = aws_db_instance.main.endpoint
}

Application discovers via Parameter Store:

# application/main.tf - no direct Terraform dependency on data layer
# App reads from Parameter Store at runtime instead

Benefits

Faster iteration:

Deploy only the layer that changed (application: ~2 minutes)
No need to wait for unchanged layers (data, foundation)
Much faster than deploying all layers together (~20+ minutes)

Reduced blast radius:

Application layer changes don’t risk data layer resources
Can destroy and recreate application layer without affecting databases
Data remains intact during application experimentation

Independent ownership:

Platform team: Foundation + Data
Application teams: Application layer

Circular Dependency Resolution

Common Gotcha

Circular dependencies are one of the most common IaC deployment failures. The fix is almost always the same: separate resource creation from rule/policy attachment.

Common Scenarios

Security Groups Referencing Each Other

# ❌ Circular dependency
resource "aws_security_group" "app" {
  ingress {
    security_groups = [aws_security_group.db.id]
  }
}

resource "aws_security_group" "db" {
  ingress {
    security_groups = [aws_security_group.app.id]  # Circular!
  }
}

Solution: Separate rules from groups

# ✅ Create groups first
resource "aws_security_group" "app" {
  name = "app-sg"
}

resource "aws_security_group" "db" {
  name = "db-sg"
}

# Then create rules separately
resource "aws_security_group_rule" "app_to_db" {
  type                     = "egress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  security_group_id        = aws_security_group.app.id
  source_security_group_id = aws_security_group.db.id
}

resource "aws_security_group_rule" "db_from_app" {
  type                     = "ingress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  security_group_id        = aws_security_group.db.id
  source_security_group_id = aws_security_group.app.id
}

IAM Policies Need Resource ARNs Before Resources Exist

# ✅ Use wildcard patterns
resource "aws_iam_role_policy" "lambda_s3" {
  role = aws_iam_role.lambda.id

  policy = jsonencode({
    Statement = [{
      Action   = ["s3:GetObject", "s3:PutObject"]
      Resource = "arn:aws:s3:::${var.environment}-app-*/*"  # Pattern
    }]
  })
}

# Bucket name matches pattern
resource "aws_s3_bucket" "data" {
  bucket = "${var.environment}-app-data-${random_id.suffix.hex}"
}

Alternative: Tag-based policies

resource "aws_iam_policy" "app_s3_access" {
  policy = jsonencode({
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:GetObject", "s3:PutObject"]
      Resource = "*"
      Condition = {
        StringEquals = {
          "s3:ExistingObjectTag/Environment" = var.environment
          "s3:ExistingObjectTag/Application" = "myapp"
        }
      }
    }]
  })
}

Resolution Strategies

Strategy	When to Use
Wildcard patterns	Resource names follow predictable convention
Separate rules from resources	Security groups, network ACLs
Tag-based policies	Multiple resources with shared access patterns
Two-pass deployment	Complex dependencies unavoidable

Dev Environment Strategies

Strategy 1: Shared Data Layer

Structure:

Shared (persistent):
- VPC, subnets
- RDS (dev-shared-db)
- ElastiCache (dev-shared-cache)

Per-developer (ephemeral):
- ECS services (dev-alice-app)
- Lambda functions
- Load balancers

Implementation:

# Shared data (created once, persistent)
resource "aws_db_instance" "shared_dev" {
  identifier = "dev-shared-db"
}

resource "aws_ssm_parameter" "shared_db_endpoint" {
  name  = "/dev-shared/database/endpoint"
  value = aws_db_instance.shared_dev.endpoint
}

# Per-developer app (created/destroyed frequently)
variable "developer_name" {}

resource "aws_ecs_service" "app" {
  name = "dev-${var.developer_name}-app"
  # Reads: /dev-shared/database/endpoint
}

Workflow:

Developers create/destroy only their application layer:

Create workspace or use separate state for their environment
Deploy with developer-specific variables (e.g., developer_name=alice)
Application connects to shared data resources
Can destroy application infrastructure without losing data
Redeploy application layer and reconnects to same shared database

Best for: Cost-effective, fast iteration, stable schema

Strategy 2: Dedicated Environments

Structure:

Per-developer (complete isolation):
- RDS (dev-alice-db)
- ElastiCache (dev-alice-cache)
- All application resources

Implementation:

variable "developer_name" {}

locals {
  environment = "dev-${var.developer_name}"
}

resource "aws_db_instance" "db" {
  identifier = "${local.environment}-db"
  instance_class = "db.t3.micro"  # Small for dev
}

resource "aws_route53_record" "db" {
  name    = "db.${local.environment}.internal"
  records = [aws_db_instance.db.endpoint]
}

Cost management:

# Tag resources for automated stop/start
resource "aws_db_instance" "db" {
  tags = {
    Schedule = "dev-business-hours"  # Stop at 6 PM, start at 8 AM
  }
}

Best for: Schema changes, complete isolation, production parity

Strategy 3: Hybrid

Structure:

Shared foundation:
- VPC, NAT gateways

Per-developer:
- RDS (small instance)
- ElastiCache (minimal)
- Application resources

Shared with isolation:
- S3 (shared bucket, isolated prefixes)

Implementation:

# Foundation layer (shared)
resource "aws_vpc" "dev" {
  cidr_block = "10.0.0.0/16"
}

# Per-developer data layer
resource "aws_db_instance" "db" {
  identifier               = "dev-${var.developer_name}-db"
  db_subnet_group_name     = data.terraform_remote_state.foundation.outputs.db_subnet_group_name
}

# S3 with prefix isolation
resource "aws_ssm_parameter" "data_prefix" {
  name  = "/dev-${var.developer_name}/s3/data-prefix"
  value = "developers/${var.developer_name}/"
}

Application scopes to prefix:

var prefix = await GetParameter($"/{environment}/s3/data-prefix");
var key = $"{prefix}my-file.json";  // developers/alice/my-file.json

Best for: Many developers, balanced cost and isolation

Choosing a Strategy

Scenario	Recommendation
Tight budget, stable schema	Shared data layer
Frequent schema changes	Dedicated environments
Many developers (10+)	Hybrid
Short-lived feature branches	Dedicated (ephemeral)
Production parity required	Dedicated
Rapid app iteration	Shared data layer

Shared Data Layer

Cost: Lowest; one database for all devs
Setup: Simple; deploy once
Isolation: Low; shared resources
Schema Changes: Difficult; affects everyone
Best for: Stable schemas, tight budgets

Dedicated Environments

Cost: Higher; per-developer resources
Setup: Complex; automate everything
Isolation: Complete; full separation
Schema Changes: Easy; isolated testing
Best for: Schema changes, production parity

Key Takeaways

Use indirection for resource discovery:

DNS provides stable names for dynamic endpoints
Parameter Store centralizes configuration
Application code never contains infrastructure-specific identifiers

Layer infrastructure by change frequency:

Separate state files per layer (foundation, data, application)
Recreate only what changes
Reduced blast radius and faster iteration

Handle circular dependencies proactively:

Wildcard patterns in IAM policies
Separate resource creation from rule association
Tag-based policies for flexible access control

Choose dev environment strategy based on needs:

Shared data: Fast, cost-effective
Dedicated: Isolated, safe experimentation
Hybrid: Balanced approach