Resilience isn't just about infrastructure redundancy—it's about how you develop, test, and deploy systems. This section covers practical patterns for building resilient systems: environment separation, dependency management, deployment strategies, and leveraging cloud provider fault isolation boundaries.

# Environment Separation: Dev/QA/Prod

Separating production from non-production resources reduces cross-environment failure and access risk and provides places to test changes before production.

Illustrative responsibilities—adapt these environments and data controls to the delivery model and test purpose:
Development (Dev):
    - Individual developers test code changes
    - Rapid iteration, frequent breakage acceptable
    - May use mock/stub dependencies
    - Data: Synthetic or anonymized

Quality Assurance (QA/Staging):
    - Integration testing, load testing, security testing
    - Production-like configuration and data volume
    - Validates changes before prod deployment
    - Data: Anonymized production snapshot or realistic synthetic

Production (Prod):
    - Live user traffic
    - High availability, monitoring, on-call
    - Changes deployed only after QA validation
    - Data: Real user data

Why separate: Strong boundaries can keep many non-production failures and data-access mistakes from affecting production. Representative pre-production tests can find defects before release, but they do not guarantee production stability.

Common Pitfalls

• Shared databases: Dev and QA sharing same database → test data pollution, accidental prod data access
• "Read-only" prod access: Developers with read-only prod access still risk data leaks, and "just this once" writes
• QA not prod-like: A QA data set or load that is not representative of the test can miss performance issues that appear only at scale
• No staging: Deploying dev → prod directly skips integration testing, increases risk

Best practice: Strongly isolate production from non-production resources and data. Make each test environment reproduce the topology, configuration, load, and data characteristics relevant to that test, using realistic synthetic or appropriately protected data. Enforce the intended boundaries with IAM/RBAC, accounts, and network controls.

# Dependency Lifecycle Management

Dependencies (DNS, authentication, databases, APIs) are often shared across systems. How do you safely change a dependency without breaking everything?

Problem: Shared Dependencies

Solution: Give shared dependencies a deliberate change lifecycle. Preserve backward compatibility where feasible; when a contract must break, version or otherwise run old and new behavior during a defined client migration.

Pattern: Versioned Dependencies

1. Deploy Auth v2 alongside v1
2. Update App 2 to use v2 (test, validate)
3. Update App 1 to use v2
4. Update App 3 to use v2
5. Decommission v1 once no clients remain

DNS example: Before changing example.com from 10.0.1.5 to 10.0.1.10, lower the TTL far enough in advance for values cached under the old TTL to expire, and keep both endpoints able to serve traffic through the planned transition. Account for clients or resolvers that retain addresses longer or serve stale data during authoritative failure, and test the change in dev/QA before production.

# Deployment Strategies

How you deploy changes affects blast radius and rollback speed.

Blue-Green Deployment

Maintain separate blue and green environments. Deploy and test the new version in the inactive environment, then shift production traffic to it.

Pros: Can provide rapid traffic rollback, pre-switch validation, and near-zero downtime when routing and state migrations are designed for it.

Cons: Requires parallel capacity whose cost depends on the workload and platform. Shared state and schema migrations must remain compatible with any version that may receive traffic.

Canary Deployment

Gradually roll out to small percentage of traffic, monitor, then expand.

Pros: Can limit initial exposure and validate a release with production signals when the canary population, metrics, and abort criteria are representative.

Cons: Requires traffic targeting and reliable comparative monitoring; rollout time, extra capacity, and cost depend on the platform and analysis window.

Rolling Deployment

Update instances one at a time (or in small batches).

Pros: Replaces capacity incrementally and may avoid provisioning a second full environment; resource needs depend on surge and availability settings.

Cons: Old and new versions must coexist safely during rollout. Rollback speed depends on the controller, capacity settings, image availability, and health checks.

# Service Dependencies and Circuit Breakers

In microservice architectures, services depend on each other. How do you prevent one service failure from cascading?

Dependency Graph

Problem: With synchronous calls, a bounded shared thread or connection pool, and enough slow Product Service requests, the API Gateway can wait on timeouts until the pool is exhausted. That can make the gateway unavailable to the frontend and create a cascading failure.

Mitigations: Use bounded timeouts, appropriately configured circuit breakers, and bulkheads where they match the failure model; together with admission control and graceful degradation, they can limit resource exhaustion and cascading impact.

# AWS Fault Isolation Boundaries

AWS provides guidance on designing systems using their fault isolation boundaries. Key takeaways:

Zonal Services (AZ-Scoped)

Some services operate within a single AZ: EBS volumes, EC2 instances, subnet-specific resources.

Design implication: Distribute across multiple AZs. Don't put all EC2 instances in one AZ. Use Auto Scaling Groups with multi-AZ configuration.

Regional Services (Region-Scoped)

S3, DynamoDB, Elastic Load Balancing, and Lambda are Regional services. Their service architectures use multiple AZs, but workload impact from an AZ failure still depends on enabled zones, targets, dependencies, capacity, and application configuration.

Design implication: Use Regional services or correctly designed multi-AZ architectures to reduce exposure to a single-AZ infrastructure failure, and verify workload capacity and dependencies. For cross-Region requirements, evaluate patterns such as S3 Cross-Region Replication and DynamoDB Global Tables as parts of a tested recovery or multi-Region design.

Control Plane vs Data Plane

AWS recommends understanding which API calls are control plane (RunInstances, DeleteBucket) vs data plane (GetObject, PutItem).

During impairments: Some existing data-plane operations may continue when control-plane operations are unavailable, but this depends on the service and failure. Design critical runtime and recovery paths to avoid unnecessary control-plane changes.

Example: Where requirements justify it, pre-provision enough resources or warm capacity to tolerate a failure without first creating new infrastructure. Do not assume that on-demand instance creation or Auto Scaling control-plane calls will be available during the same impairment.

# Key Takeaways

• Strongly isolate production from non-production; deliberately governed shared resources can still create correlated failures
• Make pre-production tests representative of the topology, configuration, load, and data characteristics relevant to the risk being tested
• Shared dependencies need a deliberate change lifecycle; preserve compatibility or provide a controlled migration for breaking contracts
• Deployment strategies include blue-green (separate environments and a traffic shift), canary (limited initial exposure), and rolling (incremental replacement); rollback behavior depends on the platform and state compatibility
• Use bounded waits and appropriate isolation, such as timeouts, circuit breakers, bulkheads, and admission control, to reduce dependency-cascade risk
• AWS: distribute zonal resources across AZs, understand each Regional service's workload dependencies, and minimize runtime reliance on control-plane operations
• Test deployment strategies in QA before prod—ensure rollback actually works