mateusz@systems ~/book/failure-domains $ cat section.md

Failure Domains & Blast Radius

A failure domain is a set of components that fail together due to a shared dependency. Understanding failure domains is fundamental to designing resilient systems—if you don't know what can fail together, you can't properly separate your redundancy.

# Types of Failure Domains

Electrical Failure Domains

Devices whose only power path is the same source lose power together when that source is lost.

PDU A × fails Server 1 Server 2 Server 3 PSU-A only PSU-A only PSU-A only → all three lose power
Electrical failure domain — if servers have only a single PSU supplied by PDU A, a PDU failure takes them all down. Mitigation: where the hardware supports it, configure two full-capacity PSUs for 1+1 redundancy and connect them to independent PDUs and power feeds.

Network Failure Domains

Devices whose only network path traverses the same switch or router lose connectivity when that device fails.

ToR Switch × fails Server A Server B Server C → all lose network connectivity
Network failure domain — these servers have only one path through the top-of-rack switch, so they lose connectivity when it fails. Mitigation: use correctly configured multi-homing to independent switches, such as active-backup bonding or an MLAG design.

Geographic Failure Domains

Natural disasters, broad power failures, or network partitions can affect an entire region.

AWS Region: us-east-1 AZ-A Servers Storage Network AZ-B Servers Storage Network regional power-grid failure can affect all AZs AZ-C Servers Storage Network
Availability zones inside one region share the same underlying geography — a regional power-grid failure (or similar large-scale event) can affect every AZ at once. Mitigation: multi-region deployment (e.g. us-east-1 + us-west-2).

# Blast Radius

Blast radius is the scope of impact when a failure occurs. Small blast radius = localized failure. Large blast radius = widespread outage.

Small Blast Radius (Good) Single server fails 1/100 capacity lost Load balancer redistributes No user impact if failover and headroom suffice Large Blast Radius (Bad) Shared database fails All hard-dependent services fail No redundancy Complete outage
Illustrative blast-radius comparison — in a 100-server equal-capacity pool with working health checks, routing, and sufficient headroom, losing one server removes 1% of installed capacity without user impact. If every service has a hard dependency on one database and no fallback, that database's failure affects all of those services.

Design principle: Minimize blast radius by avoiding shared single points of failure and distributing load across independent failure domains.

# Correlated Failures

Correlated failures occur when seemingly independent components fail due to a shared root cause.

Example 1: Configuration Changes

Step 1 Deploy config change to entire fleet at once Step 2 Bug in config crashes all servers shared failure mode, all replicas Step 3 Entire service down (100% failure rate)
Illustrative sequence — a bad configuration is pushed fleet-wide, every server hits the same failure mode, and the service becomes unavailable. Example mitigation: evaluate the change on a small canary and expand it in measured stages; choose rollout percentages and gates from the system's risk and observed canary data.

Example 2: Software Bugs Under Load

Scenario: Service has 5 replicas. Load increases, triggering memory leak in all replicas simultaneously. All 5 crash within minutes.

Problem: Replicas are identical software, hit same bug under same conditions. Redundancy didn't help—correlated failure mode.

Mitigation: Use a time-bounded staged release: keep adjacent versions compatible during a defined rollout window, expose a limited canary population, evaluate health and correctness gates, roll back when those gates fail, and retire the old version after the new one is accepted. Load shedding can also keep replicas below the failure threshold.

Example 3: Shared Dependencies

App 1 App 2 App 3 Auth Service (shared dependency) × unavailable fresh decision required → fail closed eligible operation → bounded signed-token validation
All three applications share one authentication dependency. During an outage, sensitive or side-effecting operations that require fresh authorization fail closed. Only operations explicitly designed for it may validate a cached signed token, bounded by its audience, scope, short validity, and the accepted risk that revocation information may be stale.

This is an operation-specific degraded mode, not an authentication bypass. Whether cached validation is safe depends on the protected action and the token's audience, scope, expiry, and revocation model.

# Separating Failure Domains

Distribute resources across independent failure domains to prevent correlated failures.

Poor Separation Rack 1 Primary DB Replica 1 Replica 2 Rack 2 (empty) Rack power failure → all DB instances down Good Separation Rack 1 Primary DB Rack 2 Replica 1 Rack 3 Replica 2 Rack failure → 1 instance down, others continue
Poor separation crams the primary database and both replicas into one rack, so a single rack (power or network) failure takes all three down. Good separation spreads them across three racks, so one rack failure costs only one instance while the others keep serving.

Cloud Failure Domain Separation

In AWS, distribute across Availability Zones (AZs):

Bad us-east-1a 3 web servers database primary + replicas us-east-1b (empty) us-east-1c (empty) AZ failure → total outage Good us-east-1a web db primary us-east-1b web db replica us-east-1c web db replica AZ failure → 33% capacity loss, service continues
Bad: all web and database capacity sits in one AZ, so that AZ's failure is a total outage. Good: web and database replicas are spread across three AZs, so a single AZ failure costs 33% capacity while the service keeps running.

# Real-World Example: AWS October 2025 DynamoDB Outage

On October 19–20, 2025, AWS experienced a major service disruption in US-EAST-1 after DynamoDB's DNS management system published an empty regional-endpoint record. DynamoDB, EC2, Network Load Balancing, Lambda, ECS, EKS, Fargate, and other services experienced distinct periods of elevated errors or impairment; existing EC2 instances remained healthy.

Failure domain issue: Independent DNS Enactors could race while applying and cleaning up plans. A delayed Enactor applied a stale plan; another Enactor's cleanup then deleted that now-active plan, leaving the endpoint with an empty DNS record and inconsistent state that required manual operator correction. AWS described the Enactor as deliberately designed with minimal dependencies.

Blast radius amplification: The downstream impact was not one linear cascade. DynamoDB endpoint recovery was followed by an EC2 Droplet Workflow Manager (DWFM) lease backlog and congestive collapse, a network-configuration propagation backlog, and Network Load Balancer health-check behavior that removed healthy capacity. Lambda experienced several dependency-specific periods of impact; ECS, EKS, and Fargate experienced container launch failures and cluster-scaling delays, and the last container-service impacts recovered by 2:20 PM.

Lesson: Independent reconcilers need safeguards that keep stale work and cleanup from invalidating the active plan. Recovery design must also account for queued work, feedback loops, and the distinct dependencies through which one regional event can expand its blast radius.

# Key Takeaways

  • Failure domains are components that fail together due to shared dependencies (power, network, geography)
  • Blast radius is the scope of impact; minimize by avoiding shared single points of failure
  • Correlated failures occur when independent replicas fail due to same root cause (config, code bug, dependency)
  • Separate failure domains: distribute across racks, AZs, regions depending on availability requirements
  • Protect active control-plane state from stale-plan and cleanup races, and design recovery for backlogs and feedback loops
  • Test failure scenarios to validate blast radius is as expected