Real-world outages teach us more than any theoretical discussion. This section summarizes major incidents discussed throughout this chapter and extracts common patterns and lessons.

# AWS DynamoDB DNS Outage (October 19-20, 2025)

What Happened

A delayed DNS Enactor applied a stale plan after a newer plan had been used. Cleanup by another Enactor then deleted the old plan that had become active again, leaving the regional DynamoDB endpoint in US-EAST-1 with an empty DNS record and inconsistent state. Automated repair could not resolve that state, so operators corrected it manually.

Timeline

11:48 PM PDT Oct 19: DynamoDB DNS failure begins
12:38 AM PDT Oct 20: DynamoDB DNS state identified as the outage source
 2:25 AM PDT:        DNS information restored
 4:14 AM PDT:        Incoming DWFM work throttled; selective DWFM host restarts begin
 5:28 AM PDT:        DWFM leases restored; launches begin succeeding, with throttling errors remaining
10:36 AM PDT:        Network propagation returns to normal; new launches operate normally
 1:50 PM PDT:        All EC2 APIs and new instance launches operate normally
 2:20 PM PDT:        Last container-service impacts recover; overall event ends

Services Impacted

• DynamoDB (regional-endpoint DNS failures prevented new connections and raised API error rates)
• EC2 (existing instances remained healthy, but DynamoDB-dependent DWFM state checks failed; new launches and later network configuration were impaired)
• Lambda (distinct DynamoDB, SQS-poller, NLB, EC2-capacity, and throttling impacts)
• ECS, EKS, and Fargate (container launches and cluster scaling were delayed or failed)
• AWS Management Console and STS (separate authentication/API error periods)
• NLB (some load balancers saw connection errors after alternating health checks removed healthy capacity)

Mechanisms

• Race condition: A delayed DNS Enactor overwrote a newer plan with its stale plan; another Enactor's cleanup then deleted that now-active old plan, producing the empty record and an inconsistent state that blocked automatic repair.
• Minimal-dependency design: AWS described the Enactor as deliberately designed with few dependencies.
• EC2 recovery: DynamoDB-dependent DWFM state checks failed and leases expired. After the endpoint recovered, a large lease backlog produced congestive collapse until work was throttled and selected hosts restarted.
• Networking and NLB: Network configuration propagation accumulated a backlog, while health-check behavior caused some Network Load Balancers to remove healthy capacity.
• Lambda and containers: Lambda experienced several dependency-specific periods of impact; ECS, EKS, and Fargate experienced container launch failures and cluster-scaling delays, and the last container-service impacts recovered by 2:20 PM.

Lessons Learned

• Independent reconcilers and cleanup jobs must not be able to delete a plan that has become active again
• Test inconsistent control-plane states that may require repair, not only loss of a dependency
• Recovery tests must include queued work, throttling, and feedback loops after the initiating service is restored
• Map service-specific dependencies and recovery mechanisms instead of assuming one linear cascade

# AWS S3 Outage (February 28, 2017)

What Happened

During work on the S3 billing system, an authorized operator supplied an incorrect input to an established operational command. The command removed more capacity than intended from the index and placement subsystems.

Impact

• The index subsystem was required for GET, LIST, PUT, and DELETE requests
• S3 could not service requests while the index and placement subsystems restarted
• Recovery proceeded in stages as the subsystems restored capacity and completed metadata safety checks
• Applications and AWS services that depended on S3 were also impaired

Lessons Learned

• Place hard bounds and validation around administrative commands that can remove production capacity
• Add safeguards that prevent a command from removing excessive subsystem capacity
• Exercise large subsystem restarts and metadata safety checks at current scale
• Do not treat this incident as proof that cached reads or a separate data plane remained available

# Facebook Backbone and DNS Outage (October 4, 2021)

What Happened

During routine maintenance, a command intended to assess global backbone capacity unintentionally took down every connection in Facebook's backbone network, disconnecting its datacenters globally. A bug in the command-audit tool failed to stop the change. DNS locations that could no longer reach the datacenters then declared themselves unhealthy and withdrew their own BGP advertisements, making Facebook's still-running authoritative DNS servers unreachable and disrupting the company's platforms.

Impact

Backbone connections removed:
        Facebook datacenters disconnected from one another and the internet
        DNS locations could no longer reach the datacenters
        DNS servers withdrew their BGP advertisements
        DNS servers remained operational but became unreachable

Why DNS Didn't Help

Facebook's authoritative DNS servers were configured to withdraw their BGP advertisements when they could not communicate with the company's datacenters. Once the backbone was removed, those advertisements were withdrawn, so the still-operational DNS servers could no longer be reached from the internet.

Lessons Learned

• High-blast-radius maintenance commands need effective safeguards; a bug in Facebook's command-audit tool allowed this one to run
• Primary and out-of-band recovery paths should not share one failure mode; both were unavailable here
• DNS and routing dependencies must be mapped and tested; Facebook's DNS servers deliberately withdrew BGP advertisements when they lost datacenter connectivity
• Recovery required onsite engineers, and the datacenters' physical and system security made emergency access slower

# GitHub Cross-Region Database Failover (October 21, 2018)

What Happened

Routine network maintenance interrupted connectivity between GitHub's US East Coast network hub and its primary East Coast datacenter for 43 seconds. The East Coast Orchestrator leader deselected after losing its Raft quorum. Orchestrator nodes on the West Coast and in an East Coast public-cloud site then formed a quorum and began cross-region failover to West Coast MySQL replicas.

Impact

Before:
    East Coast MySQL primaries replicate to West Coast replicas

During the 43-second interruption:
    East Orchestrator leader deselects after losing Raft quorum
    West + East public-cloud Orchestrator nodes form quorum
    Orchestrator begins promoting West Coast replicas

Result:
    A brief window of East writes had not replicated west
    Applications then wrote to the promoted West primaries
    The East and West datasets diverged

The problem was not two independent Orchestrator groups each believing they held quorum. Only the West Coast and East Coast public-cloud Orchestrator nodes established a quorum, but that quorum was permitted to promote databases across a regional boundary into a topology the application could not safely support.

Recovery

After the West Coast databases had accepted more than 30 minutes of writes, GitHub treated West as canonical and failed forward. It rebuilt East Coast replicas from backups and West Coast data before later restoring East Coast primaries. GitHub preserved the East Coast binary logs and manually analyzed the few seconds of East writes that had not reached West; its post-incident report says no user data was ultimately lost, although reconciliation continued after service recovery.

Lessons Learned

• A valid control-plane quorum does not by itself make every promotion safe for the application topology
• Set topology-aware promotion boundaries and fence old writers before routing writes to a promoted database
• Test the complete regional failover policy, including application routing, replication position, and recovery direction
• Preserve database logs needed to analyze and reconcile any writes outside the canonical history

# Google Cloud London Cooling Failure (July 19-20, 2022)

What Happened

A simultaneous failure of multiple redundant cooling systems, compounded by extreme outside temperatures, left one building that hosted part of europe-west2-a unable to maintain a safe temperature. Google powered down that part of the zone to prevent hardware damage, terminating some VMs and impairing multiple services.

Impact

• A subset of Compute Engine VMs terminated or lost connectivity, and some Persistent Disk volumes returned I/O errors
• Several regional services were also impaired because an internal traffic-routing change initially avoided all three europe-west2 zones
• Cooling-system impact lasted 4 hours 8 minutes; service restoration and residual effects continued into July 20

Lessons Learned

• Power and cooling belong in the system's failure model
• Distribute critical workloads across independent physical failure domains, but also test the routing and dependencies needed to use them
• Protective shutdowns can create service impact and should be included in capacity and recovery planning

# Cloudflare BGP Leak (June 2019)

What Happened

A BGP route leak during a configuration error caused traffic intended for Cloudflare to be routed through a small ISP, which couldn't handle the volume. This disrupted Cloudflare services.

How More-Specific Routes Expanded the Blast Radius

Cloudflare uses anycast, but the BGP optimizer split advertised prefixes into more-specific routes and Verizon propagated them globally without appropriate filtering. Networks prefer the more-specific routes, so traffic worldwide was drawn through networks that could not carry it; Cloudflare measured about 15% loss of its global traffic at the peak.

Lessons Learned

• BGP route leaks can redirect traffic to unintended destinations
• Anycast did not contain this leak because globally propagated more-specific routes overrode the broader advertisements
• BGP filtering and validation needed to prevent route leaks

# Common Patterns Across Incidents

Control-Plane Races

DynamoDB 2025: A delayed Enactor applied a stale DNS plan, and another Enactor's cleanup deleted that now-active plan. Safeguards must prevent stale work and cleanup from invalidating active state.

Recovery Amplification

DynamoDB 2025: Endpoint recovery was followed by EC2 DWFM congestive collapse, network propagation backlog, NLB health-check effects, Lambda-specific impacts, and ECS, EKS, and Fargate container launch failures and cluster-scaling delays; the last container-service impacts recovered by 2:20 PM. Restoring the initiating dependency did not clear every downstream mechanism.

Human Error

S3 2017: An incorrect command input removed too much subsystem capacity. Facebook 2021: A high-blast-radius backbone command ran because an audit-tool bug failed to stop it, which later caused DNS locations to withdraw BGP advertisements. Operational mistakes are foreseeable—use layered safeguards, bounded change velocity, validation, and tested recovery paths.

Unsafe Promotion Boundaries

GitHub 2018: Orchestrator formed a Raft quorum, but it was permitted to promote databases across a regional boundary that the application topology could not safely support. Promotion policy, fencing, and application routing must be designed and tested together.

Physical Infrastructure Matters

Google 2022: A cooling failure removed zonal capacity. Software placement and recovery design can limit physical-failure impact, but an erroneous regional routing change also impaired multi-zone services.

# Key Takeaways

• Control-plane reconcilers and cleanup jobs must protect active state from stale work
• Recovery can amplify an incident through backlogs, feedback loops, and service-specific dependencies even after the initiating fault is repaired
• Operational mistakes are foreseeable—use layered safeguards, bounded changes, validation, and tested recovery paths
• Consensus quorum and application-safe failover are separate concerns; constrain promotions by topology and fence old writers
• Physical failures can remove capacity—distribute across independent zones or sites and test the dependencies and routing needed to survive their loss
• High-blast-radius network changes need tested safeguards, validation, and independent recovery access
• Control/data plane separation can limit propagation only when selected data-plane operations retain valid state and avoid live control-plane dependencies
• Test failure scenarios regularly—untested failover leaves critical assumptions and recovery timing unvalidated