Chapter 6
Redundancy & Resilient System Design
Systems fail. Hardware fails, networks partition, software has bugs, and humans make mistakes. The difference between a minor glitch and a major outage lies in how systems are designed to handle failures. This chapter explores the principles and patterns for building resilient systems that degrade gracefully, recover automatically, and limit the blast radius of failures.
Resilience isn't just about redundancy—it's about understanding failure modes, separating concerns, managing state carefully, and testing that your failover actually works. We'll examine real-world outages (AWS, GitHub, Facebook, Cloudflare) to understand what went wrong and what we can learn.
This chapter covers practical techniques for designing resilient systems:
- Failure Domains: How to limit blast radius and prevent correlated failures
- Redundancy Models: N+1, N+2, 2N, active-active vs active-passive trade-offs
- Control vs Data Plane: Why separating them prevents cascading failures
- State Management: Consensus, quorum, and avoiding split-brain scenarios
- Geographic Distribution: Multi-datacenter patterns and consistency trade-offs
- Graceful Degradation: Circuit breakers, retries, timeouts, and health checks
- Building for Resilience: Dev/QA/prod separation, deployment strategies, dependency management
- Testing Resilience: Chaos engineering and fault injection
By the end of this chapter, you'll understand how to design systems that survive failures—and know what questions to ask when reviewing architecture for resilience.
# Chapter Sections
What is a failure domain (electrical, network, geographical). Separating failure domains to limit blast radius. Correlated failures and how to avoid them. Examples from datacenter infrastructure and cloud environments.
Redundancy ModelsN+1, N+2, 2N redundancy explained with cost/availability trade-offs. Active-active vs active-passive architectures. When to use which model. Comparison tables and diagrams.
Control Plane vs Data Plane SeparationWhat they are and why separate them. Metadata vs data: when control plane failures cascade. Real-world example: AWS S3 2017 outage (an incorrect operational command removed excessive subsystem capacity and made S3 APIs unavailable) and AWS DynamoDB October 2025 outage (a stale-plan and cleanup race in DNS management, followed by several distinct recovery failures).
State Management & ConsensusStateless vs stateful services. Quorum and split-brain scenarios. Consensus protocols (Raft/Paxos) at high level. Real-world example: GitHub's 2018 network-partition-triggered database failover and divergent-write recovery. State replication strategies.
Geographic DistributionMulti-datacenter architectures. Consistency vs latency (CAP theorem in practice, not theory). Replication strategies (sync/async/semi-sync). Examples: Netflix multi-region, Google Spanner, S3 cross-region replication.
Graceful DegradationRetry strategies and exponential backoff. Circuit breakers in practice (with pseudocode example). Timeout configuration. Health checks that actually work. Bulkheading and failure isolation.
Building Resilient Systems: Practical GuidanceEnvironment separation (dev/qa/prod). Dependency lifecycle management (DNS, auth, monitoring). Testing strategies and deployment patterns (blue-green, canary, rolling). Service dependencies and circuit breakers. AWS fault isolation boundaries guidance.
Testing ResilienceChaos engineering principles. Game days and disaster recovery drills. Risk-based choices about what to exercise and when. Building confidence in failover mechanisms.
Notable Incidents & Lessons LearnedAnalysis of major outages: AWS DynamoDB October 2025 (stale-plan/cleanup race and distinct downstream recovery mechanisms), Facebook BGP October 2021 (routes withdrawn, DNS unreachable), GitHub 2018 (network-partition-triggered database failover and divergent-write recovery), Cloudflare BGP leak 2019 (a route leak amplified by missing route filtering). Common patterns and lessons.