You cannot establish failover behavior from design alone. Redundancy and circuit breakers that have never been exercised can contain hidden configuration, dependency, capacity, or runbook failures. This section covers chaos engineering, fault injection, and game days for gathering evidence about resilience mechanisms.

# Chaos Engineering Principles

Chaos engineering is controlled experimentation used to build confidence that a system can withstand turbulent conditions. Experiments can begin in non-production and, when risk and safeguards permit, run against production conditions. Netflix's production use of Chaos Monkey helped popularize the practice.

Core Principles

• Build a measurable hypothesis: Define the steady-state metric and an experiment-specific acceptable deviation—for example, error-rate change after one instance is lost
• Vary realistic events: Exercise failures such as instance loss, latency, resource pressure, or dependency impairment
• Limit blast radius: Start in non-production or with a small target set; define guardrails, stop conditions, and recovery steps before expanding
• Automate selectively: Repeat mature, bounded experiments when automation provides continuing evidence
• Measure and learn: Compare observations with the hypothesis, record unexpected effects, and fix discovered weaknesses

Why in Production?

Why production can matter: Production has traffic, scale, data characteristics, and dependencies that staging may not reproduce. After proving the experiment in non-production, a tightly scoped production experiment can reveal production-only behavior when organizational risk permits it.

Use explicit controls: Run approved experiments while qualified responders are available, with observability, a small blast radius, stop conditions, and tested recovery steps. A planned experiment is still a production risk and is justified only when its expected evidence outweighs that risk.

# Game Days and DR Drills

Game days are scheduled events where teams simulate major failures and practice recovery procedures.

Game Day Scenarios

• Zone or Region impairment: Simulate the specific scoped dependency loss and verify the designed failover or degraded mode
• Database primary failure: Force failover to replica, validate data consistency
• Network partition: Isolate services, test split-brain handling
• Deployment gone wrong: Deploy bad code, practice rollback procedures
• Cascade containment: Apply bounded load to one service and verify admission control, timeouts, circuit breakers, and bulkheads limit downstream and upstream impact

Game Day Process

1. Plan (lead time depends on scope and approvals):
    - Define scenario (e.g., "selected dependencies in the us-east-1 Region are unavailable")
    - Write hypothesis (e.g., "traffic fails over to us-west-2 in <5 min")
    - Schedule (business hours, team on standby)
    - Notify stakeholders

2. Execute (day of):
    - Activate the approved, scoped fault injection or simulation
    - Monitor dashboards, alerts, user impact
    - Document timeline, decisions, actions taken
    - Recover when done or if unexpected issues arise

3. Debrief (promptly after the exercise):
    - What worked? What didn't?
    - Were metrics accurate?
    - Did alerts fire as expected?
    - Runbooks up to date?
    - Action items to improve

4. Fix and Iterate:
    - Fix issues discovered
    - Update runbooks, dashboards, alerts
    - Set the next exercise from workload impact, change rate, prior findings, recovery objectives, and applicable requirements

Disaster Recovery (DR) Drills

DR drills exercise backup, restore, and service-recovery procedures. They can provide evidence for applicable controls, but the required scope, frequency, and evidence depend on the organization's control mapping and assessor expectations.

DR Drill Scenario: Database Corruption
    1. Simulate corruption (in DR environment, not prod!)
    2. Restore from backup
    3. Validate recovered data with checksums and domain-specific invariants
    4. Measure end-to-end recovery time and compare it with the RTO
    5. Determine the recovered data point, calculate the elapsed data-loss window to the disruption, and compare that window with the RPO

Metrics to Validate:
    - Actual recovery time: Compare with the RTO, the maximum acceptable restoration delay
    - Actual data-loss window: Time from the recovered data point to the disruption; compare with the RPO, the maximum acceptable amount of time since the last recovery point
    - Data integrity: Checksums and domain-specific invariants pass
    - Application compatibility: The application works correctly with restored data

Illustrative objectives only—replace with workload-approved values:
    - RTO: <4 hours
    - RPO: <1 hour

# What to Test and When

Choose exercises and their cadence from the workload's risk rather than a universal schedule. Consider business impact, RTO and RPO, the rate of architectural and operational change, prior findings, the safeguards and stop conditions available, the environment in which the exercise can run, and applicable control mappings. Retest after material changes or when evidence has gone stale enough that important recovery assumptions are no longer well supported.

• Prioritize: Exercise high-impact recovery paths and assumptions whose failure would violate recovery objectives
• Scope safely: Match the target environment, blast radius, observability, stop conditions, and responder availability to the experiment's risk
• Use findings: Increase attention where changes, incidents, or previous exercises expose uncertainty; reduce repetition only when current evidence supports doing so
• Map requirements: Let organizational control mappings and assessor expectations determine any compliance evidence the exercise must produce

# Building Confidence in Failover

Failover mechanisms are only trustworthy if regularly tested. Common issues discovered during testing:

• Stale runbooks: Documented failover process doesn't match current infrastructure
• Untested scripts: Failover script has typo, fails when actually run
• Hidden dependencies: "Backup" datacenter depends on primary for DNS/auth/config
• Capacity assumptions: Backup DC can't handle 100% traffic (overloads)
• Data lag: Replica is 2 hours behind, unacceptable for failover
• Manual steps: Failover requires 10 manual steps, takes 2 hours

Use regular testing to find these issues, then track corrective actions to completion. Automation can reduce recovery time and manual error when triggers and rollback are safe, but both automated and manual recovery paths need periodic validation.

# Tools for Chaos Engineering

• Chaos Monkey (Netflix): Randomly terminates production VM instances or containers managed through supported Spinnaker backends
• Gremlin: Commercial reliability and fault-injection platform
• LitmusChaos (CNCF Incubating): Cloud-native chaos engineering platform commonly used with Kubernetes
• Pumba: Container chaos, network emulation, and stress testing for Docker, containerd, and Podman
• tc (traffic control): iproute2 utility that configures Linux kernel traffic control, including netem fault injection
• stress / stress-ng: System load generators for CPU, memory, I/O, disk, and other resources (capabilities differ by tool)

# Key Takeaways

• Chaos engineering tests explicit resilience hypotheses through controlled experiments
• Where risk permits, progress proven experiments into tightly controlled production scope to test conditions staging does not reproduce
• Game days simulate major failures (AZ down, DB failover) and practice recovery
• DR drills validate backup/restore procedures and compare actual recovery time and data-loss window with RTO and RPO
• Recovery exercises may support applicable control evidence; organizational mappings and assessor expectations determine the required scope, frequency, and evidence
• Untested failover mechanisms carry unknown risk—choose exercises and cadence from impact, recovery objectives, change rate, prior findings, safeguards, and environment