Perfect reliability is impossible. Systems will fail. Graceful degradation means your system continues providing reduced functionality rather than failing completely. This section covers patterns for handling failures gracefully: retries, circuit breakers, timeouts, and bulkheading.

# Retry Strategies

Transient failures (network blips, temporary overload) can be resolved by retrying. But naive retries make things worse.

Naive Retry (Bad)

def call_with_retries(call_service, max_retries):
    for attempt in range(max_retries + 1):  # first attempt plus retries
        try:
            return call_service()
        except TransientError:
            if attempt == max_retries:
                raise
            # no delay: retry immediately

Problem: If the service is overloaded, immediate retries amplify its load, and many clients retrying together turn into a retry storm (thundering herd).

Exponential Backoff (Better)

import time

def call_with_backoff(call_service, max_retries,
                      base_delay=0.1, max_delay=60.0):  # seconds, illustrative
    for attempt in range(max_retries + 1):  # first attempt plus retries
        try:
            return call_service()
        except TransientError:
            if attempt == max_retries:
                raise
            time.sleep(min(base_delay * 2 ** attempt, max_delay))

With these illustrative values the delays are 100 ms, 200 ms, 400 ms, 800 ms, 1.6 s, and so on, giving the service time to recover and spreading retries over time.

Exponential Backoff with Jitter

import random
import time

def call_with_jittered_backoff(call_service, max_retries,
                               base_delay=0.1, max_delay=60.0):  # seconds, illustrative
    for attempt in range(max_retries + 1):  # first attempt plus retries
        try:
            return call_service()
        except TransientError:
            if attempt == max_retries:
                raise
            delay = min(base_delay * 2 ** attempt, max_delay)
            time.sleep(delay * random.uniform(0.5, 1.0))

This bounded jitter samples between 50% and 100% of each backoff delay. If many clients fail together, the randomized delays spread their retries instead of synchronizing them.

When to retry: Retry only failures the API classifies as transient and only when the operation is known to be retry-safe—for example, an idempotent operation or a request carrying a server-supported idempotency token. Use a bounded retry budget, honor Retry-After, and remember that a timeout can mean the server completed the operation but the response was lost.

When not to retry blindly: Do not automatically repeat a non-idempotent or ambiguously completed operation without a deduplication contract. Most permanent validation and authorization errors will not improve on retry, but handle each status according to the API contract rather than rejecting the entire 4xx class.

# Circuit Breaker Pattern

Circuit breakers prevent calling a failing service repeatedly. If a service is down, stop calling it for a period (fail fast), then test recovery.

Circuit Breaker States

Circuit Breaker in Python

import time

class CircuitOpenError(Exception):
    pass

class CircuitBreaker:
    # Single-threaded sketch. A concurrent version must lock the state and admit
    # only a bounded number of half-open probes, not every caller that arrives.
    def __init__(self, failure_threshold=5, success_threshold=2, timeout=60.0):
        self.failure_threshold = failure_threshold  # open after 5 consecutive failures
        self.success_threshold = success_threshold  # close after 2 successes in half-open
        self.timeout = timeout                      # seconds before testing recovery
        self.state = "CLOSED"
        self.failure_count = 0
        self.success_count = 0
        self.opened_at = 0.0

    def call(self, operation):
        if self.state == "OPEN":
            if time.monotonic() - self.opened_at < self.timeout:
                raise CircuitOpenError("service unavailable")
            self.state = "HALF_OPEN"  # timeout elapsed: let test calls through
            self.success_count = 0

        try:
            result = operation()
        except Exception:
            self._on_failure()
            raise
        self._on_success()
        return result

    def _on_success(self):
        if self.state == "HALF_OPEN":
            self.success_count += 1
            if self.success_count >= self.success_threshold:
                self.state = "CLOSED"
                self.failure_count = 0
        else:
            self.failure_count = 0  # reset on success

    def _on_failure(self):
        if self.state == "HALF_OPEN":
            self._open()  # a failed test call reopens the circuit
        else:
            self.failure_count += 1
            if self.failure_count >= self.failure_threshold:
                self._open()

    def _open(self):
        self.state = "OPEN"
        self.opened_at = time.monotonic()


# Usage
breaker = CircuitBreaker()

def get_user(user_id):
    try:
        return breaker.call(lambda: fetch_user_data(user_id))
    except CircuitOpenError:
        return cached_user_data(user_id)  # fail fast: serve cached data

Why circuit breakers help: While open, a breaker rejects or redirects calls before attempting the dependency, which can reduce resource use and response latency and reduce pressure on the dependency. Recovery still depends on the underlying service, and the caller needs operation-specific error or fallback behavior. Circuit-state changes also provide useful monitoring signals.

# Timeout Configuration

Timeouts prevent waiting forever for failed operations. But setting them wrong causes problems.

Too Long

Timeout: 60 seconds
Service is down, takes 60s to timeout
User waits 60s for error
Thread/connection blocked for 60s
If many requests, exhausts thread pool --> cascade failure

Too Short

Timeout: 100ms
Service normally responds in 90ms
Network spike: 150ms response
Timeout fires, request aborted
Service actually succeeded, but client thinks it failed
Retries increase load, making problem worse

Right-Sizing Timeouts

Guideline: Set timeout to p99 latency + buffer. If 99% of requests complete in 500ms, set timeout to 1000ms (2x p99).

Monitor timeout rates: If >1% of requests timing out, either increase timeout or fix underlying slowness.

# Health Checks That Actually Work

Health checks determine if a service instance is healthy. Poor health checks cause false positives (marking healthy instance unhealthy) or false negatives (missing actual failures).

Bad Health Check

def health_status():  # HTTP status code for GET /health
    return 200          # OK, whatever state the dependencies are in

It reports success even when the database connection is down, the disk is full, memory is exhausted, or a critical dependency is unavailable. The load balancer keeps routing traffic to the instance, and users see errors.

Good Health Check

def health_status():  # HTTP status code for GET /health
    checks = [check_database_connection, check_dependency_connectivity,
              check_disk_space, check_memory_available]
    if all(check() for check in checks):
        return 200  # OK
    return 503      # Service Unavailable

The load balancer removes instances that fail the check from its pool, so traffic routes only to healthy instances.

Deep vs Shallow Health Checks:

• Shallow (liveness): Is process alive? Can it accept requests? Fast, frequent (every 1s).
• Deep (readiness): Can it handle requests successfully? Check dependencies. Slower, less frequent (every 10s).

Avoid: Health checks that are too expensive (calling all dependencies adds load). Health checks that modify state (don't write to DB during health check). Health checks that always succeed (useless) or always fail (removes all instances).

# Bulkheading: Isolate Failures

Bulkheads borrow their name from ship compartments that limit the spread of water. In software, they partition selected resources so one failure is less likely to cascade through those resources.

Thread Pool Bulkheading

Resource isolation: Separate resource budgets—such as concurrency, thread, or connection pools—where the failure model warrants it. Bulkheads preserve only the resources they partition; shared CPU, memory, queues, networks, and dependencies can still carry failures across partitions.

# Key Takeaways

• Bounded retries with exponential backoff and jitter can reduce synchronized retry load
• An open circuit breaker can fail fast or invoke a safe fallback while reducing calls to an impaired dependency
• Circuit breaker states: CLOSED (normal), OPEN (failing), HALF_OPEN (testing recovery)
• Timeouts should be 1.5-2x p99 latency; too short causes false failures, too long blocks resources
• Health checks must validate dependencies, not just "process alive"
• Bulkheading can isolate selected resource pools and reduce cascading-failure risk, while shared resources still need protection
• Fail fast with circuit breakers + short timeouts, then return cached/default data