Perfect reliability is impossible. Systems will fail. Graceful degradation means your system continues providing reduced functionality rather than failing completely. This section covers patterns for handling failures gracefully: retries, circuit breakers, timeouts, and bulkheading.

# Retry Strategies

Transient failures (network blips, temporary overload) can be resolved by retrying. But naive retries make things worse.

Naive Retry (Bad)

Pseudocode:
    attempt = 0
    while attempt < MAX_RETRIES:
        result = call_service()
        if result.success:
            return result
        attempt += 1
        # No delay, retry immediately

Problem: If service is overloaded, immediate retries amplify load
        Many clients can retry together --> retry storm / thundering herd

Exponential Backoff (Better)

Pseudocode:
    attempt = 0
    base_delay = 100ms  # Illustrative initial delay
    max_delay = 60s     # Illustrative cap

    while attempt < MAX_RETRIES:
        result = call_service()
        if result.success:
            return result

        attempt += 1
        delay = min(base_delay * pow(2, attempt - 1), max_delay)
        sleep(delay)

    raise ServiceUnavailableError()

Illustrative delays: 100ms, 200ms, 400ms, 800ms, 1600ms, ...
Gives service time to recover, spreads retries over time

Exponential Backoff with Jitter

Pseudocode:
    attempt = 0
    base_delay = 100ms  # Illustrative
    max_delay = 60s      # Illustrative

    while attempt < MAX_RETRIES:
        result = call_service()
        if result.success:
            return result

        attempt += 1
        delay = min(base_delay * pow(2, attempt - 1), max_delay)
        jittered_delay = delay * (0.5 + random(0, 0.5))
        sleep(jittered_delay)

    raise ServiceUnavailableError()

This bounded jitter samples between 50% and 100% of each backoff delay, reducing synchronized retries
If many clients fail together, randomized delays can spread their retry attempts

When to retry: Retry only failures the API classifies as transient and only when the operation is known to be retry-safe—for example, an idempotent operation or a request carrying a server-supported idempotency token. Use a bounded retry budget, honor Retry-After, and remember that a timeout can mean the server completed the operation but the response was lost.

When not to retry blindly: Do not automatically repeat a non-idempotent or ambiguously completed operation without a deduplication contract. Most permanent validation and authorization errors will not improve on retry, but handle each status according to the API contract rather than rejecting the entire 4xx class.

# Circuit Breaker Pattern

Circuit breakers prevent calling a failing service repeatedly. If a service is down, stop calling it for a period (fail fast), then test recovery.

Circuit Breaker States

Circuit Breaker Pseudocode

class CircuitBreaker:
    state = CLOSED
    failure_count = 0
    success_count = 0
    failure_threshold = 5      # Open after 5 failures
    success_threshold = 2      # Close after 2 successes in half-open
    timeout = 60s              # Time before testing recovery
    last_failure_time = null

    function call(operation):
        if state == OPEN:
            if now() - last_failure_time > timeout:
                state = HALF_OPEN
                success_count = 0
            else:
                raise CircuitOpenError("Service unavailable")

        if state == HALF_OPEN:
            # Allow limited requests through to test recovery
            try:
                result = operation()
                success_count += 1

                if success_count >= success_threshold:
                    state = CLOSED
                    failure_count = 0

                return result
            except Exception as e:
                state = OPEN
                last_failure_time = now()
                raise e

        if state == CLOSED:
            try:
                result = operation()
                failure_count = 0  # Reset on success
                return result
            except Exception as e:
                failure_count += 1
                last_failure_time = now()

                if failure_count >= failure_threshold:
                    state = OPEN

                raise e

# Usage:
circuit = CircuitBreaker()
try:
    result = circuit.call(lambda: fetch_user_data(user_id))
except CircuitOpenError:
    # Fail fast, return cached data or error to user
    return cached_user_data(user_id)

Why circuit breakers help: While open, a breaker rejects or redirects calls before attempting the dependency, which can reduce resource use and response latency and reduce pressure on the dependency. Recovery still depends on the underlying service, and the caller needs operation-specific error or fallback behavior. Circuit-state changes also provide useful monitoring signals.

# Timeout Configuration

Timeouts prevent waiting forever for failed operations. But setting them wrong causes problems.

Too Long

Timeout: 60 seconds
Service is down, takes 60s to timeout
User waits 60s for error
Thread/connection blocked for 60s
If many requests, exhausts thread pool --> cascade failure

Too Short

Timeout: 100ms
Service normally responds in 90ms
Network spike: 150ms response
Timeout fires, request aborted
Service actually succeeded, but client thinks it failed
Retries increase load, making problem worse

Right-Sizing Timeouts

Guideline: Set timeout to p99 latency + buffer. If 99% of requests complete in 500ms, set timeout to 1000ms (2x p99).

Monitor timeout rates: If >1% of requests timing out, either increase timeout or fix underlying slowness.

# Health Checks That Actually Work

Health checks determine if a service instance is healthy. Poor health checks cause false positives (marking healthy instance unhealthy) or false negatives (missing actual failures).

Bad Health Check

GET /health
    return HTTP 200 OK

Always returns success, even if:
    - Database connection is down
    - Disk is full
    - Memory exhausted
    - Critical dependency unavailable

Load balancer thinks instance is healthy, routes traffic
Requests fail, users see errors

Good Health Check

GET /health
    check_database_connection()
    check_dependency_connectivity()
    check_disk_space()
    check_memory_available()

    if all_checks_pass:
        return HTTP 200 OK
    else:
        return HTTP 503 Service Unavailable

Load balancer removes unhealthy instances from pool
Traffic routes only to healthy instances

Deep vs Shallow Health Checks:

• Shallow (liveness): Is process alive? Can it accept requests? Fast, frequent (every 1s).
• Deep (readiness): Can it handle requests successfully? Check dependencies. Slower, less frequent (every 10s).

Avoid: Health checks that are too expensive (calling all dependencies adds load). Health checks that modify state (don't write to DB during health check). Health checks that always succeed (useless) or always fail (removes all instances).

# Bulkheading: Isolate Failures

Bulkheads borrow their name from ship compartments that limit the spread of water. In software, they partition selected resources so one failure is less likely to cascade through those resources.

Thread Pool Bulkheading

Resource isolation: Separate resource budgets—such as concurrency, thread, or connection pools—where the failure model warrants it. Bulkheads preserve only the resources they partition; shared CPU, memory, queues, networks, and dependencies can still carry failures across partitions.

# Key Takeaways

• Bounded retries with exponential backoff and jitter can reduce synchronized retry load
• An open circuit breaker can fail fast or invoke a safe fallback while reducing calls to an impaired dependency
• Circuit breaker states: CLOSED (normal), OPEN (failing), HALF_OPEN (testing recovery)
• Timeouts should be 1.5-2x p99 latency; too short causes false failures, too long blocks resources
• Health checks must validate dependencies, not just "process alive"
• Bulkheading can isolate selected resource pools and reduce cascading-failure risk, while shared resources still need protection
• Fail fast with circuit breakers + short timeouts, then return cached/default data