mateusz@systems ~/book/control-vs-data-plane $ cat section.md

Control Plane vs Data Plane Separation

The control plane manages configuration, coordination, and metadata. The data plane handles the system's primary user-facing work and data processing. Separating their failure domains can reduce direct failure propagation, but shared dependencies can still couple them; a data plane continues through control-plane impairment only when it retains valid state and avoids recovery-time control-plane dependencies.

# What Are Control and Data Planes?

Control Plane

The control plane manages system state, configuration, and coordination. Depending on the architecture, examples include:

  • Cluster membership and leader election
  • Configuration distribution
  • Health information used for placement or failover decisions
  • API endpoints for admin operations (create resource, delete resource)
  • Metadata management (file locations, routing tables)

Characteristics: Control-plane operations are often lower-volume and less frequent than data-plane operations, but request-path dependencies and change rates are architecture-specific.

Data Plane

The data plane handles user requests and actual data processing. Examples:

  • Serving web requests
  • Reading/writing data to databases or storage
  • Packet forwarding in routers
  • Processing user transactions

Characteristics: Traffic is often high-volume and latency-sensitive; its availability target follows the workload's requirements.

Example: Network Router

Control Plane:
    - BGP protocol (exchange routing information)
    - OSPF protocol (calculate shortest paths)
    - Update routing table when topology changes
    - Typically lower packet rates than forwarding; updates vary with topology and policy

Data Plane:
    - Forward packets based on routing table
    - Potentially very high packet rates
    - Must be fast, latency-sensitive
    - Uses routing table built by control plane

# Why Separate Them?

Blast radius isolation: When the planes have separate processes, resource budgets, and dependencies, a control-plane bug is less likely to crash the data plane, and data-plane overload is less likely to starve the control plane. Shared dependencies can still couple them.

Different scaling needs: Data-plane request capacity often scales by adding workers. Consensus-backed control-plane membership has different scaling and quorum constraints, although other control-plane components can also scale horizontally.

Graceful degradation: If the data plane retains valid state and has no live control-plane dependency for an operation, that operation can continue while configuration updates stop.

Coupled (Bad) Separated + retained state Control + Data Plane same process Control Plane fails crash Entire Service Down total outage Data Plane continues if state valid → config updates blocked → allowed traffic may continue
In the coupled example, a shared-process crash takes down the service. In the separated example, a statically stable data plane can keep selected operations running from valid retained configuration while control-plane updates are unavailable.

# Real-World Example: AWS S3 Outage (February 2017)

On February 28, 2017, an operational command removed more Amazon S3 subsystem capacity in Northern Virginia than intended, making S3 unable to service requests while the affected subsystems restarted.

What happened: During work on the S3 billing system, an authorized operator supplied an incorrect input to an established command. The command removed more capacity than intended from the index and placement subsystems. 

Maintenance action:
    - Incorrect command input removed excessive index and placement capacity

Request and recovery effects:
    - The index subsystem was needed for GET, LIST, PUT, and DELETE requests
    - S3 could not service requests while the index and placement subsystems restarted
    - Recovery included metadata safety checks that took longer than expected at the service's then-current scale

Architectural caution: This incident is not evidence that a separated data plane kept reads available. AWS reported that the index subsystem was required to service the main object APIs and that S3 could not service requests while the affected subsystems restarted.

Recovery complexity: The large Regions had not completely restarted the index and placement subsystems for years, and both had grown substantially. Restarting them and completing metadata safety checks took longer than expected. Capacity-removal safeguards and restart recovery therefore belonged in the failure model.

Lesson: Bound the effect of administrative commands, validate high-impact inputs, add safeguards that prevent excessive capacity removal, and exercise recovery from a large subsystem restart.

# Real-World Example: AWS DynamoDB DNS Outage (October 2025)

On October 19-20, 2025, DynamoDB's DNS management system published an empty regional-endpoint record in US-EAST-1. DynamoDB and several other services then experienced distinct impairment and recovery periods.

Initiating mechanism: Two independent DNS Enactors raced while applying and cleaning up plans. A delayed Enactor applied a stale plan after a newer plan had been used; cleanup by another Enactor then deleted the old plan that had become active again. That left an empty DNS record and inconsistent state that required manual operator correction.

AWS described the Enactor as deliberately designed with minimal dependencies.

1. Newer DNS plan is active independent Enactors reconcile endpoint state 2. Delayed Enactor applies stale plan the older plan becomes active again 3. Cleanup deletes the active old plan cleanup believes that plan is no longer active Empty record · inconsistent state · manual repair
The DNS failure came from a stale-plan and cleanup race between independent Enactors. Deleting the plan that had become active again left the endpoint in an inconsistent state that automated repair could not resolve, so operators corrected it manually.

Downstream mechanisms: After the DynamoDB endpoint recovered, EC2's Droplet Workflow Manager (DWFM) had to re-establish leases amid a large backlog and entered congestive collapse. Network configuration propagation accumulated its own backlog. Some Network Load Balancers removed healthy capacity because of health-check behavior during the event. Lambda experienced several dependency-specific impacts; ECS, EKS, and Fargate experienced container launch failures and cluster-scaling delays, and the last container-service impacts recovered by 2:20 PM.

Distinct impact and recovery mechanisms DynamoDB endpoint DNS repair and connection recovery EC2 DWFM lease backlog · congestive collapse Network propagation configuration backlog Network Load Balancing health checks removed healthy capacity Lambda several dependency-specific impacts ECS · EKS · Fargate launch failures · scaling delays Related event, not one uniform linear dependency chain
The regional event propagated through several mechanisms with different timelines. DynamoDB endpoint recovery did not by itself clear the EC2 DWFM backlog, network propagation, NLB health-check effects, Lambda dependencies, or ECS, EKS, and Fargate launch and cluster-scaling impacts; the last container-service effects recovered by 2:20 PM.

Lesson: Protect active plans from stale reconcilers and cleanup, and test recovery states in which queued work and feedback loops persist after the initiating dependency is restored. Map service-specific dependencies instead of assuming one linear cascade.

# Metadata vs Data Separation

A specific case of control/data plane separation: metadata (where data is) vs actual data.

Distributed Storage Example

Metadata Service (Control Plane):
    - Tracks which nodes store which blocks
    - File: /home/user/data.txt
        Block 1: stored on Node A, Node B (replicas)
        Block 2: stored on Node C, Node D (replicas)

Data Service (Data Plane):
    - Stores actual blocks
    - Serves read/write requests
    - Clients query the metadata service for block locations, then contact data nodes

Failure behavior (depends on implementation and cached state):
    Metadata service unavailable: Fresh namespace and block-location operations fail;
                                  reads may continue only for locations already cached
                                  and reachable data nodes.
    Data node unavailable: Blocks with no reachable replica are unavailable;
                           replicated blocks can be retried from another known replica.

Best practice: Keep metadata and data on separate servers. Metadata service should be highly available (Raft/Paxos consensus). Data service should be horizontally scalable.

# Design Patterns for Separation

Pattern 1: Cached Control Plane State

A data-plane node can retain validated configuration locally so selected operations do not require live control-plane access. The retained state needs an explicit validity window and fail-safe behavior. 

Data Plane Node (illustrative):
    - Fetches and validates configuration from the control plane on a chosen refresh interval
    - Retains the last-known-valid configuration in memory or durable local storage
    - Serves only operations that remain safe under that configuration
    - If the control plane is unreachable, continues while the retained state is still valid

Graceful degradation: Configuration updates stop; safe, pre-authorized traffic may continue

Pattern 2: Separate Resource Pools

Run control and data planes in separate resource pools where the failure model warrants it. Dedicated quotas, VMs, or hosts isolate direct CPU and memory contention, but shared infrastructure and dependencies can remain. 

Control Plane: 3 dedicated servers (small, for coordination)
Data Plane: 100 servers (large, for handling user traffic)

Separate CPU quotas can keep a data-plane spike from consuming the control plane's allocation
Separate memory limits can contain a control-plane leak within its pool

Pattern 3: Read-Only Data Plane

Control plane handles writes (slow, needs coordination). Data plane handles reads (fast, cacheable).

Client write read Control Plane validate · coordinate Data Plane cached · fast reads replicate control plane down → writes fail, reads continue
Write requests route through the control plane, which validates, coordinates, and replicates to the data plane. Read requests go straight to the data plane's cache, skipping the control plane entirely. If the control plane fails, writes stop but reads keep working.

# Key Takeaways

  • Control plane manages config/metadata; data plane handles user requests
  • Separate their failure domains where practical; isolation reduces direct failure propagation, but shared dependencies can still couple them
  • Retain validated last-known control-plane state where it is safe, and fail closed when stale state would be unsafe
  • S3 2017: Incorrect command input removed excessive index and placement capacity; bounded commands, safeguards, and restart testing are the supported lessons
  • DynamoDB 2025: A stale-plan and cleanup race corrupted active DNS state, followed by several distinct backlogs and dependency-specific recovery mechanisms
  • Reconciliation and cleanup paths must protect active state, and recovery tests must include backlogs and feedback loops
  • Metadata and data should be separated (different servers, different scaling properties)