mateusz@systems ~/book/geographic-distribution $ cat section.md

Geographic Distribution

Distributing systems across multiple datacenters provides resilience against datacenter-level failures (natural disasters, power grid outages, region-wide network partitions). But geographic distribution introduces latency, consistency challenges, and operational complexity. This section explores multi-datacenter patterns and the trade-offs involved.

# CAP Theorem in Practice

The CAP theorem states that in a distributed system, you can have at most two of three properties: Consistency, Availability, Partition tolerance. In practice, network partitions happen, so you must choose between consistency and availability during partitions.

Consistency (all nodes see same data) Partition Tolerance (network failures) Availability (responds even during failures) During a network partition, choose one: CP: Consistent but not available (reject writes until partition heals) AP: Available but not consistent (allow writes, resolve conflicts later)
The CAP triangle — Consistency, Partition Tolerance, and Availability trade off pairwise. Network partitions are unavoidable, so during one, a system must pick CP (reject writes, stay consistent) or AP (accept writes, resolve conflicts later).

In practice: CAP is often oversimplified. Real systems offer tunable consistency (e.g., choose consistency level per operation) and degrade gracefully rather than binary CP/AP choice.

# Multi-Datacenter Patterns

Pattern 1: Primary-Secondary (Active-Passive)

In one active-passive pattern, a single primary accepts writes while read-only standbys replicate asynchronously. This example uses cascading replication: DC2 follows DC1, and DC3 follows DC2.

DC1 (us-east) Primary DB Writes go here Async replication DC2 (us-west) Secondary DB (read-only) Async replication DC3 (eu-west) Secondary DB (read-only) Failover example: detect and fence DC1, then promote a suitable standby
Illustrative primary-standby topology — DC1 accepts writes; read-only DC2 follows DC1, and read-only DC3 follows DC2 through cascading asynchronous replication. Failover requires external detection, fencing, standby selection and promotion, and client redirection.

Pros: A single-writer topology can simplify write-conflict handling, and readable standbys can offload reads. Consistency and durability still depend on the replication and commit settings.

Cons: Distant clients incur latency when writes route to one primary. Promotion requires failure detection, fencing, standby selection, and client redirection; standby write capacity is reserved, although hot standbys can serve reads.

Use case: Traditional databases, compliance requirements for single source of truth, applications not designed for multi-master.

Pattern 2: Multi-Master (Active-Active)

Multiple datacenters accept writes simultaneously. Replication is bidirectional.

DC1 (us-east) Primary DB DC2 (us-west) Primary DB Conflict resolution (last-write-wins, CRDT, etc.) Both DCs accept writes Replicate changes to each other Handle conflicts when the same key is updated
Multi-master (active-active) — DC1 and DC2 both accept writes and replicate directly to each other; conflicting updates to the same key are resolved (last-write-wins, CRDTs, etc.) and fed back to both sides.

Pros: Low write latency (users write to nearest DC), instant failover (other DCs already active), better resource utilization.

Cons: Complex conflict resolution, eventual consistency (replicas lag), risk of data conflicts (same key modified in both DCs).

Use case: Global applications (social media, e-commerce), NoSQL databases (Cassandra, DynamoDB Global Tables), CRDTs (conflict-free replicated data types).

Pattern 3: Sharded by Geography

Partition records by a policy-defined home region. Each shard has one authoritative owner for writes.

Group A write Group B write Group C write route to owner route to owner route to owner Region A site Owns group A records Region B site Owns group B records Region C site Owns group C records Cross-region query: reaches the remote owning DC
Policy-based geographic sharding — each group has one authoritative regional owner for writes, and a query needing another group's records communicates with that remote owner. The ownership rule does not describe where replicas or other copies may exist.

Pros: A single authoritative owner reduces concurrent cross-region writers for a shard and avoids cross-region coordination for writes issued near that owner. It can support a residency policy when the rest of the data lifecycle is configured consistently.

Cons: Queries spanning regions add network latency, moving or failing over a shard's ownership requires controlled data-and-routing changes, and unequal regional demand can create uneven load. Conflicts can still arise from an unsafe ownership transition, stale routing, or writes to replicas that were not kept read-only.

Policy scope: A requirement might assign a class of records to an approved home region. The design must separately account for replicas, backups, analytics, logs, support access, failover destinations, and cross-region queries. Geographic sharding can help implement such a policy, but the topology alone is not a legal or regulatory guarantee.

# Replication Strategies

Synchronous Cross-DC Replication

DC1 (us-east) DC2 (us-west) Write to local DB Replicate to DC2 (wait for ACK) DC2 confirms write Return success to client Latency: RTT to remote DC (50-200ms typical)
Synchronous cross-DC replication — DC1 writes locally, replicates to DC2, and waits for DC2's acknowledgment before returning success to the client. Latency includes a full round trip to the remote DC.

Pros: Strong consistency (both DCs have data before commit), zero data loss on DC failure.

Cons: High latency (cross-DC RTT added to every write), availability risk (if DC2 unreachable, writes fail).

Example: Google Spanner (synchronous replication with Paxos across DCs).

Asynchronous Cross-DC Replication

DC1 (us-east) DC2 (us-west) Write to local DB Return success to client after local commit Replicate to DC2 (background, async) Latency: local write latency only (1-10ms)
Asynchronous cross-DC replication — DC1 commits locally and returns success without waiting for DC2; replication to DC2 happens afterward in the background, so latency reflects only the local write.

Pros: Low latency (no cross-DC wait), availability (DC2 down doesn't block writes).

Cons: Eventual consistency (DC2 lags behind DC1), data loss risk (if DC1 crashes before replication).

Example: MySQL async replication, AWS RDS cross-region read replicas.

Comparison Table

Characteristic Sync Async
Write Latency High
(50-200ms)		 Low
(1-10ms)
Consistency Strong
(immediate)		 Eventual
(seconds lag)
Data Loss Risk None Possible
(if DC crashes)
Availability Lower
(needs remote DC)		 Higher
(DC independent)
Use Case Financial, transactional Social media, analytics, logs

# Real-World Examples

Netflix Multi-Region

In a 2013 engineering post, Netflix described active-active service delivery across two US AWS Regions, us-east-1 and us-west-2. Normal US traffic was split roughly 50/50, and Netflix tested shifting most of that traffic to us-west-2; eu-west-1 was shown as a separate existing deployment.

Strategy in that 2013 design: Keep services on the user call path stateless, keep their dependencies local to each Region, replicate Cassandra data asynchronously, and steer users with a combination of UltraDNS and Route 53.

Google Spanner

Spanner provides globally distributed SQL database with strong consistency. Uses Paxos for synchronous replication across datacenters and TrueTime API for distributed transactions.

Trade-off: In a dual- or multi-Region instance, read-write transactions are processed in the leader Region and writes commit through a voting majority, so latency depends on client, leader, and quorum placement. A strong read can use a non-leader replica but may need a timestamp round trip to the leader; a sufficiently stale read can often be served by a nearby caught-up replica without that round trip.

AWS S3 Cross-Region Replication

S3 Cross-Region Replication asynchronously copies eligible objects under configured rules from a source bucket to one or more destination buckets in other Regions. Ordinary CRR has no seconds-to-minutes guarantee; with S3 Replication Time Control enabled, AWS says most objects replicate in seconds and designs RTC to replicate 99.99% within 15 minutes. Its SLA commitment is 99.9% within 15 minutes for each replication Region pair during a billing month.

Use cases: Keep regional copies for applicable compliance requirements, place copies nearer global readers, or support a multi-Region recovery/failover design whose replication, access controls, routing, and failback behavior are configured and tested.

# Consistency vs Latency Trade-Off

The fundamental trade-off: Strong consistency requires coordination (slow). Low latency requires accepting stale reads or eventual consistency.

Property Strong Consistency (Sync Replication) Eventual Consistency (Async Replication)
Read Behavior Every read sees the latest write Reads may see stale data temporarily
Latency High (cross-DC coordination) Low (local reads/writes)
Use When Correctness is critical (banking, inventory) Availability and speed matter more (social feed, recommendations)

# Key Takeaways

  • CAP theorem: choose consistency or availability during network partitions (partitions will happen)
  • Multi-DC patterns: primary-secondary (simple, slower failover), multi-master (complex, fast failover), geographically sharded (single-owner partitions with cross-region query and migration costs)
  • A geographic owner can reduce concurrent cross-region writers and support a residency policy, but every copy and access path must be covered; sharding alone is not a legal guarantee
  • Sync replication: strong consistency, high latency (50-200ms writes)
  • Async replication: eventual consistency, low latency, data loss risk
  • Netflix: a documented 2013 two-US-Region active-active design with asynchronous Cassandra replication
  • Spanner: globally distributed SQL with strong consistency (synchronous Paxos)
  • Choose based on requirements: financial/transactional needs sync, social/content can use async