Network topology determines how data flows through a datacenter, how much bandwidth is available, and what happens when switches fail. The evolution from traditional 3-tier architectures to modern leaf-spine designs reflects changing traffic patterns and scale requirements.

# Traditional 3-Tier Architecture

The classic datacenter network has three layers: access, aggregation, and core.

Access Layer: Top of Rack (ToR) switches connect servers within a rack.

Aggregation Layer: Aggregates multiple ToR uplinks, provides Layer 2/3 boundary, often where VLANs terminate.

Core Layer: High-capacity switches that route traffic between aggregation blocks (pods) and provide connectivity to the internet or WAN.

Limitations of 3-Tier

• Oversubscription: Uplinks from access → aggregation → core create bottlenecks. Typical 4:1 or 8:1 oversubscription means not all servers can use full bandwidth simultaneously [1].
• East-West traffic bottleneck: Traffic between servers in different pods must hairpin through aggregation and core layers, adding latency and consuming bandwidth.
• Spanning Tree Protocol (STP): To prevent loops, STP blocks redundant links, wasting capacity. Active links carry all traffic while backup links sit idle.
• Scaling complexity: Adding capacity requires upgrading core switches (expensive, disruptive).

# Leaf-Spine Architecture

Many large datacenters use folded-Clos fabrics, often described as leaf-spine. A two-tier leaf-spine can offer a uniform path structure and can be built at 1:1, but real fabrics may be oversubscribed and larger deployments may use additional stages. Major cloud providers including Google [2], Meta (Facebook) [3], and Microsoft Azure [4] have adopted variants of this architecture for their datacenters.

In the two-tier fabric shown, each leaf connects to every spine and to its local servers; each spine connects to every leaf and not directly to servers. Larger Clos fabrics can add stages or planes, so these exact link rules are not universal.

Advantages of Leaf-Spine

• Provisioning: A chosen oversubscription ratio, including 1:1 where workload and budget justify it.
• Predictable latency: In the two-tier fabric shown, servers on different leafs traverse source leaf → one spine → destination leaf (two inter-switch links); same-leaf, multi-stage Clos, and inter-datacenter paths differ.
• ECMP (Equal-Cost Multi-Path): Multiple paths between leafs (via different spines) enable load balancing [5][6]. All links are active, no STP blocking.
• Horizontal scaling: Within available ports, cabling, and control-plane capacity, add a leaf to attach racks or add spine capacity and let routing reconverge over the new equal-cost paths. When switch radix or stage capacity is exhausted, expansion can require another fabric stage/pod or hardware replacement.
• Failure resilience: In the fully meshed two-tier example, after routing detects the failure and reconverges, removal of one spine leaves alternate paths through the remaining spines but reduces aggregate capacity; transient loss can occur, and multiple or correlated failures can still partition the fabric.

Comparison Table

# North-South vs East-West Traffic

Understanding traffic patterns is critical for network design.

North-South Traffic

North-South: Traffic between external clients (internet, branch offices) and datacenter servers. Typical in web applications: user requests come in (south), responses go out (north).

East-West Traffic

East-West: Traffic between servers within the datacenter. Microservices, database replication, distributed storage, and inter-service communication generate massive E/W traffic.

East-west traffic can dominate in distributed-service and storage-heavy environments, but the mix is workload- and provider-specific. In 2014, Cisco reported a Global Cloud Index estimate of 76% east-west, 17% north-south, and 7% inter-datacenter traffic [7]; treat those figures as historical context, not a current universal ratio. High internal-bandwidth demand is one reason large operators adopted Clos fabrics.

# Oversubscription Ratios

Oversubscription measures the ratio of downstream bandwidth to upstream bandwidth. If 48 servers with 10Gbps NICs connect to a ToR switch with only 40Gbps of uplinks, the oversubscription is 12:1 (480Gbps downstream vs 40Gbps upstream).

Why oversubscribe? Cost. Building a fully non-blocking network (1:1) is expensive. Most workloads don't need every server transmitting at full speed simultaneously.

When it matters: HPC workloads, distributed storage (e.g., Ceph, GPFS), and database clusters often require low oversubscription (2:1 or 1:1) to avoid bottlenecks. Web servers handling mostly external traffic can tolerate higher oversubscription (4:1 or more) [1].

# Layer 2 vs Layer 3 Switching

Layer 2 (L2): Switching based on MAC addresses. All devices in the same L2 domain (VLAN) can communicate directly. Broadcasts flood the entire domain.

Layer 3 (L3): Routing based on IP addresses. Each subnet is isolated; routers forward traffic between subnets. Broadcasts stay within subnets.

A common modern design uses an L3 routed underlay from the leaf uplinks through the fabric, limiting underlay broadcast and failure scope. Server-facing subnet boundaries are design-specific, and tenant L2 segments can still span leafs through overlays such as EVPN-VXLAN.

# References

# Key Takeaways

• Folded-Clos/leaf-spine fabrics are common in large datacenters; their path structure, oversubscription, and scale depend on the design
• East-west traffic can dominate in distributed-service and storage-heavy environments, but the mix is workload- and provider-specific
• Oversubscription is a cost trade-off; HPC and storage need low ratios, web apps can tolerate higher
• ECMP in leaf-spine enables all links to be active, unlike STP blocking in 3-tier
• L3 to the leaf reduces broadcast domains and simplifies network management