Network Protocols
Routing protocols determine how traffic flows through networks. Understanding when BGP vs OSPF matters, how VLANs isolate traffic, and why MTU mismatches cause mysterious failures is essential for troubleshooting production issues.
# BGP: Border Gateway Protocol
BGP is the protocol that routes traffic across the internet and between autonomous systems (AS). In datacenters, BGP is used to:
- Exchange routes with internet service providers (ISPs)
- Route traffic between geographically distributed datacenters
- Implement anycast (same IP advertised from multiple locations)
Key characteristic: BGP is a path-vector protocol. Routers advertise reachable networks and the AS path to reach them. BGP selects routes based on policy (AS path length, local preference, etc.), not just shortest path.
When BGP matters: Multi-datacenter setups, connections to ISPs, and failover between internet uplinks. BGP changes can take minutes to propagate globally.
Real-World Outage: Facebook October 2021
On October 4, 2021, Facebook (and Instagram, WhatsApp) became unreachable globally for ~6 hours. The root cause: a BGP configuration change withdrew all of Facebook's route advertisements.
Before:
Internet <--> [BGP: Facebook AS announces 157.240.0.0/16] <--> Facebook Datacenters
Routes exist, traffic flows normally
After BGP withdrawal:
Internet <--> [BGP: No routes to Facebook IPs] <-X-> Facebook Datacenters
No one knows how to reach Facebook's IP addresses
What happened: A routine maintenance command accidentally withdrew all BGP route announcements. The internet's routers "forgot" how to reach Facebook's servers. Even Facebook's DNS servers became unreachable because they too were advertised via BGP.
Why DNS didn't help: DNS resolution requires network connectivity. If BGP routes are gone, you can't even query DNS servers to find Facebook's IPs. This is a control plane failure—the data (Facebook's servers) was fine, but routing information was lost.
Lesson: BGP is critical infrastructure. Misconfigurations can make your entire network vanish from the internet. BGP changes should have safeguards, staged rollouts, and automated validation.
# OSPF: Open Shortest Path First
OSPF is an interior gateway protocol (IGP) used within a single administrative domain (like a datacenter or campus network). It's the most common protocol for routing within datacenters.
Key characteristic: OSPF is a link-state protocol. Routers exchange topology information (who's connected to whom) and independently calculate shortest paths using Dijkstra's algorithm.
Advantages: Fast convergence (sub-second to seconds after topology changes), supports ECMP (multiple equal-cost paths), hierarchical design (areas reduce overhead).
When OSPF matters: Internal datacenter routing, especially in leaf-spine architectures where you need fast failover and ECMP load balancing.
BGP vs OSPF Comparison
+--------------------+------------------+----------------------+ | Characteristic | BGP | OSPF | +--------------------+------------------+----------------------+ | Scope | Inter-AS | Intra-AS (single | | | (between | domain) | | | organizations) | | +--------------------+------------------+----------------------+ | Algorithm | Path-vector | Link-state | | | (policy-based) | (shortest path) | +--------------------+------------------+----------------------+ | Convergence Speed | Slow (minutes) | Fast (sub-second to | | | | seconds) | +--------------------+------------------+----------------------+ | Scalability | Scales globally | Scales to thousands | | | (internet-wide) | of routers (with | | | | areas) | +--------------------+------------------+----------------------+ | Use Case | Internet routing,| Datacenter routing, | | | multi-DC, anycast| campus networks | +--------------------+------------------+----------------------+ | Configuration | Complex, policy- | Simpler, mostly | | | heavy | automatic | +--------------------+------------------+----------------------+
# VLANs: Virtual LANs
VLANs logically segment a physical network into isolated broadcast domains. Devices on VLAN 10 can't communicate with VLAN 20 without a router, even if they're on the same physical switch.
Physical Switch (single hardware)
|
+-- VLAN 10 (Production) [Server A] [Server B]
|
+-- VLAN 20 (Development) [Server C] [Server D]
|
+-- VLAN 30 (Management) [BMC] [PDU]
Server A (VLAN 10) cannot talk to Server C (VLAN 20) without routing
Why use VLANs:
- Security isolation (separate prod/dev/mgmt traffic)
- Reduce broadcast domain size (broadcasts only flood within VLAN)
- Simplify network management (logical grouping vs physical cabling)
VLAN tagging: 802.1Q tags (4 bytes) added to Ethernet frames identify which VLAN the frame belongs to. Trunk ports carry multiple VLANs; access ports belong to one VLAN.
# MTU and Jumbo Frames
MTU (Maximum Transmission Unit) is the largest packet size that can be transmitted without fragmentation. Standard Ethernet MTU is 1500 bytes. Jumbo frames increase this to 9000 bytes.
Why jumbo frames matter: For storage and HPC workloads, larger frames reduce CPU overhead (fewer packets to process for the same data). A 1GB file transfer requires ~700,000 packets at 1500-byte MTU, but only ~120,000 packets at 9000-byte MTU.
MTU mismatch problems: If one device sends 9000-byte frames but an intermediate router has MTU 1500, packets get fragmented (slow) or dropped (if "Don't Fragment" bit is set). Symptoms: small packets (ping) work fine, but large transfers (scp, NFS writes) mysteriously fail or stall.
Common scenario: Hybrid cloud setups where on-prem datacenter uses jumbo frames (9000) but cloud VPN connection has MTU 1500. Large database backups fail or perform poorly due to MTU mismatch.
Debugging MTU issues: Use ping -M do -s 8972 (Linux) to test path MTU.
If ping with 9000-byte payload fails but 1472-byte succeeds, you've found an MTU bottleneck.
# LAN vs WAN Characteristics
Understanding LAN (Local Area Network) vs WAN (Wide Area Network) differences helps you design systems that perform well across both.
+--------------------+-----------------+--------------------+ | Characteristic | LAN | WAN | +--------------------+-----------------+--------------------+ | Latency | Sub-millisecond | 10-100ms+ | | | (< 1ms typical) | (geographic) | +--------------------+-----------------+--------------------+ | Bandwidth | 1-100 Gbps | 1-10 Gbps | | | (cheap) | (expensive) | +--------------------+-----------------+--------------------+ | Packet Loss | Near zero | 0.1-1% typical | | | (< 0.01%) | | +--------------------+-----------------+--------------------+ | MTU | Often 9000 | Often 1500 | | | (jumbo frames) | (standard) | +--------------------+-----------------+--------------------+ | Protocols | OSPF, Layer 2 | BGP, MPLS | +--------------------+-----------------+--------------------+ | Cost | Low per Gbps | High per Gbps | +--------------------+-----------------+--------------------+
Design implication: Systems optimized for LAN (e.g., chatty protocols with many round-trips) perform poorly over WAN. Use batching, caching, and async replication for WAN links.
# Key Takeaways
- BGP routes traffic between datacenters/internet; misconfiguration can make networks vanish (Facebook 2021)
- OSPF handles internal datacenter routing with fast convergence and ECMP support
- VLANs provide logical network segmentation for security and broadcast domain reduction
- MTU mismatches cause mysterious failures; jumbo frames improve storage/HPC performance
- LAN vs WAN characteristics (latency, bandwidth, packet loss) require different system designs
- Understanding protocols helps troubleshoot outages: "Is this BGP, routing, or application-level?"