Routing protocols determine how traffic flows through networks. Understanding when BGP vs OSPF matters, how VLANs isolate traffic, and why MTU mismatches cause mysterious failures is essential for troubleshooting production issues.

# BGP: Border Gateway Protocol

BGP is the protocol that routes traffic across the internet and between autonomous systems (AS). In datacenters, BGP is used to:

• Exchange routes with internet service providers (ISPs)
• Route traffic between geographically distributed datacenters
• Implement anycast (same IP advertised from multiple locations)

Key characteristic: BGP is a path-vector protocol. Routers advertise reachable networks and the AS path to reach them. BGP selects routes based on policy (AS path length, local preference, etc.), not just shortest path.

When BGP matters: Multi-datacenter setups, connections to ISPs, and failover between internet uplinks. BGP changes can take minutes to propagate globally.

Real-World Outage: Facebook October 2021

On October 4, 2021, Facebook, Instagram, and WhatsApp were unreachable globally for about six hours. During backbone maintenance, a command intended to assess available capacity unintentionally disconnected all of Facebook's backbone connections. A bug in the tool that audited such commands failed to stop it. (Outage report)

What happened: The backbone disconnection isolated Facebook's datacenters. DNS locations could no longer communicate with them, so those locations declared themselves unhealthy and withdrew their own BGP advertisements. The DNS servers remained operational but became unreachable, and the internet could no longer resolve Facebook's services.

Why recovery was difficult: The same network and DNS failure disrupted the normal tools and both primary and out-of-band network access used to diagnose the outage. Engineers had to reach the datacenters onsite before restoring the backbone, then return services gradually to manage recovery load.

Lesson: Protect high-blast-radius network commands with tested safeguards, staged execution where practical, and validation. Map dependencies such as DNS, and preserve a recovery path that does not depend on the network or services being repaired.

# OSPF: Open Shortest Path First

OSPF is an interior gateway protocol (IGP) used within a single administrative domain (like a datacenter or campus network). It's the most common protocol for routing within datacenters.

Key characteristic: OSPF is a link-state protocol. Routers exchange topology information (who's connected to whom) and independently calculate shortest paths using Dijkstra's algorithm.

Advantages: Fast convergence (sub-second to seconds after topology changes), supports ECMP (multiple equal-cost paths), hierarchical design (areas reduce overhead).

When OSPF matters: Internal datacenter routing, especially in leaf-spine architectures where you need fast failover and ECMP load balancing.

BGP vs OSPF Comparison

# VLANs: Virtual LANs

VLANs logically segment a physical network into isolated broadcast domains. Devices on VLAN 10 can't communicate with VLAN 20 without a router, even if they're on the same physical switch.

Why use VLANs:

• Layer-2 traffic separation (security still depends on trunk, routing, ACL, and firewall policy)
• Reduce broadcast domain size (broadcasts only flood within VLAN)
• Simplify network management (logical grouping vs physical cabling)

VLAN tagging: 802.1Q tags (4 bytes) added to Ethernet frames identify which VLAN the frame belongs to. Trunk ports carry multiple VLANs; access ports belong to one VLAN.

# MTU and Jumbo Frames

MTU (Maximum Transmission Unit) is the largest packet size that can be transmitted without fragmentation. The conventional Ethernet IP MTU is 1500 bytes. “Jumbo frame” has no single standardized size, although an IP MTU around 9000 bytes is common.

Why jumbo frames matter: For storage and HPC workloads, larger frames reduce CPU overhead (fewer packets to process for the same data). A 1GB file transfer requires ~700,000 packets at 1500-byte MTU, but only ~120,000 packets at 9000-byte MTU.

MTU mismatch problems: If an IPv4 packet is larger than a downstream link's MTU, a router may fragment it when DF is clear; with DF set, the router drops it and returns ICMP “Fragmentation Needed.” IPv6 routers never fragment and instead return ICMPv6 “Packet Too Big.” If those signals are blocked, small probes can work while larger transfers such as scp or NFS writes stall.

Common scenario: Hybrid cloud setups where on-prem datacenter uses jumbo frames (9000) but cloud VPN connection has MTU 1500. Large database backups fail or perform poorly due to MTU mismatch.

Debugging MTU issues: On Linux over IPv4, use ping -M do -s 8972 to test a 9000-byte IP packet (8972 bytes of ICMP data + 8-byte ICMP header + 20-byte IPv4 header). If that fails but ping -M do -s 1472, which produces a 1500-byte IPv4 packet, succeeds, the path does not carry a 9000-byte packet end to end.

# LAN vs WAN Characteristics

Understanding LAN (Local Area Network) vs WAN (Wide Area Network) differences helps you design systems that perform well across both.

Design implication: Systems optimized for LAN (e.g., chatty protocols with many round-trips) perform poorly over WAN. Use batching, caching, and async replication for WAN links.

# Key Takeaways

• Backbone failures can trigger DNS locations to withdraw BGP advertisements; safeguard high-blast-radius commands and recovery access (Facebook 2021)
• OSPF handles internal datacenter routing with fast convergence and ECMP support
• VLANs provide logical network segmentation for traffic separation and broadcast-domain reduction
• MTU mismatches cause mysterious failures; jumbo frames improve storage/HPC performance
• LAN vs WAN characteristics (latency, bandwidth, packet loss) require different system designs
• Understanding protocols helps troubleshoot outages: "Is this BGP, routing, or application-level?"