High-Performance Computing (HPC) workloads—scientific simulations, machine learning training, large-scale data analytics—demand ultra-low latency and high bandwidth. Traditional TCP/IP networking introduces overhead (CPU cycles, kernel context switches, memory copies) that becomes a bottleneck. RDMA (Remote Direct Memory Access) bypasses these limitations, enabling microsecond latencies and multi-hundred gigabit throughput.

# Why RDMA Matters

A conventional buffered TCP/IP send commonly involves a user-to-kernel copy and CPU protocol processing:

RDMA benefits:

• Zero-copy data path: The RNIC can transfer data from registered application memory without an intermediate kernel buffer
• Kernel bypass: After connection and memory-registration setup, application data can use a kernel-bypass path
• Lower CPU usage: The RNIC offloads data movement and transport work, leaving the CPU to post work and handle completions
• Low latency: 1-2 microseconds typical (vs 10-100µs for TCP)
• High bandwidth: Current adapters offer link rates through 800 Gbps, depending on transport and hardware

When RDMA matters: Distributed storage (Ceph with RBD, GPFS, WekaFS), MPI applications (parallel scientific computing), machine learning training (multi-GPU clusters with NCCL), low-latency databases (RAMCloud, FaRM).

# RDMA Technologies Comparison

Three major RDMA technologies exist, each with different transport layers and use cases.

# InfiniBand

InfiniBand (IB) is a purpose-built high-performance network fabric. It uses dedicated IB switches, cables, and NICs (called HCAs - Host Channel Adapters).

Architecture: One deployment pattern separates Ethernet management traffic from an InfiniBand compute/storage fabric; adapter and port counts, and whether compute and storage share a fabric, depend on topology and redundancy requirements.

Speeds: EDR (100 Gbps), HDR (200 Gbps), NDR (400 Gbps), and XDR (800 Gbps) per port.

Advantages: Lowest latency, highest bandwidth, credit-based link flow control that prevents congestion-related receive-buffer overflow on each flow-controlled data virtual lane, proven in HPC for decades.

Disadvantages: Expensive (specialized switches/NICs), separate network to manage, not directly interoperable with Ethernet; IP can run over InfiniBand via IPoIB, with routing or gateway infrastructure required to reach external networks.

Common in: Top500 supercomputers, HPC clusters, high-end storage arrays (NetApp, Pure Storage backend).

# RoCE: RDMA over Converged Ethernet

RoCE brings RDMA to Ethernet networks, allowing RDMA over standard Ethernet switches (with caveats). Two versions exist:

RoCE v1: RDMA directly over Ethernet (Layer 2). Non-routable, limited to single subnet. Rarely used today.

RoCE v2: RDMA over UDP/IP (Layer 3). Routable across subnets, dominant version.

Lossless Ethernet Requirement

RDMA assumes lossless transport (no packet drops). Standard Ethernet drops packets when congested. RoCE requires Priority Flow Control (PFC) to pause senders and prevent drops.

PFC caveat: Misconfiguration or persistent congestion can cause head-of-line blocking: PFC pauses all flows sharing the affected priority on a link, while unpaused priorities can continue. It requires careful configuration and monitoring.

Advantages: Uses commodity Ethernet infrastructure, converged network (one fabric for RDMA and TCP/IP), lower cost than InfiniBand.

Disadvantages: Slightly higher latency than IB, requires PFC configuration, more complex to debug than IB.

Common in: Cloud storage backends (Azure Premium Storage), distributed databases, Kubernetes storage (Rook/Ceph).

# iWARP: RDMA over TCP

iWARP runs RDMA over TCP/IP, so it can traverse routed IP networks without requiring PFC-capable switches; endpoints still need iWARP-capable RNICs and normal TCP/IP reachability.

Advantages: Works across standard routed IP networks with iWARP-capable endpoints, no need for lossless-Ethernet switch configuration, easier to deploy over WAN.

Disadvantages: Highest latency of RDMA options (TCP overhead), lower performance, declining ecosystem (fewer vendors).

Use case: RDMA over WAN links, environments where lossless Ethernet isn't feasible. Less common today; RoCE v2 has largely displaced it.

# Network Fabric Design for HPC

HPC clusters use specialized network topologies to minimize latency and maximize bandwidth for parallel workloads.

Fat-Tree Topology

Used for: MPI applications requiring all-to-all communication, machine learning training with data parallelism.

Dragonfly Topology

Groups of switches fully connected internally, sparse connections between groups. Optimizes for cost while maintaining low diameter (few hops between any two nodes).

Used for: Large-scale HPC (thousands to hundreds of thousands of nodes), balances cost and performance.

# Performance Considerations

Latency-sensitive workloads: Choose InfiniBand or RoCE v2. Avoid extra network hops (use leaf-spine, not 3-tier). Tune OS for low latency (disable CPU frequency scaling, pin threads).

Bandwidth-intensive workloads: Use RDMA with multi-path (ECMP or bonded interfaces). Ensure storage network is separate from management to avoid contention. Monitor for PFC PAUSE storms (in RoCE).

Mixed workloads: Separate compute and storage networks. Use InfiniBand or RoCE for storage backend, standard Ethernet for client access. Prevents storage I/O from impacting application latency.

# Key Takeaways

• RDMA bypasses kernel, achieves 1-2µs latency vs 10-100µs for TCP/IP
• InfiniBand offers lowest latency but requires dedicated fabric; RoCE v2 uses Ethernet with PFC
• RoCE requires lossless Ethernet (PFC)—poorly controlled PFC can cause head-of-line blocking within a traffic class
• iWARP runs over TCP/IP (higher latency) but works on any network; declining adoption
• HPC clusters use fat-tree or dragonfly topologies for non-blocking bandwidth
• Separate storage and compute networks to prevent I/O contention
• RDMA critical for distributed storage, ML training, and MPI-based parallel computing