Comparisons
Ethernet vs InfiniBand for AI clusters
How the fabric affects gradient synchronisation, and the factors that decide between RDMA Ethernet and InfiniBand.
GIRIVON Engineering · Updated 19 Nov 2026 · 1 min read
Distributed training synchronises gradients on every step. The time that takes — and how much of it overlaps with compute — decides how well a cluster scales.
The bandwidth arithmetic
A bandwidth-optimal ring all-reduce moves 2 (n − 1) / n × S bytes per participant, where S is the gradient size (Patarasuk & Yuan, 2009). For an 8-billion-parameter model with BF16 gradients, S ≈ 16 GB.
On 16 servers of 8 GPUs, with one 400 Gb/s (50 GB/s) network port per GPU and a hierarchical all-reduce:
- Inter-node: 2 × 15/16 × (16 GB ÷ 8) ÷ 50 GB/s ≈ 0.075 s
- Intra-node over NVLink (450 GB/s per direction on H100 SXM): 2 × 7/8 × 16 GB ÷ 450 GB/s ≈ 0.062 s
If a step's compute takes 1.5 s and half of the communication overlaps, scaling efficiency is about 1.5 ÷ (1.5 + 0.069) ≈ 96%. Halve the network bandwidth and the inter-node term doubles.
Where the technologies differ
Both deliver 400 Gb/s-class links today. The differences are in transport and operations:
- InfiniBand: credit-based lossless transport, very low latency, mature congestion control and in-network collective offload. Popular for large training clusters and tightly coupled MPI codes.
- RDMA over Converged Ethernet (RoCE v2): RDMA on standard Ethernet switching, integrates with existing IP networks and tooling, suits multi-tenant and mixed environments — but needs careful lossless and congestion configuration at scale.
How to choose
Start from the workload (training versus inference versus storage), scale, latency sensitivity, your team's operational experience and integration needs. The Ethernet vs InfiniBand Advisor scores these factors transparently, and the AI Cluster Interconnect Calculator quantifies the bandwidth effect for your model.