Skip to content
GIRIVON
Infrastructure+
Systems+
Components+
Computing Devices+
Solutions+
Compute Tools+
Knowledge+

AI Cluster Interconnect Calculator

Estimate the time to synchronise gradients each training step with a hierarchical ring all-reduce — inside nodes over the GPU interconnect and between nodes over the network — and the resulting scaling efficiency.

2 for BF16 gradients.

NVLink on H100 SXM is 900 GB/s bidirectional (vendor).

Results

Gradient size
16 GB
Intra-node all-reduce
62.2 ms
Inter-node all-reduce
75 ms
Exposed communication per step
68.6 ms
Scaling efficiency
95.6%

Bandwidth-optimal ring all-reduce moves 2(n−1)/n of the data per participant. Latency terms, protocol efficiency and congestion lower real performance.

What it calculates

  • Gradient size
  • Intra-node all-reduce time
  • Inter-node all-reduce time
  • Scaling efficiency

Worked example

Using the default inputs below; change any input above to recalculate.

Inputs
Model parameters8 billion
Gradient bytes per parameter2
GPUs per node8
Nodes16
Intra-node bandwidth per GPU (one direction)450 GB/s
Network bandwidth per GPU400 GbE
Compute time per step1.5 s
Communication overlapped with compute50 %
Results
Gradient size16 GBcalculated
Intra-node all-reduce62.2 mstheoretical
Inter-node all-reduce75 mstheoretical
Exposed communication per step68.6 msestimated
Scaling efficiency95.6%estimated

Methodology

Hierarchical all-reduce: a ring inside each node over the GPU interconnect, then a ring across nodes in which each GPU's NIC carries 1/g of the gradient. Each ring moves 2(n−1)/n × data ÷ bandwidth (Patarasuk & Yuan, 2009). Exposed time is the non-overlapped part; efficiency = compute ÷ (compute + exposed).

Formulas

Ring
t = 2(n − 1)/n × S ÷ B
Efficiency
E = t_compute ÷ (t_compute + t_comm × (1 − overlap))

Assumptions

  • Pure data parallelism; one NIC per GPU.

Limitations

  • Tensor and pipeline parallel traffic is not included.
  • Real collectives achieve a fraction of line rate.