AI Cluster Interconnect Calculator
Estimate the time to synchronise gradients each training step with a hierarchical ring all-reduce — inside nodes over the GPU interconnect and between nodes over the network — and the resulting scaling efficiency.
Results
- Gradient size
- 16 GBcalculated
- Intra-node all-reduce
- 62.2 mstheoretical
- Inter-node all-reduce
- 75 mstheoretical
- Exposed communication per step
- 68.6 msestimated
- Scaling efficiency
- 95.6%estimated
Bandwidth-optimal ring all-reduce moves 2(n−1)/n of the data per participant. Latency terms, protocol efficiency and congestion lower real performance.
What it calculates
- Gradient size
- Intra-node all-reduce time
- Inter-node all-reduce time
- Scaling efficiency
Worked example
Using the default inputs below; change any input above to recalculate.
| Model parameters | 8 billion |
|---|---|
| Gradient bytes per parameter | 2 |
| GPUs per node | 8 |
| Nodes | 16 |
| Intra-node bandwidth per GPU (one direction) | 450 GB/s |
| Network bandwidth per GPU | 400 GbE |
| Compute time per step | 1.5 s |
| Communication overlapped with compute | 50 % |
| Gradient size | 16 GB | calculated |
|---|---|---|
| Intra-node all-reduce | 62.2 ms | theoretical |
| Inter-node all-reduce | 75 ms | theoretical |
| Exposed communication per step | 68.6 ms | estimated |
| Scaling efficiency | 95.6% | estimated |
Methodology
Hierarchical all-reduce: a ring inside each node over the GPU interconnect, then a ring across nodes in which each GPU's NIC carries 1/g of the gradient. Each ring moves 2(n−1)/n × data ÷ bandwidth (Patarasuk & Yuan, 2009). Exposed time is the non-overlapped part; efficiency = compute ÷ (compute + exposed).
Formulas
- Ring
- t = 2(n − 1)/n × S ÷ B
- Efficiency
- E = t_compute ÷ (t_compute + t_comm × (1 − overlap))
Assumptions
- Pure data parallelism; one NIC per GPU.
Limitations
- Tensor and pipeline parallel traffic is not included.
- Real collectives achieve a fraction of line rate.