GPU Computing
Training memory: why mixed-precision Adam needs about 16 bytes per parameter
Weights, gradients and optimizer states — and how tensor, pipeline and ZeRO/FSDP sharding divide them across GPUs.
GIRIVON Engineering · Updated 19 Nov 2026 · 1 min read
Full training (or full fine-tuning) with the Adam optimizer in mixed precision keeps, for every parameter:
| State | Precision | Bytes |
|---|---|---|
| Weights used in forward/backward | BF16 | 2 |
| Gradients | BF16 | 2 |
| FP32 master copy of weights | FP32 | 4 |
| Adam first moment | FP32 | 4 |
| Adam second moment | FP32 | 4 |
| Total | 16 |
This is the accounting used in the ZeRO paper (Rajbhandari et al., 2020). An 8-billion-parameter model therefore needs about 128 GB of training state, and a 70-billion-parameter model about 1.12 TB — before activations.
Activations
Activation memory depends on sequence length, micro-batch size and architecture. For a transformer layer, Korthikanti et al. (2022) give about s · b · h · (34 + 5 · a · s / h) bytes without recomputation, where s is sequence length, b micro-batch, h hidden size and a attention heads. Fused attention kernels avoid materialising the attention matrix, and activation checkpointing stores only each layer's input (about 2 · s · b · h bytes) at the cost of recomputation.
Dividing the state across GPUs
- Tensor parallelism (TP) splits each layer's matrices across GPUs inside a server.
- Pipeline parallelism (PP) places different layers on different GPUs.
- ZeRO / FSDP data parallelism shards optimizer states (stage 1), then gradients (stage 2), then weights (stage 3) across the data-parallel group.
With ZeRO-3 across 64 GPUs, an 8B model's 128 GB of state becomes about 2 GB per GPU — leaving room for activations.
GPU memory does not simply add up
Eight 80 GB GPUs are not one 640 GB device. Each GPU still holds its own activations, buffers and CUDA context, and sharding adds communication over NVLink or the network. That is why the interconnect matters as much as the memory.
Model your own configuration in the GPU Memory (VRAM) Calculator.