Skip to content
GIRIVON
Infrastructure+
Systems+
Components+
Computing Devices+
Solutions+
Compute Tools+
Knowledge+

GPU Computing

Training memory: why mixed-precision Adam needs about 16 bytes per parameter

Weights, gradients and optimizer states — and how tensor, pipeline and ZeRO/FSDP sharding divide them across GPUs.

GIRIVON Engineering · Updated 19 Nov 2026 · 1 min read

Full training (or full fine-tuning) with the Adam optimizer in mixed precision keeps, for every parameter:

State Precision Bytes
Weights used in forward/backward BF16 2
Gradients BF16 2
FP32 master copy of weights FP32 4
Adam first moment FP32 4
Adam second moment FP32 4
Total 16

This is the accounting used in the ZeRO paper (Rajbhandari et al., 2020). An 8-billion-parameter model therefore needs about 128 GB of training state, and a 70-billion-parameter model about 1.12 TB — before activations.

Activations

Activation memory depends on sequence length, micro-batch size and architecture. For a transformer layer, Korthikanti et al. (2022) give about s · b · h · (34 + 5 · a · s / h) bytes without recomputation, where s is sequence length, b micro-batch, h hidden size and a attention heads. Fused attention kernels avoid materialising the attention matrix, and activation checkpointing stores only each layer's input (about 2 · s · b · h bytes) at the cost of recomputation.

Dividing the state across GPUs

  • Tensor parallelism (TP) splits each layer's matrices across GPUs inside a server.
  • Pipeline parallelism (PP) places different layers on different GPUs.
  • ZeRO / FSDP data parallelism shards optimizer states (stage 1), then gradients (stage 2), then weights (stage 3) across the data-parallel group.

With ZeRO-3 across 64 GPUs, an 8B model's 128 GB of state becomes about 2 GB per GPU — leaving room for activations.

GPU memory does not simply add up

Eight 80 GB GPUs are not one 640 GB device. Each GPU still holds its own activations, buffers and CUDA context, and sharding adds communication over NVLink or the network. That is why the interconnect matters as much as the memory.

Model your own configuration in the GPU Memory (VRAM) Calculator.