GPU Memory (VRAM) Calculator
Break down the GPU memory a transformer model needs per GPU: weights, gradients, optimizer states, activations and KV cache, for inference, full fine-tuning or pre-training, LoRA and QLoRA, under tensor, pipeline and ZeRO/FSDP data parallelism.
Results
- Weights per GPU
- 2.01 GBcalculated
- Gradients per GPU
- 2.01 GBcalculated
- Optimizer states per GPU
- 12.04 GBcalculated
- Activations per GPUKorthikanti et al. (2022)
- 18.25 GBestimated
- Logits for the loss
- 2.1 GBestimated
- Framework & CUDA context
- 2 GBestimated
- Total per GPU
- 38.41 GBcalculated
- NVIDIA H100 SXM5 80GB capacity
- 80 GBVendor-reported
- Verdict
- Fits (≤95% of capacity)calculated
- GPUs in this configurationTP 1 × PP 1 × DP 8
- 8calculated
GPU memory does not pool automatically. Eight 80 GB GPUs are not one 640 GB device: the model must be explicitly partitioned (tensor, pipeline or sharded data parallelism), and every GPU keeps its own activations, buffers and CUDA context.
Pipeline parallelism keeps activations for several in-flight micro-batches on early stages; this calculator shows a single micro-batch.
What it calculates
- Weights per GPU
- Gradients per GPU
- Optimizer states per GPU
- Activations per GPU
- KV cache per GPU
- Total per GPU and fit verdict
- Total GPUs in the configuration
Worked example
Using the default inputs below; change any input above to recalculate.
| Workload | Full training / fine-tuning (mixed-precision Adam) |
|---|---|
| Parameters | 8.03 billion |
| Layers | 32 |
| Hidden size | 4096 |
| Attention heads | 32 |
| Key/value heads | 8 |
| Head dimension | 128 |
| Feed-forward size | 14336 |
| Vocabulary size | 128256 |
| Sequence length | 4096 tokens |
| Micro-batch size per GPU | 1 |
| Weight precision (inference / LoRA base) | BF16 (2 bytes) |
| KV cache precision (inference) | BF16 (2 bytes) |
| Activation checkpointing | No |
| Weights per GPU | 2.01 GB | calculated |
|---|---|---|
| Gradients per GPU | 2.01 GB | calculated |
| Optimizer states per GPU | 12.04 GB | calculated |
| Activations per GPU | 18.25 GB | estimated |
| Logits for the loss | 2.1 GB | estimated |
| Framework & CUDA context | 2 GB | estimated |
| Total per GPU | 38.41 GB | calculated |
| NVIDIA H100 SXM5 80GB capacity | 80 GB | Vendor-reported |
| Verdict | Fits (≤95% of capacity) | calculated |
| GPUs in this configuration | 8 | calculated |
Methodology
Mixed-precision training with Adam keeps BF16 weights (2 bytes/parameter) and gradients (2 bytes) plus FP32 master weights and two FP32 moments (12 bytes): about 16 bytes per parameter (Rajbhandari et al., 2020). Tensor and pipeline parallelism divide all of these by TP × PP; ZeRO stage 1 additionally shards optimizer states across the data-parallel group, stage 2 also gradients, and stage 3 (equivalent to FSDP) also weights.
Activation memory per transformer layer follows Korthikanti et al. (2022): s·b·h·(10 + 24/t) bytes with tensor-parallel degree t, plus 5·a·s²·b/t bytes for attention scores when attention is not fused. Full activation checkpointing stores only each layer's input (2·s·b·h bytes) and recomputes the rest. The output logits for the loss are counted in FP32.
LoRA freezes the base model and trains low-rank adapters of rank r on the chosen projection matrices; only adapters have gradients and optimizer states. QLoRA stores the base in 4-bit NormalFloat with double quantisation, about 4.13 bits per parameter (Dettmers et al., 2023). Inference memory is weights plus KV cache.
Formulas
- Training state
- M = (2 + 2 + 12) × N ÷ (TP × PP), then ÷ DP for sharded parts
- Activations / layer
- A = s·b·h·(10 + 24/t) [+ 5·a·s²·b/t without fused attention]
- Checkpointed
- A = 2·s·b·h per layer
- LoRA adapter
- r × (d_in + d_out) per adapted matrix
- KV cache
- 2 × L × H_kv × d_head × b_kv × s × batch
Assumptions
- Standard decoder-only transformer.
- Optimizer is Adam/AdamW with FP32 states.
- QLoRA base ≈ 0.516 bytes/parameter.
Limitations
- Activation formula was derived for GPT-style blocks; SwiGLU MLPs and other variants differ somewhat.
- Memory fragmentation and temporary buffers can add several gigabytes.
- Does not model CPU or NVMe offloading.
Frequently asked questions
Can I fine-tune a 70B model on one GPU?
Full fine-tuning needs roughly 16 bytes per parameter — over a terabyte for 70B — so it requires many GPUs. QLoRA can fit a 70B base in about 36 GB for weights plus activations and adapters, which is why it is commonly run on a single 48–80 GB GPU.
Which precision formats are supported?
FP32, BF16, FP16, FP8, INT8 and INT4 for weights and KV cache in inference; training assumes mixed precision (BF16 compute with FP32 optimizer states).