Skip to content
GIRIVON
Infrastructure+
Systems+
Components+
Computing Devices+
Solutions+
Compute Tools+
Knowledge+

GPU Memory (VRAM) Calculator

Break down the GPU memory a transformer model needs per GPU: weights, gradients, optimizer states, activations and KV cache, for inference, full fine-tuning or pre-training, LoRA and QLoRA, under tensor, pipeline and ZeRO/FSDP data parallelism.

Workload
Model
Batch
LoRA
Parallelism
GPU

Used only when GPU is set to Custom.

Used only when GPU is set to Custom.

Used only when GPU is set to Custom.

Used only when GPU is set to Custom.

Results

Weights per GPU
2.01 GB
Gradients per GPU
2.01 GB
Optimizer states per GPU
12.04 GB
Activations per GPUKorthikanti et al. (2022)
18.25 GB
Logits for the loss
2.1 GB
Framework & CUDA context
2 GB
Total per GPU
38.41 GB
NVIDIA H100 SXM5 80GB capacity
80 GB
Verdict
Fits (≤95% of capacity)
GPUs in this configurationTP 1 × PP 1 × DP 8
8

GPU memory does not pool automatically. Eight 80 GB GPUs are not one 640 GB device: the model must be explicitly partitioned (tensor, pipeline or sharded data parallelism), and every GPU keeps its own activations, buffers and CUDA context.

Pipeline parallelism keeps activations for several in-flight micro-batches on early stages; this calculator shows a single micro-batch.

What it calculates

  • Weights per GPU
  • Gradients per GPU
  • Optimizer states per GPU
  • Activations per GPU
  • KV cache per GPU
  • Total per GPU and fit verdict
  • Total GPUs in the configuration

Worked example

Using the default inputs below; change any input above to recalculate.

Inputs
WorkloadFull training / fine-tuning (mixed-precision Adam)
Parameters8.03 billion
Layers32
Hidden size4096
Attention heads32
Key/value heads8
Head dimension128
Feed-forward size14336
Vocabulary size128256
Sequence length4096 tokens
Micro-batch size per GPU1
Weight precision (inference / LoRA base)BF16 (2 bytes)
KV cache precision (inference)BF16 (2 bytes)
Activation checkpointingNo
Results
Weights per GPU2.01 GBcalculated
Gradients per GPU2.01 GBcalculated
Optimizer states per GPU12.04 GBcalculated
Activations per GPU18.25 GBestimated
Logits for the loss2.1 GBestimated
Framework & CUDA context2 GBestimated
Total per GPU38.41 GBcalculated
NVIDIA H100 SXM5 80GB capacity80 GBVendor-reported
VerdictFits (≤95% of capacity)calculated
GPUs in this configuration8calculated

Methodology

Mixed-precision training with Adam keeps BF16 weights (2 bytes/parameter) and gradients (2 bytes) plus FP32 master weights and two FP32 moments (12 bytes): about 16 bytes per parameter (Rajbhandari et al., 2020). Tensor and pipeline parallelism divide all of these by TP × PP; ZeRO stage 1 additionally shards optimizer states across the data-parallel group, stage 2 also gradients, and stage 3 (equivalent to FSDP) also weights.

Activation memory per transformer layer follows Korthikanti et al. (2022): s·b·h·(10 + 24/t) bytes with tensor-parallel degree t, plus 5·a·s²·b/t bytes for attention scores when attention is not fused. Full activation checkpointing stores only each layer's input (2·s·b·h bytes) and recomputes the rest. The output logits for the loss are counted in FP32.

LoRA freezes the base model and trains low-rank adapters of rank r on the chosen projection matrices; only adapters have gradients and optimizer states. QLoRA stores the base in 4-bit NormalFloat with double quantisation, about 4.13 bits per parameter (Dettmers et al., 2023). Inference memory is weights plus KV cache.

Formulas

Training state
M = (2 + 2 + 12) × N ÷ (TP × PP), then ÷ DP for sharded parts
Activations / layer
A = s·b·h·(10 + 24/t) [+ 5·a·s²·b/t without fused attention]
Checkpointed
A = 2·s·b·h per layer
LoRA adapter
r × (d_in + d_out) per adapted matrix
KV cache
2 × L × H_kv × d_head × b_kv × s × batch

Assumptions

  • Standard decoder-only transformer.
  • Optimizer is Adam/AdamW with FP32 states.
  • QLoRA base ≈ 0.516 bytes/parameter.

Limitations

  • Activation formula was derived for GPT-style blocks; SwiGLU MLPs and other variants differ somewhat.
  • Memory fragmentation and temporary buffers can add several gigabytes.
  • Does not model CPU or NVMe offloading.

Frequently asked questions

Can I fine-tune a 70B model on one GPU?

Full fine-tuning needs roughly 16 bytes per parameter — over a terabyte for 70B — so it requires many GPUs. QLoRA can fit a 70B base in about 36 GB for weights plus activations and adapters, which is why it is commonly run on a single 48–80 GB GPU.

Which precision formats are supported?

FP32, BF16, FP16, FP8, INT8 and INT4 for weights and KV cache in inference; training assumes mixed precision (BF16 compute with FP32 optimizer states).