AI Computing
How much GPU memory does an LLM need?
Inference memory is weights plus KV cache plus overhead. Here is the arithmetic, with worked numbers for Llama 3.1 8B and 70B.
GIRIVON Engineering · Updated 19 Nov 2026 · 2 min read
Serving a large language model needs GPU memory for three things: the model weights, the KV cache for every active sequence, and runtime overhead (activations, the CUDA context and framework buffers).
1. Weights
Weights take parameters × bytes per parameter:
| Precision | Bytes per parameter | Llama 3.1 8B (8.03B) | Llama 3.1 70B (70.6B) |
|---|---|---|---|
| BF16 / FP16 | 2 | 16.1 GB | 141.2 GB |
| FP8 / INT8 | 1 | 8.0 GB | 70.6 GB |
| INT4 (≈4.5 bits with group scales) | 0.5625 | 4.5 GB | 39.7 GB |
2. KV cache
During generation the model stores a key and a value vector per layer, per key/value head, for every token in context:
KV bytes = 2 × layers × KV heads × head dimension × bytes × tokens × sequences
Llama 3.1 8B has 32 layers, 8 KV heads (grouped-query attention) and a head dimension of 128. In BF16 that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB) per token — about 1.07 GB for one 8,192-token sequence. Llama 3.1 70B has 80 layers, so the cache is 320 KiB per token, or about 2.7 GB per 8K-token sequence. Thirty-two concurrent 8K sequences on the 70B model need roughly 86 GB of KV cache — more than half the size of the BF16 weights.
3. Overhead and usable memory
Add roughly 5–15% for activations and buffers, and remember that not all of a GPU's marketed memory is usable. A practical planning rule is to fill no more than about 90% of each GPU.
Putting it together
- Llama 3.1 8B, BF16, 8 × 8K sequences: 16.1 GB + 8.6 GB + overhead ≈ 27 GB — one 48 GB or 80 GB GPU.
- Llama 3.1 70B, BF16: 141 GB of weights alone exceeds any single 80 GB GPU; it needs two or more GPUs with tensor parallelism, or one 141 GB+ accelerator.
- Llama 3.1 70B, INT4: about 40 GB of weights fits a single 48 GB or 80 GB GPU, with accuracy trade-offs you should evaluate.
Why speed depends on memory bandwidth
Each generated token reads the active weights once per batch step, so decode speed is usually bounded by memory bandwidth: tokens per second per sequence ≲ bandwidth ÷ bytes read per step. Quantisation helps twice — it shrinks memory and raises that upper bound.
Use the LLM Inference Calculator to run these numbers for your model, context length and concurrency.