GPUs & accelerators
How much GPU memory does a local LLM need? A practical rule of thumb
Model weights, precision and the KV cache decide whether a model fits on your GPU. Here is the arithmetic, with worked examples for 8B and 70B models.
By GIRIVON Engineering · 21 November 2026 · 2 min read
The first question when choosing a GPU for local language models is simple: does the model fit in GPU memory? Two things take up most of the space — the weights and the KV cache.
1. Weights: parameters × bytes per parameter
| Precision | Bytes per parameter | Memory per billion parameters |
|---|---|---|
| FP16 / BF16 | 2 | about 2 GB |
| FP8 / INT8 | 1 | about 1 GB |
| 4-bit (e.g. GPTQ, AWQ) | 0.5 | about 0.5–0.6 GB (scales add a little) |
So an 8-billion-parameter model needs about 16 GB at FP16 and about 4.5 GB at 4-bit; a 70-billion-parameter model needs about 140 GB at FP16 and roughly 35–40 GB at 4-bit.
2. KV cache: grows with context length
Each token of context stores keys and values for every layer:
KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value
For a Llama-3-style 8B model (32 layers, 8 KV heads, head dimension 128) at FP16 that is 2 × 32 × 8 × 128 × 2 = 128 KiB per token — about 1 GiB for 8,000 tokens. A Llama-3-style 70B model (80 layers, 8 KV heads) needs 320 KiB per token, so a 32,000-token context adds about 10 GiB. Serving several users multiplies the cache by the number of concurrent sequences.
3. Leave headroom
Frameworks need working memory for activations, CUDA context and fragmentation. Plan for 10–20% above weights plus cache.
Worked examples
- 8B model, 4-bit, 8K context, one user: ~4.5 GB + ~1 GB + headroom → fits comfortably on a 12–16 GB GPU.
- 70B model, 4-bit, 32K context, one user: ~38 GB + ~10 GB + headroom → needs about 56–64 GB: two 32 GB GPUs or one 48–96 GB professional GPU.
- 70B model, FP16: ~140 GB of weights alone → multi-GPU server territory.
These are estimates — measured usage depends on the framework, quantisation format and batch size. To size a specific model, context and concurrency, use the LLM inference calculator and the GPU VRAM calculator, then compare cards in the GPU price list.
- #llm
- #vram
- #inference