Skip to content
GIRIVON
Infrastructure+
Systems+
Components+
Computing Devices+
Solutions+
Compute Tools+
Knowledge+

GPUs & accelerators

How much GPU memory does a local LLM need? A practical rule of thumb

Model weights, precision and the KV cache decide whether a model fits on your GPU. Here is the arithmetic, with worked examples for 8B and 70B models.

By GIRIVON Engineering · 21 November 2026 · 2 min read

The first question when choosing a GPU for local language models is simple: does the model fit in GPU memory? Two things take up most of the space — the weights and the KV cache.

1. Weights: parameters × bytes per parameter

Precision Bytes per parameter Memory per billion parameters
FP16 / BF16 2 about 2 GB
FP8 / INT8 1 about 1 GB
4-bit (e.g. GPTQ, AWQ) 0.5 about 0.5–0.6 GB (scales add a little)

So an 8-billion-parameter model needs about 16 GB at FP16 and about 4.5 GB at 4-bit; a 70-billion-parameter model needs about 140 GB at FP16 and roughly 35–40 GB at 4-bit.

2. KV cache: grows with context length

Each token of context stores keys and values for every layer:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value

For a Llama-3-style 8B model (32 layers, 8 KV heads, head dimension 128) at FP16 that is 2 × 32 × 8 × 128 × 2 = 128 KiB per token — about 1 GiB for 8,000 tokens. A Llama-3-style 70B model (80 layers, 8 KV heads) needs 320 KiB per token, so a 32,000-token context adds about 10 GiB. Serving several users multiplies the cache by the number of concurrent sequences.

3. Leave headroom

Frameworks need working memory for activations, CUDA context and fragmentation. Plan for 10–20% above weights plus cache.

Worked examples

  • 8B model, 4-bit, 8K context, one user: ~4.5 GB + ~1 GB + headroom → fits comfortably on a 12–16 GB GPU.
  • 70B model, 4-bit, 32K context, one user: ~38 GB + ~10 GB + headroom → needs about 56–64 GB: two 32 GB GPUs or one 48–96 GB professional GPU.
  • 70B model, FP16: ~140 GB of weights alone → multi-GPU server territory.

These are estimates — measured usage depends on the framework, quantisation format and batch size. To size a specific model, context and concurrency, use the LLM inference calculator and the GPU VRAM calculator, then compare cards in the GPU price list.

Keep reading