LLM Inference Calculator
Estimate how much accelerator memory a large language model needs for inference — weights, KV cache and runtime overhead — how many GPUs are required to hold it, and the theoretical upper bound on token-generation speed set by memory bandwidth.
Results
- Model weights
- 16.06 GBcalculated
- KV cache per token
- 131.07 KBcalculated
- KV cache (8,192 tokens × 8 sequences)
- 8.59 GBcalculated
- Runtime overhead
- 2.46 GBestimated
- Total memory required
- 27.11 GBcalculated
- GPUs required (NVIDIA H100 SXM5 80GB)80 GB each, 90% usable
- 1calculated
- Memory per GPU
- 27.11 GBcalculated
- Decode speed per sequenceMemory-bandwidth upper bound at full context
- 95.1 tokens/stheoretical
- Aggregate decode throughput
- 761 tokens/stheoretical
- Time to first token2,000-token prompt
- 65 msestimated
Memory figures use decimal units (1 GB = 10⁹ bytes). Usable GPU memory is lower than the marketed capacity.
Decode speed is an upper bound: real serving stacks reach a fraction of it, depending on kernels, batching and scheduling.
What it calculates
- Model weight memory
- KV cache per token and in total
- Total inference memory
- Minimum GPU count
- Decode tokens per second (upper bound)
- Time to first token (estimate)
Worked example
Using the default inputs below; change any input above to recalculate.
| Total parameters | 8.03 billion |
|---|---|
| Active parameters per token | 8.03 billion |
| Transformer layers | 32 |
| Key/value heads | 8 |
| Head dimension | 128 |
| Weight precision | BF16 (2 bytes) |
| KV cache precision | BF16 (2 bytes) |
| Context length per sequence | 8192 tokens |
| Concurrent sequences (batch) | 8 |
| Prompt length (for time to first token) | 2000 tokens |
| Runtime overhead | 10 % |
| GPU | NVIDIA H100 SXM5 80GB |
| Usable fraction of GPU memory | 90 % |
| Achievable memory-bandwidth efficiency | 70 % |
| Model weights | 16.06 GB | calculated |
|---|---|---|
| KV cache per token | 131.07 KB | calculated |
| KV cache (8,192 tokens × 8 sequences) | 8.59 GB | calculated |
| Runtime overhead | 2.46 GB | estimated |
| Total memory required | 27.11 GB | calculated |
| GPUs required (NVIDIA H100 SXM5 80GB) | 1 | calculated |
| Memory per GPU | 27.11 GB | calculated |
| Decode speed per sequence | 95.1 tokens/s | theoretical |
| Aggregate decode throughput | 761 tokens/s | theoretical |
| Time to first token | 65 ms | estimated |
Methodology
Weights occupy parameters × bytes per parameter. The KV cache stores one key and one value vector per layer, per key/value head, for every token in every active sequence. Overhead covers activations, the CUDA context and framework buffers as a percentage you control.
The GPU count is the smallest number whose usable memory holds the total. Token generation (decode) is usually memory-bandwidth bound: each step reads the active weights once plus the KV cache of the batch. Dividing those bytes by aggregate bandwidth × an efficiency factor gives the time per step and an upper bound on tokens per second.
Prefill (processing the prompt) is compute bound: about 2 × active parameters floating-point operations per prompt token, divided by the GPUs' vendor-reported BF16 dense peak × a utilisation factor.
Formulas
- Weights
- M_w = P × b_w
- KV cache
- M_kv = 2 × L × H_kv × d_head × b_kv × context × sequences
- GPUs
- N = ⌈(M_w + M_kv) × (1 + overhead) ÷ (VRAM × usable)⌉
- Decode step
- t_step = (P_active × b_w + M_kv) ÷ (N × bandwidth × efficiency)
- Prefill
- t_prefill = 2 × P_active × prompt_tokens ÷ (N × peak_BF16 × utilisation)
Assumptions
- Worst case for decode: every sequence is at the full context length.
- INT4 is counted as about 4.5 bits per weight to include group-wise quantisation scales.
- GPU figures are vendor-reported datasheet values (dense, without sparsity).
Limitations
- Does not model paged-attention fragmentation, speculative decoding, prefix caching or continuous-batching schedulers.
- Tensor-parallel communication overhead is not modelled; multi-GPU throughput is optimistic.
- Multi-head latent attention and other KV-compression schemes need different KV formulas.
Frequently asked questions
How much GPU memory does a 70B model need?
In BF16 the weights alone are about 141 GB (70.6 billion parameters × 2 bytes) for Llama 3.1 70B, before KV cache and overhead — more than one 80 GB GPU. With 4-bit quantisation the weights fall to roughly 40 GB. Use the preset and set your context length and concurrency to see the full total.
Why does context length matter so much?
The KV cache grows linearly with context length and with the number of concurrent sequences. At long contexts and high concurrency it can exceed the size of the weights.
Can I add the memory of several GPUs together?
Only if the model is split across them with tensor or pipeline parallelism. Each GPU still needs its own working memory, and splitting adds communication over the GPU interconnect.