Skip to content
GIRIVON
Infrastructure+
Systems+
Components+
Computing Devices+
Solutions+
Compute Tools+
Knowledge+

LLM Inference Calculator

Estimate how much accelerator memory a large language model needs for inference — weights, KV cache and runtime overhead — how many GPUs are required to hold it, and the theoretical upper bound on token-generation speed set by memory bandwidth.

Model

Equal to total parameters for dense models; lower for mixture-of-experts.

Fewer than attention heads when the model uses grouped-query attention.

Precision
Serving

Activations, CUDA context and framework buffers.

GPU

Used only when GPU is set to Custom.

Used only when GPU is set to Custom.

Used only when GPU is set to Custom.

Used only when GPU is set to Custom.

Results

Model weights
16.06 GB
KV cache per token
131.07 KB
KV cache (8,192 tokens × 8 sequences)
8.59 GB
Runtime overhead
2.46 GB
Total memory required
27.11 GB
GPUs required (NVIDIA H100 SXM5 80GB)80 GB each, 90% usable
1
Memory per GPU
27.11 GB
Decode speed per sequenceMemory-bandwidth upper bound at full context
95.1 tokens/s
Aggregate decode throughput
761 tokens/s
Time to first token2,000-token prompt
65 ms

Memory figures use decimal units (1 GB = 10⁹ bytes). Usable GPU memory is lower than the marketed capacity.

Decode speed is an upper bound: real serving stacks reach a fraction of it, depending on kernels, batching and scheduling.

What it calculates

  • Model weight memory
  • KV cache per token and in total
  • Total inference memory
  • Minimum GPU count
  • Decode tokens per second (upper bound)
  • Time to first token (estimate)

Worked example

Using the default inputs below; change any input above to recalculate.

Inputs
Total parameters8.03 billion
Active parameters per token8.03 billion
Transformer layers32
Key/value heads8
Head dimension128
Weight precisionBF16 (2 bytes)
KV cache precisionBF16 (2 bytes)
Context length per sequence8192 tokens
Concurrent sequences (batch)8
Prompt length (for time to first token)2000 tokens
Runtime overhead10 %
GPUNVIDIA H100 SXM5 80GB
Usable fraction of GPU memory90 %
Achievable memory-bandwidth efficiency70 %
Results
Model weights16.06 GBcalculated
KV cache per token131.07 KBcalculated
KV cache (8,192 tokens × 8 sequences)8.59 GBcalculated
Runtime overhead2.46 GBestimated
Total memory required27.11 GBcalculated
GPUs required (NVIDIA H100 SXM5 80GB)1calculated
Memory per GPU27.11 GBcalculated
Decode speed per sequence95.1 tokens/stheoretical
Aggregate decode throughput761 tokens/stheoretical
Time to first token65 msestimated

Methodology

Weights occupy parameters × bytes per parameter. The KV cache stores one key and one value vector per layer, per key/value head, for every token in every active sequence. Overhead covers activations, the CUDA context and framework buffers as a percentage you control.

The GPU count is the smallest number whose usable memory holds the total. Token generation (decode) is usually memory-bandwidth bound: each step reads the active weights once plus the KV cache of the batch. Dividing those bytes by aggregate bandwidth × an efficiency factor gives the time per step and an upper bound on tokens per second.

Prefill (processing the prompt) is compute bound: about 2 × active parameters floating-point operations per prompt token, divided by the GPUs' vendor-reported BF16 dense peak × a utilisation factor.

Formulas

Weights
M_w = P × b_w
KV cache
M_kv = 2 × L × H_kv × d_head × b_kv × context × sequences
GPUs
N = ⌈(M_w + M_kv) × (1 + overhead) ÷ (VRAM × usable)⌉
Decode step
t_step = (P_active × b_w + M_kv) ÷ (N × bandwidth × efficiency)
Prefill
t_prefill = 2 × P_active × prompt_tokens ÷ (N × peak_BF16 × utilisation)

Assumptions

  • Worst case for decode: every sequence is at the full context length.
  • INT4 is counted as about 4.5 bits per weight to include group-wise quantisation scales.
  • GPU figures are vendor-reported datasheet values (dense, without sparsity).

Limitations

  • Does not model paged-attention fragmentation, speculative decoding, prefix caching or continuous-batching schedulers.
  • Tensor-parallel communication overhead is not modelled; multi-GPU throughput is optimistic.
  • Multi-head latent attention and other KV-compression schemes need different KV formulas.

Frequently asked questions

How much GPU memory does a 70B model need?

In BF16 the weights alone are about 141 GB (70.6 billion parameters × 2 bytes) for Llama 3.1 70B, before KV cache and overhead — more than one 80 GB GPU. With 4-bit quantisation the weights fall to roughly 40 GB. Use the preset and set your context length and concurrency to see the full total.

Why does context length matter so much?

The KV cache grows linearly with context length and with the number of concurrent sequences. At long contexts and high concurrency it can exceed the size of the weights.

Can I add the memory of several GPUs together?

Only if the model is split across them with tensor or pipeline parallelism. Each GPU still needs its own working memory, and splitting adds communication over the GPU interconnect.