Skip to content
GIRIVON
Infrastructure+
Systems+
Components+
Computing Devices+
Solutions+
Compute Tools+
Knowledge+

AI Inference Capacity Calculator

Size an inference deployment from the number of users, their request rate and token counts, and the throughput you have measured for your model on your hardware.

Demand
Capacity

Replace with your benchmark for this model, precision and sequence length.

From the LLM calculator (Llama 3.1 8B in BF16 is 128 KiB).

Results

Request rate
1.67 requests/s
Output token demand
667 tokens/s
Prompt token demand
2,500 tokens/s
Concurrent streams (Little's law)
33.3
Replica load
0.57 replicas
Replicas with headroom
1
GPUs required
1
KV cache for concurrent streamsAcross all replicas
8.3 GB

Throughput inputs must come from a benchmark with the same model, precision, batch size and sequence lengths.

Peaks above the hourly average need extra headroom or autoscaling.

What it calculates

  • Request rate
  • Output and prompt token demand
  • Concurrent streams
  • Replicas and GPUs
  • KV cache memory for concurrency

Worked example

Using the default inputs below; change any input above to recalculate.

Inputs
Active users1000
Requests per user per hour6
Prompt tokens per request1500
Output tokens per request400
Target speed per user20 tokens/s
Measured output throughput per replica1500 tokens/s
Measured prompt throughput per replica20000 tokens/s
GPUs per replica1
KV cache per token128 KiB
Headroom30 %
Results
Request rate1.67 requests/scalculated
Output token demand667 tokens/scalculated
Prompt token demand2,500 tokens/scalculated
Concurrent streams (Little's law)33.3calculated
Replica load0.57 replicascalculated
Replicas with headroom1calculated
GPUs required1calculated
KV cache for concurrent streams8.3 GBcalculated

Methodology

Requests per second are users × requests per hour ÷ 3600. Multiplying by tokens per request gives the token demand for prompts (prefill) and outputs (decode).

Each replica's capacity is your measured throughput. The fraction of a replica needed is the sum of decode demand ÷ decode capacity and prefill demand ÷ prefill capacity; headroom is added and the result is rounded up.

Concurrent streams follow Little's law: arrival rate × time each request spends generating (output tokens ÷ per-user speed).

Formulas

Arrival rate
λ = users × requests_per_hour ÷ 3600
Concurrency
L = λ × (output_tokens ÷ tokens_per_second_per_user)
Replicas
R = ⌈(λ·out ÷ C_out + λ·in ÷ C_in) × (1 + headroom)⌉

Assumptions

  • Load is spread evenly over the hour.
  • Prefill and decode share replica time additively.

Limitations

  • Does not model latency percentiles, queueing at high utilisation or burst traffic.
  • Default throughput values are example inputs, not benchmarks.