AI Inference Capacity Calculator
Size an inference deployment from the number of users, their request rate and token counts, and the throughput you have measured for your model on your hardware.
Results
- Request rate
- 1.67 requests/scalculated
- Output token demand
- 667 tokens/scalculated
- Prompt token demand
- 2,500 tokens/scalculated
- Concurrent streams (Little's law)
- 33.3calculated
- Replica load
- 0.57 replicascalculated
- Replicas with headroom
- 1calculated
- GPUs required
- 1calculated
- KV cache for concurrent streamsAcross all replicas
- 8.3 GBcalculated
Throughput inputs must come from a benchmark with the same model, precision, batch size and sequence lengths.
Peaks above the hourly average need extra headroom or autoscaling.
What it calculates
- Request rate
- Output and prompt token demand
- Concurrent streams
- Replicas and GPUs
- KV cache memory for concurrency
Worked example
Using the default inputs below; change any input above to recalculate.
| Active users | 1000 |
|---|---|
| Requests per user per hour | 6 |
| Prompt tokens per request | 1500 |
| Output tokens per request | 400 |
| Target speed per user | 20 tokens/s |
| Measured output throughput per replica | 1500 tokens/s |
| Measured prompt throughput per replica | 20000 tokens/s |
| GPUs per replica | 1 |
| KV cache per token | 128 KiB |
| Headroom | 30 % |
| Request rate | 1.67 requests/s | calculated |
|---|---|---|
| Output token demand | 667 tokens/s | calculated |
| Prompt token demand | 2,500 tokens/s | calculated |
| Concurrent streams (Little's law) | 33.3 | calculated |
| Replica load | 0.57 replicas | calculated |
| Replicas with headroom | 1 | calculated |
| GPUs required | 1 | calculated |
| KV cache for concurrent streams | 8.3 GB | calculated |
Methodology
Requests per second are users × requests per hour ÷ 3600. Multiplying by tokens per request gives the token demand for prompts (prefill) and outputs (decode).
Each replica's capacity is your measured throughput. The fraction of a replica needed is the sum of decode demand ÷ decode capacity and prefill demand ÷ prefill capacity; headroom is added and the result is rounded up.
Concurrent streams follow Little's law: arrival rate × time each request spends generating (output tokens ÷ per-user speed).
Formulas
- Arrival rate
- λ = users × requests_per_hour ÷ 3600
- Concurrency
- L = λ × (output_tokens ÷ tokens_per_second_per_user)
- Replicas
- R = ⌈(λ·out ÷ C_out + λ·in ÷ C_in) × (1 + headroom)⌉
Assumptions
- Load is spread evenly over the hour.
- Prefill and decode share replica time additively.
Limitations
- Does not model latency percentiles, queueing at high utilisation or burst traffic.
- Default throughput values are example inputs, not benchmarks.