AI Agent Infrastructure Calculator
AI agents make many model calls per task, each with long, growing contexts. Estimate the token traffic a fleet of agents generates and the GPUs needed to serve it from your measured throughput.
Results
- Model calls
- 4.44 calls/scalculated
- Output token demand
- 1,333 tokens/scalculated
- Prompt tokens to compute
- 13,333 tokens/scalculated
- GPUs required3 replicas incl. 30% headroom
- 3calculated
- Tokens per day
- 1,267.2 millioncalculated
Agent workloads are dominated by prompt (prefill) tokens; prefix caching of shared system prompts and history can reduce them substantially.
What it calculates
- Model calls per second
- Prompt and output token demand
- GPUs required
- Daily token volume
- Daily serving cost
Worked example
Using the default inputs below; change any input above to recalculate.
| Concurrently active agents | 500 |
|---|---|
| Tasks per agent per hour | 4 |
| Model calls per task | 8 |
| Prompt tokens per call | 3000 |
| Output tokens per call | 300 |
| Prompt prefix cache hit rate | 0 % |
| Measured output throughput per replica | 1500 tokens/s |
| Measured prompt throughput per replica | 20000 tokens/s |
| GPUs per replica | 1 |
| Headroom | 30 % |
| Cost per GPU-hour (optional) | 0 ₹ |
| Model calls | 4.44 calls/s | calculated |
|---|---|---|
| Output token demand | 1,333 tokens/s | calculated |
| Prompt tokens to compute | 13,333 tokens/s | calculated |
| GPUs required | 3 | calculated |
| Tokens per day | 1,267.2 million | calculated |
Methodology
Calls per second = agents × tasks per hour × calls per task ÷ 3600. Prompt and output token demand follow from tokens per call, with cached prompt tokens removed from the prefill demand.
GPU demand is computed exactly as in the inference capacity calculator: the sum of decode and prefill demand divided by your measured replica throughputs, plus headroom.
Formulas
- Call rate
- λ = agents × tasks_per_hour × calls_per_task ÷ 3600
- Replicas
- R = ⌈(λ·out ÷ C_out + λ·in·(1 − cache) ÷ C_in) × (1 + headroom)⌉
Assumptions
- Agents are active simultaneously and evenly over the hour.
- Tool execution time does not consume GPU capacity.
Limitations
- Does not model long-tail latency or multi-model routing.
- Default throughput values are example inputs.