Skip to content
GIRIVON
Infrastructure+
Systems+
Components+
Computing Devices+
Solutions+
Compute Tools+
Knowledge+

AI Inference

Serving trained models to users and applications with predictable latency.

What matters, in order

  1. Memory for weights and KV cache
  2. Memory bandwidth (token generation speed)
  3. Throughput per watt and per rupee
  4. Batching and concurrency

Engineering guidance

Token generation is usually memory-bandwidth bound: each generated token reads the model weights once per batch step. Quantisation (FP8, INT8, INT4) reduces memory and increases tokens per second at some accuracy cost; the KV cache grows with context length and concurrent users.

Size it

Categories

Products tagged for AI Inference

28 products

Compare selected