AI Training Compute Calculator
Estimate the compute needed to train a transformer model from its parameter count and training tokens, then convert it into wall-clock time on a given GPU fleet, GPU-hours, electricity and optional cloud cost.
Results
- Total training compute6 × parameters × tokens
- 4.80e+22 FLOPscalculated
- Time to train64 × NVIDIA H100 SXM5 80GB at 40% MFU
- 21.9 daysestimated
- GPU-hours
- 33,704estimated
- Energy (facility)
- 56,151 kWhestimated
- Electricity cost
- ₹4.49 lakhestimated
- Minimum GPUs to hold training stateFully sharded, excluding activations
- 2calculated
- Compute-optimal tokens (Chinchilla ≈ 20 × parameters)
- 160 billionRule of thumb
Wall-clock time also depends on data loading, checkpointing, failures and restarts, which are not included.
What it calculates
- Total training FLOPs
- Time to train
- GPU-hours
- Energy (kWh)
- Electricity cost
- Cloud cost
- Minimum GPUs to hold training state
Worked example
Using the default inputs below; change any input above to recalculate.
| Model parameters | 8 billion |
|---|---|
| Training tokens | 1000 billion |
| GPU | NVIDIA H100 SXM5 80GB |
| Number of GPUs | 64 |
| Model FLOPs utilisation (MFU) | 40 % |
| System power ÷ GPU power | 1.7 |
| Facility PUE | 1.4 |
| Electricity tariff | 8 ₹/kWh |
| Cloud price per GPU-hour (optional) | 0 ₹ |
| Total training compute | 4.80e+22 FLOPs | calculated |
|---|---|---|
| Time to train | 21.9 days | estimated |
| GPU-hours | 33,704 | estimated |
| Energy (facility) | 56,151 kWh | estimated |
| Electricity cost | ₹4.49 lakh | estimated |
| Minimum GPUs to hold training state | 2 | calculated |
| Compute-optimal tokens (Chinchilla ≈ 20 × parameters) | 160 billion | Rule of thumb |
Methodology
Training a dense transformer takes about 6 floating-point operations per parameter per token: 2 for the forward pass and 4 for the backward pass (Kaplan et al., 2020). Multiplying by the training tokens gives total compute.
Time is that compute divided by the cluster's delivered throughput: GPUs × vendor-reported BF16 dense peak × model FLOPs utilisation (MFU). Energy multiplies GPU board power by a system factor for host components, by hours, and by the facility PUE.
The minimum GPU count uses about 16 bytes per parameter for mixed-precision Adam (BF16 weights and gradients, FP32 master weights and two FP32 optimizer moments; Rajbhandari et al., 2020), sharded across GPUs.
Formulas
- Compute
- C = 6 × N × D
- Time
- T = C ÷ (GPUs × peak_BF16 × MFU)
- Energy
- E = GPUs × P_GPU × power_factor × T × PUE
- Compute-optimal data
- D_opt ≈ 20 × N (Hoffmann et al., 2022)
Assumptions
- Dense transformer; attention FLOPs are small relative to 6ND at typical sequence lengths.
- GPU peak figures are vendor-reported dense values.
- Power factor and PUE are your inputs.
Limitations
- Mixture-of-experts models need active parameters in place of total parameters for compute.
- Very long sequences add attention FLOPs not captured by 6ND.
- Does not include evaluation, failed runs or hyperparameter searches.
Frequently asked questions
What MFU should I assume?
Published large-scale training runs commonly report 30–50% MFU on modern GPUs. Smaller or less-optimised jobs can be lower. Use your own measured figure when you have one.
Does fine-tuning use the same formula?
Full fine-tuning does — use the fine-tuning token count. Parameter-efficient methods such as LoRA still run forward and backward passes through the frozen model, so compute is lower but not proportional to trainable parameters.