Skip to content
GIRIVON
Infrastructure+
Systems+
Components+
Computing Devices+
Solutions+
Compute Tools+
Knowledge+

AI Training Compute Calculator

Estimate the compute needed to train a transformer model from its parameter count and training tokens, then convert it into wall-clock time on a given GPU fleet, GPU-hours, electricity and optional cloud cost.

Model
Cluster

Used only when GPU is set to Custom.

Used only when GPU is set to Custom.

Used only when GPU is set to Custom.

Used only when GPU is set to Custom.

Share of peak throughput actually achieved. Well-tuned large runs commonly report 30–50%.

Energy

Adds host CPUs, memory, NICs and fans. Dense 8-GPU servers are often 1.6–1.9× the GPUs' board power.

Results

Total training compute6 × parameters × tokens
4.80e+22 FLOPs
Time to train64 × NVIDIA H100 SXM5 80GB at 40% MFU
21.9 days
GPU-hours
33,704
Energy (facility)
56,151 kWh
Electricity cost
₹4.49 lakh
Minimum GPUs to hold training stateFully sharded, excluding activations
2
Compute-optimal tokens (Chinchilla ≈ 20 × parameters)
160 billion

Wall-clock time also depends on data loading, checkpointing, failures and restarts, which are not included.

What it calculates

  • Total training FLOPs
  • Time to train
  • GPU-hours
  • Energy (kWh)
  • Electricity cost
  • Cloud cost
  • Minimum GPUs to hold training state

Worked example

Using the default inputs below; change any input above to recalculate.

Inputs
Model parameters8 billion
Training tokens1000 billion
GPUNVIDIA H100 SXM5 80GB
Number of GPUs64
Model FLOPs utilisation (MFU)40 %
System power ÷ GPU power1.7
Facility PUE1.4
Electricity tariff8 ₹/kWh
Cloud price per GPU-hour (optional)0 ₹
Results
Total training compute4.80e+22 FLOPscalculated
Time to train21.9 daysestimated
GPU-hours33,704estimated
Energy (facility)56,151 kWhestimated
Electricity cost₹4.49 lakhestimated
Minimum GPUs to hold training state2calculated
Compute-optimal tokens (Chinchilla ≈ 20 × parameters)160 billionRule of thumb

Methodology

Training a dense transformer takes about 6 floating-point operations per parameter per token: 2 for the forward pass and 4 for the backward pass (Kaplan et al., 2020). Multiplying by the training tokens gives total compute.

Time is that compute divided by the cluster's delivered throughput: GPUs × vendor-reported BF16 dense peak × model FLOPs utilisation (MFU). Energy multiplies GPU board power by a system factor for host components, by hours, and by the facility PUE.

The minimum GPU count uses about 16 bytes per parameter for mixed-precision Adam (BF16 weights and gradients, FP32 master weights and two FP32 optimizer moments; Rajbhandari et al., 2020), sharded across GPUs.

Formulas

Compute
C = 6 × N × D
Time
T = C ÷ (GPUs × peak_BF16 × MFU)
Energy
E = GPUs × P_GPU × power_factor × T × PUE
Compute-optimal data
D_opt ≈ 20 × N (Hoffmann et al., 2022)

Assumptions

  • Dense transformer; attention FLOPs are small relative to 6ND at typical sequence lengths.
  • GPU peak figures are vendor-reported dense values.
  • Power factor and PUE are your inputs.

Limitations

  • Mixture-of-experts models need active parameters in place of total parameters for compute.
  • Very long sequences add attention FLOPs not captured by 6ND.
  • Does not include evaluation, failed runs or hyperparameter searches.

Frequently asked questions

What MFU should I assume?

Published large-scale training runs commonly report 30–50% MFU on modern GPUs. Smaller or less-optimised jobs can be lower. Use your own measured figure when you have one.

Does fine-tuning use the same formula?

Full fine-tuning does — use the fine-tuning token count. Parameter-efficient methods such as LoRA still run forward and backward passes through the frozen model, so compute is lower but not proportional to trainable parameters.