Skip to content
GIRIVON
Infrastructure+
Systems+
Components+
Computing Devices+
Solutions+
Compute Tools+
Knowledge+

AI Dataset & Checkpoint Storage Calculator

Estimate the storage an AI project needs for raw and processed datasets, retained versions, model checkpoints and replicas, and the read throughput required to keep GPUs fed.

≈14 for full mixed-precision Adam state (BF16 weights + FP32 master + two moments); 2 for weights only.

Results

Raw dataset
20 TB
Datasets incl. processed copies and versions
80 TB
One checkpoint
112 GB
Total with replicas
161.12 TB
Required read throughput
2.56 GB/s

What it calculates

  • Dataset storage
  • Checkpoint size and retention
  • Total with replicas
  • Required read throughput

Worked example

Using the default inputs below; change any input above to recalculate.

Inputs
Samples100 million
Average sample size200 KB
Processed copy size (× raw)1
Dataset versions retained2
Model parameters8 billion
Checkpoint bytes per parameter14
Checkpoints retained5
Copies (replication)2
GPUs reading64
Samples per second per GPU200
Results
Raw dataset20 TBcalculated
Datasets incl. processed copies and versions80 TBcalculated
One checkpoint112 GBcalculated
Total with replicas161.12 TBcalculated
Required read throughput2.56 GB/scalculated

Methodology

Dataset storage = samples × size × (1 + processed ratio) × versions. A full training checkpoint stores weights and optimizer state — about 14 bytes per parameter for mixed-precision Adam. Totals are multiplied by the number of copies. Throughput = GPUs × samples/s × sample size.

Formulas

Checkpoint
C = N × 14 B
Throughput
B = GPUs × samples/s × size

Assumptions

  • Uniform sample size.

Limitations

  • Checkpoint bursts need write bandwidth sized separately from read throughput.