AI Dataset & Checkpoint Storage Calculator
Estimate the storage an AI project needs for raw and processed datasets, retained versions, model checkpoints and replicas, and the read throughput required to keep GPUs fed.
Results
- Raw dataset
- 20 TBcalculated
- Datasets incl. processed copies and versions
- 80 TBcalculated
- One checkpoint
- 112 GBcalculated
- Total with replicas
- 161.12 TBcalculated
- Required read throughput
- 2.56 GB/scalculated
What it calculates
- Dataset storage
- Checkpoint size and retention
- Total with replicas
- Required read throughput
Worked example
Using the default inputs below; change any input above to recalculate.
| Samples | 100 million |
|---|---|
| Average sample size | 200 KB |
| Processed copy size (× raw) | 1 |
| Dataset versions retained | 2 |
| Model parameters | 8 billion |
| Checkpoint bytes per parameter | 14 |
| Checkpoints retained | 5 |
| Copies (replication) | 2 |
| GPUs reading | 64 |
| Samples per second per GPU | 200 |
| Raw dataset | 20 TB | calculated |
|---|---|---|
| Datasets incl. processed copies and versions | 80 TB | calculated |
| One checkpoint | 112 GB | calculated |
| Total with replicas | 161.12 TB | calculated |
| Required read throughput | 2.56 GB/s | calculated |
Methodology
Dataset storage = samples × size × (1 + processed ratio) × versions. A full training checkpoint stores weights and optimizer state — about 14 bytes per parameter for mixed-precision Adam. Totals are multiplied by the number of copies. Throughput = GPUs × samples/s × sample size.
Formulas
- Checkpoint
- C = N × 14 B
- Throughput
- B = GPUs × samples/s × size
Assumptions
- Uniform sample size.
Limitations
- Checkpoint bursts need write bandwidth sized separately from read throughput.