Training Data Storage Selection: Object Store, Parallel File Systems & Local NVMe

Experiment Goal

The storage question for a training cluster isn't "buy the biggest one" — it's letting three storage classes each do their job:

• Where do datasets live? Where do checkpoints write? Where does archival go?
• Why is "everything on a parallel file system" both expensive and slow?
• What's the division of labor between object store, parallel file systems, and local NVMe?

This article walks through training data storage selection with a decision flow: positioning of three classes, data-flow analysis of training loads, tiered architecture, and a landing checklist. Note: this is a review + architecture-analysis route based on generic engineering practice, without first-hand experiment data; specific products and parameters follow official docs.

Section 1: Positioning — No All-Purpose Storage

Storage	Strengths	Weaknesses	Role in training
Local NVMe	ultra-low latency, very high per-node bandwidth	limited capacity, not shared, data-loss risk	cache/temp, hot-dataset shards, inference cache
Parallel file system (Lustre/BeeGFS/GPFS)	large-scale shared, high concurrent bandwidth, POSIX	expensive, weak small files, complex ops	training data loading, checkpoint IO, shared workspace
Object store (S3/MinIO/Ceph RGW)	massive capacity, cheap, highly available, scales on demand	high latency, no POSIX, weak small-object concurrency	dataset archival, checkpoint archival, images/models

Core insight: the three classes have completely different cost curves and performance curves — local NVMe is fastest and priciest (smallest capacity), object store is cheapest and slowest (unlimited capacity), parallel FS sits in between. Training system design is essentially placing data on the tier matching its access frequency, not picking "the best one".

Section 2: Training Data-Flow Analysis — Choose Storage by Path

Training data travels several paths from source to GPU, each with different storage demands:

Data production / collection
   ↓ archival
Object store (raw datasets, history, checkpoint snapshots)
   ↓ distribution / loading
Parallel file system (working copies for training reads, shared workspace, checkpoint IO)
   ↓ hot data
Local NVMe (cache/shards, reducing PFS pressure)
   ↓
GPU reads

Data-flow path	Traits	Suitable storage	Why
Dataset archival	large, cold, low frequency	object store	cheap capacity, HA; doesn't consume PFS hot capacity
Training data loading	high bandwidth, concurrent reads	parallel FS	shared + high concurrency (verified with Chapter 04 benchmark)
Checkpoint writes	large-block writes, must persist	parallel FS	high-bandwidth writes + shared recovery (multi-card sync read)
Checkpoint archival	low frequency, long-term	object store	cold archival; avoids PFS capacity filled by history
Hot datasets / small samples	high frequency, small	local NVMe cache	cuts PFS read pressure; prefetch training data locally

Section 3: Tiered Architecture — Recommended Landing Shape

Object store (source/archive/snapshots)
   │ on-demand loading / prefetch
Parallel file system (working copies: training data + checkpoints + shared workspace)
   │ prefetch (before training start)
Local NVMe (per-node cache: hot shards, temp data)

Landing points:

1. Object store as the "data lake base": raw datasets, historical checkpoints, model snapshots all live there; capacity and cost stay controlled;
2. Parallel FS only for "live data": training working copies and current checkpoints; use capacity monitoring to prevent being filled by history (Chapter 04 benchmark + capacity planning);
3. Local NVMe as the "acceleration layer": prefetch dataset shards to node-local storage before training (or DataLoader on-demand cache) to cut PFS read pressure — biggest win for repeatedly-read datasets;
4. Automated tier migration: checkpoints auto-archive to object store after completion (TTL policy); training data migrates between PFS/NVMe by usage frequency — manual data moves don't scale.

Section 4: Selection Decision Flow

Dataset scale / access pattern
├─ massive cold data / archival → object store
├─ high-concurrency shared reads (main training load) → parallel FS
├─ per-node high-frequency small data → local NVMe
└─ multi-tier combo: object(source) + PFS(working) + NVMe(cache) ← production recommended

Judgment basis: pick against the four dimensions — capacity, concurrency, access frequency, shareability; don't select by "expensive means right".

Selection Checklist

• [ ] Dataset archival in object store, not consuming PFS hot capacity
• [ ] Training working copies & checkpoints on parallel FS (high-concurrency IO verified)
• [ ] Hot-data/cache tier on local NVMe; prefetch mechanism ready
• [ ] Checkpoint auto-archival (TTL policy); PFS capacity monitoring with alerts
• [ ] Tier migration automated; manual data moves eliminated
• [ ] Per-tier capacity/cost standards unified, into Chapter 02 capacity planning total

Engineering Takeaways

• No all-purpose storage, only sensible division of labor: local NVMe, parallel FS, object store each own a segment of the data flow; the combination is the production architecture.
• Choose storage by data flow, not by brand: first draw "where data comes from and goes to", map each path against traits, and the selection becomes clear.
• Automated migration is the soul of tiering: tiering without automation isn't tiering — TTL archival and prefetch must land with the architecture.

Next article preview: with the storage scheme settled, the provisioning pipeline must support scale delivery — the next article, Large-Scale Provisioning Pipeline: Golden Image, Version Management & Canary Rollout, breaks down image engineering.