Object Storage & Training Data Streaming: The S3-to-GPU Read Path

Note: This is a review article (no first-hand experimental data), based on public practice and data engineering.

Experiment Objective

More training data now lives in object storage (S3/MinIO/cloud OSS) than parallel file systems — it is cheap, infinitely scalable, and the ecosystem is standard. But "data in object storage, how do GPUs eat it efficiently" is an engineering challenge: direct reads are slow, random reads explode, streams drop frequently. Goal: dissect object storage and training data integration — object storage architecture and data organization (why it is bad for random small reads) → streaming reads and cache tiers (S3→local disk/memory cache) → throughput optimization and resumed reads → division of labor with parallel file systems. After reading you can answer: when training data lives in object storage, how to design the read path so it does not slow training.

Code Walkthrough

Part 1: Object storage data organization — big objects + prefix sharding

# Best practice: organize training data as "big files + prefix shards"
# Bad: 100k small 1KB jsons (random small reads, QPS explosion)
# Good: pack into shard files (e.g. 512MB each), prefix-partitioned
#   s3://bucket/dataset/v2026-10/shard-000000.tar
#   s3://bucket/dataset/v2026-10/shard-000001.tar
# Metadata: manifest.json lists shards, sample counts, checksums

Key points:

1. Object storage's nature: built for "large-object sequential reads," high per-request latency (tens of ms), throughput via concurrency, random small reads QPS-limited — packing data into large shards (hundreds of MB) is rule one; 100k small files will blow up the request queue.
2. Prefix sharding is the basis of parallel reads: with prefix-partitioned shards, multiple readers pull different shards concurrently and throughput scales linearly.
3. Manifest is the single entry: training reads only the manifest (shard list), never buckets — listing a bucket is the slowest object-store operation; avoid it.

Part 2: Streaming reads and cache tiers

# Core: S3 does not feed GPUs directly; a local disk cache layer sits between
# Architecture: S3(cold) → local NVMe cache(hot) → DataLoader
# Implementation: cache libraries (fsspec block cache etc.) cache S3 objects
#       block-wise to local disk; second access hits local
from fsspec.implementations.http import HTTPFileSystem  # illustrative
# Engineering: cache policy = LRU + prefetch (next block by shard order)

Key points:

• Cache tiering is mandatory: S3 latency is tens of ms; training DataLoader needs µs-ms reads — a "local NVMe cache layer" must sit in between (the concrete implementation of ch04's hot/cold tiering).
• Cache granularity is "blocks," not "files": block-wise caching (e.g. 4MB blocks) fetches only needed blocks on random access, avoiding whole-file pulls.
• Prefetch aligns with training rhythm: training consumes shards in order; the prefetcher pulls the next shard ahead — hiding network latency behind compute (same as ch11's data-parallel prefetch).

Principles

Why object storage cannot directly feed training

Trait	Object storage	Training need
per-request latency	tens of ms	millisecond-level
random small reads	QPS-limited	high-frequency random
throughput	concurrency-stacked (per-object capped)	stable high bandwidth
cost	low (cold data)	want low

Core contradiction: object storage is "cheap but slow," training is "fast but expensive" — the solution is not either/or but cache tiering: S3 holds cold data, local disk caches hot data, training faces only the cache layer. That is "data tiering" (ch04's selection article) applied to object storage.

Division of labor with parallel file systems

Dimension	Object store (S3/MinIO)	Parallel FS (Lustre/GPFS/BeeGFS)
role	cold data, archive, data lake	hot data, random small reads, checkpoints
read pattern	large-object sequential	arbitrary
throughput	concurrency-stacked	high bandwidth low latency
cost	low	high

Engineering judgment: raw datasets (large, read-only, cold) → object storage; training hot data/checkpoints → parallel FS or local NVMe. A cache layer bridges them — "object storage as the lake, PFS/NVMe as the hot tier" is the mainstream data architecture for large clusters.

Faults and consistency

Reading training data from object storage means handling three fault classes: network jitter (retry + backoff), missing objects (manifest validation), data corruption (checksum validation — as in ch04's benchmark article). Retries need an upper bound (e.g. 3 exponential backoffs), then fall back to local cache or error — never let unbounded retries stall training.

Results

(Estimated, not measured; S3 + local NVMe cache)

Option	Effective throughput	Notes
direct S3 reads (small files)	low (QPS explosion)	not recommended
large shards + concurrent S3	mid (network/object-limited)	runnable
S3 + local block cache + prefetch	high (near-local)	recommended

Engineering Takeaways

1. Pack before connecting: data into hundred-MB shards + manifest is rule one for object storage — small files are the biggest killer.
2. Cache tiering is standard: S3 → local NVMe block cache → DataLoader, three layers; training faces only the cache layer.
3. Prefetch hides latency: prefetch by shard order, hide network latency behind compute; bounded retries, never let read failures stall training.
4. Clear PFS division: object storage for the cold lake, PFS/NVMe for hot data and checkpoints — both storage classes serve distinct roles.

Next article preview: data path done; how do you know whether storage is actually saturated — "Storage Observability & Performance Governance."