Skip to content

Data Pipeline Tuning: Choosing Between DataLoader, DALI and WebDataset ​

Summary: When nvidia-smi shows GPU utilization oscillating between 40% and 60% and step time jitters, the usual suspect is not the model or the network but a data pipeline that cannot keep up. The hard part is not which knob to turn, but deciding first whether the bottleneck is CPU, storage or GPU decoding — get that wrong and raising num_workers from 8 to 32 just makes memory the bottleneck first.

1. Three pipelines that solve three different problems ​

ApproachData shapeWhat it solvesTypical cost
DataLoader + map-style DatasetSmall indexed datasets, annotation tablesMost general; random access to any sampleOne random read per sample; hammered by many small files
DataLoader + IterableDataset (WebDataset is the canonical implementation)Large tar shards, object storageSequential reads, instant startup, no metadataExact cross-node balance is hard; needs shard resampling
DALIImage/video decoding and augmentationMoves decode and augmentation to the GPU, bypassing CPU limitsGPU memory, tighter coupling with training, useless for text

These are not old-versus-new substitutes. Text corpora are easiest with WebDataset, image classification gains most from DALI once CPU decoding saturates, and mid-sized datasets with annotation tables are still simplest with a map-style DataLoader.

2. Locate first: three bottlenecks, three kinds of evidence ​

Do not tune by intuition. Each bottleneck has concrete evidence:

BottleneckEvidenceImplication
CPU side (Python decode/augment)top -H shows a dataloader process pegging one core; GPU utilization moves inversely with CPURaise num_workers or move decoding to the GPU
Storage sideiostat -x 1 shows rising r_await and bandwidth far below device limitsSwitch to sequential reads, change shard format, add clients
GPU side (decode on device)GPU utilization is high but throughput does not grow; memory eaten by decodingReduce DALI concurrency or decode on CPU again

A cheap and effective test: run 50 steps with fake data (torch.randn batches, no disk, no decode) to get the pure-compute step time, then 50 steps with real data. The difference is the fixed cost of the data pipeline. If that gap exceeds 30% of total step time, any model-side optimization is paying a tariff to the data pipeline.

3. DataLoader: documented defaults and real costs of five parameters ​

The defaults in the official signature are easy to overlook (PyTorch v2.12 documentation):

python
DataLoader(dataset, batch_size=1, shuffle=False, num_workers=0,
           pin_memory=False, drop_last=False, timeout=0,
           worker_init_fn=None, *, prefetch_factor=2,
           persistent_workers=False, in_order=True)
ParameterDefaultMeaning and cost
num_workers00 loads in the main process and blocks compute; subprocesses only start above 0
prefetch_factor2 (only when num_workers>0; None when 0)Each worker prefetches 2 batches, so 2 * num_workers batches are in flight
pin_memoryFalsePins Tensors for faster host-to-device copies
persistent_workersFalseWorkers are torn down at the end of each iteration and recreated next epoch
in_orderTrueSetting it False gives up FIFO delivery; the documentation explicitly warns it harms reproducibility and can feed a skewed distribution

Three traps:

One: prefetch_factor multiplies memory. Batches in flight = num_workers × prefetch_factor. Eight workers × 2 means 16 batches resident. If a batch is 64 images at 512×512 RGB (~200 MB), prefetching alone holds 3 GB of host memory.

Two: worker memory is not "as needed", it is multiplied from the parent. The documentation states that workers consume the same amount of CPU memory as the parent process for all Python objects accessed from them — total usage is roughly number of workers × sizeof parent process. So loading the full filename list or annotation table in Dataset.__init__ and running 32 workers copies that data 32 times. The documented fix is non-refcounted representations such as NumPy, PyArrow or Pandas.

Three: do not return CUDA Tensors from workers. The documentation recommends against it in multi-process loading; use pin_memory=True and let the main process do the host-to-device copy.

4. The duplicate-replica trap in IterableDataset ​

With num_workers>0, every worker receives an independent copy of the Dataset object. For map-style datasets that is fine, since the main-process sampler hands out indices. For IterableDataset, "every worker iterates the same stream from the beginning" simply means duplicated data.

The documented fixes: split inside __iter__ using torch.utils.data.get_worker_info(), or configure replicas in worker_init_fn based on worker_info.id. Also note that in iterable mode drop_last drops the last non-full batch of each worker's replica, not one batch for the whole dataset — silently losing samples in distributed runs.

5. WebDataset: why sequential reads can saturate local disks ​

The format has exactly two conventions: shards are tar files named like something-000000.tar, and one training sample is a set of files sharing a basename (PMC4991227_00003.json + .png). The official README states the payoff concretely: purely sequential I/O pipelines are 3–10× faster than random access on local drives, which is also why the format fits object storage — object stores are bad at small random reads by design.

Typical usage:

python
import webdataset as wds

ds = (wds.WebDataset(urls, shardshuffle=True, nodesplitter=wds.split_by_node)
      .shuffle(1000)
      .decode("pil")
      .to_tuple("png", "json")
      .map(preprocess)
      .batched(16))

Four things to know:

  • shardshuffle must be set explicitly. Leaving it unset triggers the official warning (WebDataset(shardshuffle=...) is None; set explicitly to False or a number), and an undefined shard order breaks the assumption that every epoch sees a different order.
  • Two shuffles are two different things: shardshuffle randomizes shard order (coarse, cross-epoch), while .shuffle(buffer) randomizes samples inside a shard, bounded by the buffer size. With only the former, sample order within a shard never changes.
  • You need enough shards. split_by_node and split_by_worker hand shards out round-robin; if shard count is below nodes × workers per node, some nodes get nothing and idle.
  • Exact balance is impossible. The README is explicit that achieving a perfectly balanced sample count across many nodes for a fixed epoch size is tricky, and multi-node training normally uses shard resampling. This is the format's main disadvantage versus map-style datasets.

Interplay with storage: on a parallel file system, size shards above the stripe width (a few hundred MB of shard with 4 MB stripes) so a single shard is read from several OSTs in parallel; on object storage, keep shards in the tens-to-hundreds-of-MB range so an oversized GET does not make retries expensive.

6. DALI: only when CPU decoding is genuinely the bottleneck ​

The core idea is moving decode and augmentation to the GPU: fn.decoders.image(device="mixed") reads bytes on the CPU and decodes on the GPU, taking the expensive JPEG/video decode off the CPU.

Its applicable range is narrow but clear:

  • Image/video training where top shows decode processes saturating the CPU while GPUs idle;
  • Long augmentation chains (resize + crop + color + normalize) that bottleneck on CPU;
  • Stable batch sizes and regular data formats.

It does not fit: text/sequence corpora (DALI has no matching operators), NLP tasks with wildly varying sample lengths, or training already tight on GPU memory — decoding competes with activations for the same HBM.

A pragmatic rule: quantify the CPU decode share first (py-spy or cProfile for 30 seconds). If decode plus augmentation is under 15% of total step time, DALI's gain rarely outweighs the extra dependency and debugging cost.

ScenarioRecommended pipelineKey parameters/design
Image classification/detection, 10⁵–10⁷ samples with annotation tablesMap-style DataLoader; DALI only once CPU decode saturatesnum_workers ≈ 4–8 per GPU; pin_memory=True; persistent_workers=True
LLM pretraining corpora (10⁹+ samples, tar shards)IterableDataset + WebDatasetExplicit shardshuffle; shards ≥ nodes×workers; use shard resampling
Multimodal (image + text + long video)WebDataset for storage and sharding, DALI for decode as neededSequential shard reads first, then GPU decode for the image column only
Small datasets that fit in memorynum_workers=0 with full preloadDo not add multiprocessing or sharding; debugging cost exceeds the gain

8. A repeatable localization workflow ​

  1. Measure the compute baseline: 50 steps on fake data, record step_time_compute.
  2. Measure real data: 50 steps with the same configuration, record step_time_real; data_overhead = step_time_real - step_time_compute.
  3. Watch the CPU: top -H / py-spy dump to see whether time goes to decode, augmentation or the Python loop.
  4. Watch storage: compare r_await and rkB/s from iostat -x 1 against device limits, and against the file system's per-client bandwidth ceiling.
  5. Change one variable at a time: num_workers first (doubling up to half the cores), then prefetch_factor, and only then consider a different pipeline shape. Re-run steps 1 and 2 after each change.

9. Six pitfalls ​

  • "More num_workers is always better": worker memory ≈ workers × parent memory, so memory usually gives out before throughput improves.
  • Putting the filename list in the Dataset: string lists are refcounted, so 32 workers means 32 copies — switch to PyArrow/NumPy.
  • Forgetting to shard an IterableDataset: every worker iterates the same stream, feeding duplicates and producing suspiciously smooth loss.
  • Not setting shardshuffle in WebDataset: the library warns explicitly, and a fixed shard order defeats per-epoch shuffling.
  • Fewer shards than nodes×workers: some ranks receive nothing and idle, capping aggregate throughput.
  • Treating DALI as a universal fix: no operators for text, and on memory-tight jobs decoding competes with activations and makes things slower.

To close: the first rule of data pipelines is measure the overhead share first; the second is match the data shape to what storage is good at — sequential reads for object storage, shards wider than the stripe for parallel file systems, GPU decoding only after CPU decoding saturates.

AI-HPC Expert · Contact: openaihpc@gmail.com