Parallel File System Benchmarking & Tuning: IOR/mdtest in Practice

Experiment Goal

After Lustre, BeeGFS, or GPFS is installed and mounted, "reads and writes feel fast" doesn't count — training clusters require storage that is measurable, reproducible, and regression-testable:

• At delivery acceptance, what benchmark proves the storage meets the design spec (e.g. "2GB/s per node read, 100K IOPS metadata")?
• When training slows down, how do you tell whether it's the storage, network, or application layer?
• After every scaling or tuning change, how do you confirm things didn't get worse?

This article walks through parallel file system benchmarking with IOR and mdtest as the main line: parameter semantics, workload modeling, result judgment, and tuning paths. Commands are reproducible operations; concrete numbers vary with hardware and environment, so methods are described without fabricating benchmarks.

Section 1: Understand the Division of Labor

Tool	Measures	Training load modeled
IOR	Data bandwidth (read/write, block size & concurrency controllable)	Training data loading, checkpoint writes, inference cache
mdtest	Metadata op rates (create/stat/delete, file count controllable)	Many small files: dataset listing scans, resume-training directory walks

Both are required: high bandwidth but slow metadata = "fast on big files, slow on small ones"; the reverse = "file ops fine, but data flow insufficient."

Section 2: IOR — Command & Parameter Breakdown

# Basic bandwidth test: 64 processes × 16MB blocks, 1GiB file each
mpirun -np 64 -hostfile hosts.txt \
  /usr/local/bin/ior -a POSIX -o /mnt/pfs/ior-test.dat \
  -w -r -b 1g -t 16m -F -e -g \
  -i 1

Parameter semantics:

Parameter	Meaning	Tuning notes
-w -r	write then read	measures both bandwidths
-b 1g	file size per process 1GiB	total = processes × block size; cover the real training-set scale
-t 16m	transfer block 16MiB	simulates large contiguous IO (training data/checkpoint); small blocks (e.g. 4k) for random small IO
-F	one file per process	simulates concurrent per-process writes (checkpoint shards); drop for shared-file
-e	with fsync	simulates "must hit disk"; without it measures pure cache throughput
-g	gaps between operations	avoids interference

Judgment:

• Record Aggregate write/read bandwidth and IOPS (given directly in IOR output);
• Compare against pre-deployment design specs and vendor reference values;
• The key is a regression baseline: save the first acceptance run, then rerun the same command after every scaling/tuning change — a benchmark without a baseline is meaningless.

Section 3: mdtest — Metadata Rates

# Metadata test: 64 processes, 10K files each (dirs adjustable)
mpirun -np 64 -hostfile hosts.txt \
  /usr/local/bin/mdtest -d /mnt/pfs/mdtest-dir \
  -n 10000 -i 3 \
  -C -T -r

• -C (create), -T (stat), -r (remove) measure three metadata op classes;
• Watch files/s — the direct metric for dataset listing scans and checkpoint metadata updates;
• Directory count matters: 10K files in one dir vs per-process dirs give wildly different results — model the real layout (datasets are usually organized in directories), not all in one directory.

Section 4: Workload Modeling — Benchmarks Should "Look Like Training"

Blindly running IOR defaults often mismatches training experience. Modeling principles:

1. Checkpoint scenario: large-block writes + fsync (-e), shared or sharded files, block size by model weight shard (2–8MB) — mirrors the checkpoint path of Chapter 11;
2. Data loading scenario: sequential large reads (-t 1m~16m) — read throughput is the direct constraint on training data supply;
3. Small-file scenario: when datasets consist of many small files (images, logs, scientific data), mdtest create/stat rates are the real bottleneck; also run IOR -t 4k for random small IO;
4. Concurrency scale: align process count with real training concurrency (e.g. for 512-GPU training, benchmark with at least 128–256 processes) — single-process "very high" numbers don't represent cluster scenarios.

Engineering observation: the first judgment in a performance problem is not "storage is slow" but "which metric says slow". If training is slow, check whether it's data loading (read bandwidth / small files) or checkpoint (write + fsync), then pick the matching benchmark — a wrong benchmark means a wrong investigation direction.

Section 5: Common Bottlenecks & Tuning Paths

Symptom	Diagnosis	Tuning direction
Large-block read below spec	IOR -t 16m read below design	client concurrency, network (RDMA), server striping width & block size
Small-file/metadata slow	mdtest create/stat low	metadata server/target count, directory sharding, aggregation policies
Throughput collapses after fill	checkpoint write + -e collapses	write cache policy, stripe count, OST/MGT balance, tiering (NVMe write cache)
Intermittent client stalls	single-point latency jitter (IOR P-value spread)	network QoS, client mount options (e.g. lustre lazystatfs), server IRQ binding

Note: tuning changes should be one variable at a time, immediately re-measured with the same command — parallel FS variables couple strongly (stripes × clients × network); mixed changes can't be attributed.

Benchmark Checklist

• [ ] Benchmark parameters model real loads (checkpoint / data loading / small files separately)
• [ ] Process count covers cluster scale (at least 1/4 of training concurrency)
• [ ] First acceptance run saved as baseline (bandwidth/IOPS/metadata rates + full command)
• [ ] After every scaling/tuning, rerun the same command and compare
• [ ] On slowdown, pick the matching benchmark by "data loading vs checkpoint"
• [ ] One tuning variable at a time, re-measure immediately

Engineering Takeaways

• Benchmarks are a ruler, not a report card: set the ruler (design spec + acceptance baseline) so scaling and tuning have evidence and troubleshooting has reference.
• Load modeling determines benchmark value: benchmarks that can't surface training pain points (checkpoint fsync, small-file scans) are decoration no matter how nice the numbers.
• Reproducibility > peak: same command, same environment, same baseline — that's what makes numbers comparable; ad-hoc peak tuning has no long-term value.

Next article preview: with storage throughput and metadata meeting spec, large-scale pretraining begins — the next article, MoE Training Engineering: Expert Parallelism & Load Balancing in Practice, focuses on the engineering challenges of training MoE models on clusters.