Data Integrity in Practice: End-to-End Checksums, Scrub, and Silent Corruption Response
Summary: The most expensive assets in a training cluster are not the GPUs but the checkpoints that took weeks to produce and the corpora that took weeks to clean. RAID and replicas answer "the disk died but the data survived"; they do not answer "the disk is fine but the data is already wrong" — that is silent data corruption. Defending against it requires an end-to-end checksum chain from application to media, plus periodic scrubs. Most clusters conflate the two, so checksums stay enabled and scrubs still run while bad blocks remain undetected.
1. Why silent corruption slips through
Silent corruption means data read back differs from what was written, while the read itself reports no error. It can originate in four places: the media (sector degradation, lost write cache), controllers and cables, file system or RAID firmware (a checksum computed wrong or bypassed), and the software path (buffer misalignment, incorrect read-modify-write).
RAID and replicas share one precondition: they only compare against checksums they compute themselves, and only while data passes through them. A bad block that is never read is invisible to RAID forever; a controller that returns wrong data but reports success is invisible to the file system; a RAID card that loses writes in an unprotected cache can, during rebuild, use stale data to "repair" fresh data.
So accept the engineering fact: no single layer guarantees integrity. The workable approach is layered checksums, where each layer does what it is good at, plus periodic active reads — because unread data cannot be known to be bad.
2. What each of the three layers covers
| Layer | Coverage | Typical mechanism | What it misses |
|---|---|---|---|
| Media/device | Sector level | On-disk ECC, RAID parity, end-to-end T10 PI | Controller/firmware errors, lost write cache, blocks never read |
| File system | File data blocks | Block checksums, mirror/EC redundancy, scrub | Data that was written wrong yet is internally consistent |
| Application | Semantic correctness | sha256 manifests for checkpoints/shards, verify-after-write | Requires application work; files outside the manifest are uncovered |
The layers stack rather than substitute. A file-system scrub finds media degradation but cannot find "this checkpoint was already corrupt when written"; an application manifest catches logical corruption but costs extra I/O and upkeep. For AI clusters the sensible split is routine file-system scrubs plus manifest verification for key artifacts, with hardware PI/RAID handling the middle.
3. ZFS: what checksum and scrub really mean
ZFS documents this most precisely, and understanding it makes every other file system easier to reason about.
The checksum property selects the algorithm. Per the official zfsprops manual: the default on "automatically selects an appropriate algorithm" (currently fletcher4, which may change in future releases); off disables integrity checking on user data, and the manual states plainly that "Disabling checksums is NOT a recommended practice". Options include fletcher2, fletcher4, sha256, sha512, skein, edonr and blake3, the last four requiring the corresponding pool features. One practical detail: changing the property only affects newly written data, so a switch is not retroactive — historical blocks adopt the new algorithm only when rewritten or migrated.
Quantify the cost before choosing: the chksum_bench kstat (/proc/spl/kstat/zfs/chksum_bench on Linux) reports measured speeds of each algorithm on the running system. Assuming "sha256 must be slow" is often wrong, especially on CPUs with acceleration instructions.
Two redundancy knobs that are easy to overlook:
copies=1|2|3— additional copies beyond pool-level redundancy (mirror/raidz), placed on different disks where possible;redundant_metadata=all|most|some|none, defaultall. The manual states the consequences: withmost, "at worst about 1000 blocks (of recordsize bytes each) of user data can be lost if a single on-disk block is corrupt"; withnone, "an entire dataset can be lost". This is not a free performance knob.
Scrub semantics (zpool-scrub(8)):
- A normal scrub examines all data in the pool and verifies each block's checksum; a thorough scrub (
-t) additionally decrypts and/or decompresses blocks as they are read, to catch the rare corruption "where the checksum matches the data, but decryption and/or decompression fails" (encrypted datasets need keys loaded); - Scrub differs from resilver: resilver only examines data ZFS knows is out of date, whereas scrubbing examines all data to discover silent errors from hardware faults or disk failure;
- For mirror, raidz or draid devices, ZFS automatically repairs damage found during a scrub;
- A scrub runs in two parts: metadata scanning, which sorts blocks into large sequential ranges, then block scrubbing. That is why
zpool statusshows both a "scanned" and an "issued" rate (the documented example is403M / 405M scanned at 100M/s, 68.4M / 405M issued at 10.0M/s) — metadata scanning is fast, actual data reading is not, and they are different metrics; - Only one scrub or resilver runs at a time; a scrub cannot start until an in-progress resilver completes;
- A scrub can be paused (
-p), stopped (-s) and resumed; pause state is checkpointed to disk, so after a reboot or export/import it stays paused until explicitly resumed; -S/-Erestrict the date range (formatYYYY-MM-DD HH:MM), useful for incremental policies such as "verify only last month's artifacts";-Ccontinues from the last saved txg;-escrubs only files already reported byzpool status -v(requiring thehead_errlogfeature and at least one prior full scrub);- Periodic scrubs need no custom cron: official systemd timers exist, e.g.
systemctl enable zfs-scrub-weekly@rpool.timer --now, plus a monthly variant.
Locating bad blocks: zpool status -v lists affected paths. Note the documented exception: for encrypted datasets with keys not loaded, an unrepairable checksum error cannot include the file name in the verbose report — which is why the runbook must say "load keys first, then collect evidence".
4. Lustre: what network checksums and LFSCK actually cover
The Lustre manual promises exactly two things about integrity:
- Network data integrity: all data sent from the client to the OSS carries a checksum, protecting against corruption during transfer. This protects links and NICs, not data at rest.
- LFSCK: an online distributed consistency checker that can run while the file system is in use. It verifies the FID stored with each directory entry and the linkEA of each inode, regenerating them when invalid, and restores consistency between storage components after a major file system error. It checks metadata consistency, not file data content.
That boundary matters: treating LFSCK as "Lustre's scrub" and concluding the data is fine is simply wrong. Lustre does not scan data content; data-layer integrity depends on the backing file system.
What you actually do at the data layer depends on the OST backend:
- ldiskfs backend: offline
e2fsck. The manual gives a concrete expectation — reasonable check times for valid file systems are "5-30 minutes per TiB", but may "increase significantly if substantial errors are detected and need to be repaired". This is why scrub windows must be sized for the worst case, not the ideal one. - ZFS backend:
zpool scrub, with exactly the semantics described above.
Check state before acting. lfs df appends a status flag to each target's line, and two are directly integrity-related: D means degraded (a failed drive in the RAID device or an ongoing reconstruction; new allocations avoid the target, but existing files still read from it), and R means read-only — the manual's wording is that the target file system was marked read-only due to corruption detected by ldiskfs or ZFS, no modifications are allowed, and it needs to be unmounted and e2fsck or zpool scrub run to repair. The right response to an R flag is to stop the bleeding and preserve the state, not to reboot and hope.
5. BeeGFS: online checking is allowed, but official docs discourage automatic repair
BeeGFS ships beegfs-fsck as its file system checking tool (listed among administrator utilities in the official installation documentation), and the docs include both a "BeeGFS File System Check" section and a "Suggested Procedure for a Large File System" — following the official procedure beats inventing one.
Two documented positions are worth quoting:
- Version 8.3 improved the robustness of running fsck online (ignoring
ENOENTerrors when a storage service's chunk fetcher reads recently deleted chunks), yet the same note says "it is still recommended to execute fsck in read-only mode on online systems"; - Version 8.2 re-enabled orphan chunk deletion as a repair action, but behind an interactive confirmation prompt (typing a confirmation phrase), and states that "when running fsck with the automatic repair option (discouraged), orphan chunk deletion is always skipped".
In other words, BeeGFS's stance is: checking can happen online, repairing should happen with a human watching. That is a different philosophy from ZFS's repair-on-detection — the former is conservative (avoid deleting the wrong thing), the latter is faster (trusting redundancy). Worth writing into the runbook rather than discovering, mid-incident, that nobody knew automatic repair was disabled.
6. Application layer: manifests for key artifacts
File-system layers cannot see logical corruption. In AI clusters the artifacts that most need this layer are checkpoints and cleaned dataset shards.
The minimal viable pattern is "hash on write, verify before use":
# 1) After the write completes, emit a manifest next to the data (back it up with the data)
find /ckpt/run-20261119 -type f -print0 \
| xargs -0 sha256sum > /ckpt/run-20261119.sha256
# 2) Verify before resuming training; any FAILED line must stop the job
sha256sum -c /ckpt/run-20261119.sha256
# 3) Verify before pruning old checkpoints; keep the previous one if verification fails
sha256sum -c /ckpt/run-20261118.sha256 && rm -rf /ckpt/run-20261117Three disciplines matter more than the script:
- The manifest shares the data's lifecycle — backing up data without its manifest breaks the chain;
- A verification failure must be a hard failure, never "retry and continue", or corrupt data enters the next run;
- Deletion always follows verification, and a failed verification keeps the last good checkpoint — the single rule that saves you in resume-from-checkpoint scenarios.
State the cost too: full sha256 is pure CPU and I/O, and on multi-TB checkpoints it noticeably slows the save path. The pragmatic split is tiered: full verification for primary checkpoints, size-and-count checks for intermediates, or sampling for shards.
7. After a bad block is found: locate, isolate, repair, preserve evidence
The order is not optional, and "preserve evidence" is the step most often skipped, which is why the same failure repeats.
- Locate:
zpool status -vfor paths and vdevs;lfs df'sD/Rflags then the specific OST for Lustre;beegfs-fsckoutput for inode/chunk in BeeGFS. - Isolate: stop allocating new data to the affected target (Lustre's degraded flag already does this) and pull the affected files out of the training queue so they stop reaching the GPUs.
- Repair: prefer redundancy (mirror/raidz/EC rebuild) and restore from backup; ZFS repairs redundant blocks during scrub automatically. Do not let a tool "do its best" when no second copy exists — most repair tools aim for readability, not correctness.
- Preserve evidence: keep
zpool statusoutput, the list of failing files, timestamps, and the load, temperature and power events of the moment. Root causes of silent corruption often live outside the media (power delivery, backplanes, HBA firmware, memory ECC events); without the scene you will hit the same wall again.
8. Operating rules: frequency, windows, monitoring
- Frequency follows data change volume, not the calendar. Clusters with heavy daily deltas and aging disks need weekly scrubs as a floor; read-only archive pools can be monthly. ZFS's official weekly/monthly systemd timers are designed around exactly this granularity.
- Windows: a scrub is a full sequential read and will compete with training I/O. The effective levers are limited but real: avoid checkpoint peaks, restrict scope with
-S/-Efor incremental verification, and pause with-pto resume outside the window (pause survives reboot). - Metrics: scrub completion and duration,
repairedbytes, checksum error counts, and new failing files per scrub. "Repaired bytes staying above zero" is the alert that matters — it means the media is degrading continuously, not suffering a one-off incident. - Scrub is not backup: scrub repairs only blocks that have redundancy; it cannot help when every replica is bad, when two blocks in an EC stripe fail, or when someone deletes data by mistake. Scrub and backup are separate obligations.
9. Five common misconceptions
- "RAID with parity prevents silent corruption": parity works only when data is read and the parity was computed correctly; blocks never read are never known to be bad.
- "Running LFSCK proves the data is fine": LFSCK checks metadata consistency, not data content.
- "More frequent scrubs are always better": a scrub is a full read; too frequent and it steadily eats training bandwidth while accelerating the discovery of problems on aging disks — schedule by data change instead.
- "Stronger checksums are always better": stronger algorithms cost CPU and pool features, and changing the property only affects new data; measure with
chksum_benchfirst. - "Online fsck can just repair automatically": BeeGFS officially discourages automatic repair, and ZFS's automatic repair requires redundancy. Confirm redundancy and backups before repairing.
To close: integrity comes from layered checksums, periodic active reads, and preserving evidence before repair — not from a RAID card or a single command.
Related Articles
- GPUDirect Storage in Practice: A Direct Data Path from NVMe into GPU Memory
- Training Data Storage Selection: Object Store, Parallel File Systems & Local NVMe
- Lustre Client Tuning: Striping Layouts, Multi-NIC LNet and I/O Parameters
- Capacity and Quota Operations: Diagnosing inode Exhaustion, the Two Watermarks, and Growth Forecasting