Skip to content

Distributed Checkpointing in Practice: DCP's Shard Layout, Async Save, and Resharding ​

Summary: torch.distributed.checkpoint (DCP) is not about "saving a bit faster". It fixes two structural flaws: torch.save requires gathering sharded parameters into one full state_dict, which blows past per-GPU memory, and what it writes is welded to the parallel topology of that moment, so it cannot be loaded on a different cluster. DCP's own definition is two sentences — each rank writes only its shard, and load-time resharding is supported. This article turns those two sentences into operational contracts, a directory layout, a parameter budget, and a validation procedure.

1. The two points where torch.save actually fails ​

Many teams attribute slow checkpoints to "not enough storage bandwidth", swap in NVMe, widen striping, and see no change. The real cause is the call shape: torch.save receives an already assembled full state_dict. Under FSDP or tensor parallelism that full state_dict either requires an all-gather (doubling memory and triggering a cluster-wide collective) or is assembled only on rank 0 (a single-process memory and bandwidth bottleneck). The former OOMs at 70B scale; the latter funnels the write bandwidth of an entire cluster through one process.

DCP's documentation states the differences plainly:

  • It produces multiple files per checkpoint, with at least one per rank;
  • It operates in place, meaning that the model should allocate its data first and DCP uses that storage instead.

The second point is the key to every pitfall below: DCP never moves your data for you — it writes into tensors you already allocated. That is also why load requires building the model under the new topology and moving it to the device before calling load.

2. Three core contracts ​

ContractWhat the docs saySymptom when violated
In-place semantics"operates in place… model should allocate its data first"Parameters unchanged after load, or load errors out
Identical keys"Each rank must have the same keys in their state_dict… Mismatched keys may result in hangs or errors"A hang, which is far harder to debug than an error
Stateful protocolObjects with state_dict/load_state_dict get the former called before save and the latter after loadCustom module state silently lost

The load side carries three documented warnings that must each be honoured:

  1. All tensors must already be allocated on their destination device ("All tensors in state_dict must be allocated on their destination device prior to calling this function");
  2. Non-tensor data is loaded with torch.load() and modified in place on the state_dict — so non-tensor fields (epoch, dataloader state) belong in the state_dict, not in a sidecar json;
  3. You must call load_state_dict on the root module, otherwise load post-processing and non-tensor data never propagate.

Point 3 is the usual cause of "load succeeded but results are wrong": dcp.load wrote into the state_dict, but the module only takes over after load_state_dict.

3. What actually happens during a save ​

The six-step save, why .metadata appears, and what async_save does not cover

DCP splits saving into planner and storage layers, with a fixed documented call sequence:

  1. all ranks: set checkpoint_id;
  2. all ranks: set_up_storage_writer();
  3. all ranks: prepare_local_plan() — each rank computes which slices it will write;
  4. coordinator only: prepare_global_plan() — aggregate, decide ownership and deduplication;
  5. all ranks: write_data();
  6. coordinator only: finish() — write the metadata and mark this checkpoint successful.

Step 6 explains the .metadata file. For FileSystemWriter the documented shape is:

  • one file per write request;
  • when rank coordination is enabled, one global .metadata; when it is not (use_collectives=False), one __{rank}.metadata per rank;
  • its assumptions are that the checkpoint path is empty or non-existent, and that file creation is atomic.

"One file per write request" is why DCP checkpoints routinely contain tens of thousands of small files. That connects straight to the storage chapter: one large-model checkpoint is a small-file storm, hitting the metadata service and capacity quotas simultaneously — exactly the inode curve discussed in Capacity and Quota Operations.

4. Why get_state_dict is mandatory ​

Hand model.state_dict() to DCP under mixed FSDP/DDP, or with tensor/pipeline parallelism, and two things break: model keys have inconsistent FQNs, and optimizer state_dicts key on parameter IDs rather than names. The documented purpose of the API covers both: get_state_dict returns keys as canonical FQNs (the names named_parameters() would give with no parallelism applied) and converts optimizer parameter IDs to the same FQNs, so results can be passed straight to dcp.save() / dcp.load() and can be "resharded with a different number of trainers and/or different parallelisms".

python
from torch.distributed.checkpoint.state_dict import get_state_dict, set_state_dict
import torch.distributed.checkpoint as dcp

model_sd, optim_sd = get_state_dict(model, optimizers=[optim])
sd = {"model": model_sd, "optim": optim_sd}
dcp.save(sd, checkpoint_id=f"/ckpt/step-{step}")

One timing constraint on the load side is easy to miss: set_state_dict (and set_optimizer_state_dict) may only be called before backward() or after step(), otherwise optimizer state is not initialised correctly. When resuming, "load before backward" is a hard requirement, not a stylistic choice.

Wrapping model and optimizer in a Stateful object — the pattern used in the DCP examples — is often preferable, because dcp.load then calls load_state_dict for you and removes the failure mode described in section 2.

5. What FileSystemWriter's six parameters really mean ​

Documented signature and defaults:

python
FileSystemWriter(path, single_file_per_rank=True, sync_files=True,
                 thread_count=1, per_thread_copy_ahead=10000000,
                 cache_staged_state_dict=False, overwrite=True,
                 serialization_format=SerializationFormat.TORCH_SAVE)
ParameterDefaultEffectRecommendation
single_file_per_rankTrueOne file per rank vs one file per write requestKeep True: file count drops from "tens of thousands" to "thousands per rank", greatly reducing metadata pressure
sync_filesTrueWhether to fsync after writingKeep True in production; only disable for throughput benchmarking, and only with "power loss means data loss" stated explicitly
thread_count1Writer threads per rankDo not raise it while a single sequential file already saturates bandwidth; for many small files, try 2–4 together with per_thread_copy_ahead
per_thread_copy_ahead10000000 (~9.5 MiB)Look-ahead copy window in bytesRaising it alongside thread count smooths tail latency, at the cost of host memory
cache_staged_state_dictFalseCache the staged copy for reuseEnable only when saving the same step repeatedly; otherwise it holds a full extra copy
overwriteTrueWhether to overwrite an existing targetSet False in production: combined with per-step directories, a repeated write fails loudly instead of silently clobbering a good checkpoint

overwrite=False is not fussiness. DCP assumes the path is empty or non-existent; allowing overwrite means a half-written failed checkpoint lands in the same directory as the previous good one, metadata and shards corrupt each other, and you only find out when recovery fails.

6. What async_save actually makes asynchronous ​

The documentation describes two steps: first de-stage the state_dict onto staging storage (CPU memory by default), then call save in a separate thread. Note the order — the asynchronous part is writing, not copying. The GPU→CPU copy happens at the moment you call async_save, unless you explicitly use asynchronous staging.

The return value is AsyncSaveResponse with two futures, and confusing them leads to wrong "did the checkpoint succeed" conclusions:

  • staging_completion: the local copy is complete — past this point training may safely keep mutating parameters;
  • upload_completion: the checkpoint is fully written — only this one means the save succeeded.

Two documented constraints: the feature is experimental, and "MUST CALL CLOSE AFTER LAST CHECKPOINT IS SAVED" — without it you leak the thread pool and cached storage blocks inside the default stager.

python
resp = dcp.async_save(sd, checkpoint_id=f"/ckpt/step-{step}")
resp.staging_completion.result()      # copy is safe, keep training
# ... train a few steps, then at a health check or exit:
resp.upload_completion.result()       # actually written; failures surface here
stager.close()

async_checkpointer_type accepts THREAD (default) and PROCESS. THREAD saves memory and shares page tables but GIL and host copies interfere; PROCESS isolates cleanly at the cost of a cross-process transfer and extra memory.

7. The memory bill for async saving ​

This is the most under-estimated part. The documentation defines staging as giving the AsyncStager a chance to create a "training-safe" representation of the state dict — any updates to module data after staging completes must not be reflected in it; the default implementation is a full copy on CPU RAM.

The bill is therefore explicit:

  • Host memory must hold at least one full checkpoint (model plus optimizer state, at save precision);
  • use_pinned_memory=True (default) is faster but uses more memory, as StagingOptions documents;
  • use_shared_memory=True (default) enables efficient IPC to a checkpoint process and only pays off in PROCESS mode;
  • DefaultStager is not thread-safe — one instance per thread, or external synchronisation.

The four StagingOptions defaults are documented as use_pinned_memory=True, use_shared_memory=True, use_async_staging=True, use_non_blocking_copy=True, and the docs note that CUDA-dependent features raise an exception when CUDA is unavailable — so in a CPU-only debugging environment either disable them or expect the exception.

Do the arithmetic: a 70B model (bf16 weights ≈ 140 GB) with two Adam moments (≈ 560 GB at fp32) is a checkpoint in the hundreds of GB, and no single host holds that copy. That is why, at this scale, async_save either needs PROCESS plus shared memory plus a chunking strategy, or you go back to synchronous saving and fit the save inside the checkpoint interval. Size the host memory first, then decide on async — never the reverse.

8. Resharding: the boundary and three practices ​

The documented capability is one sentence: DCP handles load-time resharding, which enables saving in one cluster topology and loading into another. That works because of canonical FQNs from section 4 — the saved metadata records global shapes and slices of logical tensors, not "rank 3's second file". On restore each rank reads only what it needs ("each rank will try to read the least amount of data necessary to fulfill the requested state_dict").

Three practices:

  1. Always save via get_state_dict, never a hand-written model.state_dict() — topology independence comes from precisely this;
  2. On load, build the model under the new topology and to(device) first, then dcp.load — in-place semantics mean the reverse order writes into an empty table;
  3. allow_partial_load defaults to False, documented as "will raise a runtime error if a key is present in state_dict, but not in the checkpoint". When converting between LoRA and full-parameter checkpoints, or after adding a head, turn it on and print the missing-key list; otherwise you get a model that runs but has randomly initialised layers — the most expensive class of silent failure.

Two format converters are worth remembering: dcp_to_torch_save and torch_save_to_dcp, both carrying the same documented warning — run them on a single rank to avoid OOM. And online loading with resharding straight from torch.save is possible via BroadcastingTorchSaveReader plus DynamicMetaLoadPlanner: the coordinator reads the whole checkpoint, then broadcasts and shards it, which the docs say avoids loading the entire checkpoint on each rank and thus reduces OOM risk, at the cost of one broadcast.

9. Four hard rules for production ​

  1. One directory per step, never reuse a path: /ckpt/step-1000/, /ckpt/step-2000/, together with overwrite=False. A half-finished save can then never enter a good checkpoint's directory.
  2. The success criterion is .metadata present plus upload_completion resolved, not "there are files in the directory". The directory appears early by design, given DCP's write sequence.
  3. Validate by sampling after saving: spin up a small process group (say 1/8 scale) and run the same get_state_dict + dcp.load with allow_partial_load=False. Missing keys surface here instead of three days later during a resume.
  4. Align two numbers with the storage team: file count and total bytes per checkpoint. The former drives metadata service pressure (see the inode watermarks in Capacity and Quota Operations); the latter drives striping and bandwidth (see Lustre client tuning). single_file_per_rank=True is the first choice that makes both numbers look better.

10. Five pitfalls ​

  1. Assuming async_save also makes copying asynchronous: it stages first (a full GPU→CPU copy) and only then writes in the background; the peak lands at call time, not in the background.
  2. Treating staging_completion as success: that only means the copy is ready; upload_completion means written. Waiting on the former loses entire checkpoints on power loss or a full disk.
  3. Forgetting load_state_dict on the root module: dcp.load only fills the state_dict; non-tensor data and post-processing do not propagate automatically.
  4. Hand-writing model.state_dict() instead of get_state_dict: optimizer parameter IDs and post-parallel FQNs break resharding outright, and the checkpoint will not load on a different topology.
  5. Never calling close(): the docs require it after the last checkpoint; otherwise thread pools and cached storage blocks leak and long-running jobs slowly eat host memory.

AI-HPC Expert · Contact: openaihpc@gmail.com