Elastic Training and Fault Tolerance: torchrun Rendezvous, Restart Semantics, and State Recovery
Three days into a run, one GPU out of 64 throws an ECC error and its process dies. The intuitive expectation is "torchrun will swap the machine out and resume from the last checkpoint". What actually happens: the entire worker group is killed immediately and the job terminates, because the default value of --max-restarts is 0.
Elastic training is widely mistaken for "automatic resume". It is not. What it does is three things: detect membership changes, restart every surviving worker together, and re-form a worker group. Where training resumes from is not the framework's job — your script has to recover it from a checkpoint. This article draws that boundary: who restarts, which quantities change after a restart, what must be in the state, and what to look at when things break.
1. The restart unit is the worker group, not a single process
The official description of failure modes has only three entries, and each one dictates an operational decision:
| Failure type | Documented behavior | What it means for you |
|---|---|---|
| Worker failure | Of n workers, if k≤n fail, all workers are stopped and restarted up to max_restarts times | A single GPU OOM restarts the whole group; this is not hot replacement |
| Agent failure | The local worker group fails; the job manager decides between gang termination and node replacement | torchrun does not decide for you — Slurm or K8s does |
| Node failure | Isomorphic to agent failure | A node disappearing equals its agent disappearing |
There is one more easily missed hard rule: on failures or membership changes, all surviving workers are killed immediately, with no graceful shutdown. So a checkpoint being written at the moment of failure may be only half-written — which is exactly why Section 6 insists on a success criterion.
2. After a restart, neither rank nor world size can be trusted
In torchrun's vocabulary, a node runs LOCAL_WORLD_SIZE workers forming a LocalWorkerGroup; the union of all nodes' LocalWorkerGroups is the WorkerGroup. Under elasticity the docs give two explicit warnings:
RANKis not stable: between restarts, workers on the same node can be assigned a completely different range of ranks. "Rank 7 is always on node 3" is a false assumption.WORLD_SIZEcan change: with--nnodes=min:maxand min≠max, nodes may leave and join, so world size changes and must not be hardcoded.
| Variable | Meaning | When it changes |
|---|---|---|
RANK | Global index of the worker within the worker group | Every re-rendezvous |
LOCAL_RANK | Index within the local group, usually the GPU index | Mostly stable, but never a global identity |
WORLD_SIZE | Total workers in the worker group | On elastic scaling |
GROUP_RANK | Rank of the worker group (0..max_nnodes) | With multiple groups or nodes |
TORCHELASTIC_RESTART_COUNT | Worker-group restarts so far | +1 per restart |
TORCHELASTIC_MAX_RESTARTS | Configured ceiling | Set by --max-restarts |
TORCHELASTIC_RUN_ID | The rendezvous run id of this job | Matches --rdzv-id |
Practical consequence: any identity written to logs, filenames, or an external system should be TORCHELASTIC_RUN_ID plus step — not rank. Rank is only for picking a device within the current run.
3. The real defaults of eight flags explain most "it didn't work"
These defaults come from the argument parser in torch/distributed/run.py of PyTorch v2.11.0, not from memory:
| Flag | Default | What matters |
|---|---|---|
--nnodes | 1:1 | Only min:max gives elasticity; min must be >0 and ≤max, otherwise an assertion fires immediately |
--nproc-per-node | 1 | Accepts auto/cpu/gpu/xpu or an integer; only gpu means one process per GPU |
--rdzv-backend | static | The default is not c10d! Static forms fixed membership by --node-rank, with no elasticity |
--rdzv-endpoint | empty | Under static it falls back to master_addr:master_port; under c10d you must pass host:port |
--rdzv-id | none | Only nodes with the same id join the same worker group |
--max-restarts | 0 | No restarts by default — the direct cause of "the whole job exited" |
--monitor-interval | 0.1 (seconds) | How often the agent polls worker state |
--start-method | spawn | fork/forkserver also allowed; relevant to CUDA context safety |
--master-addr / --master-port | 127.0.0.1 / 29500 | Only meaningful for static rendezvous; ignored with a warning otherwise |
--standalone | off | When set, forces rdzv-backend=c10d, rdzv-endpoint=localhost:0, and rdzv-id=<uuid>, overriding explicit values |
Three symptoms follow directly:
- "I configured 4 restarts and it still exits" — if you used
--standalone,rdzv-idand the endpoint are overridden butmax_restartsis not; if you never wrote--max-restarts, it is 0. - "I wrote
--nnodes=1:4and it still cannot scale" — with the defaultstaticbackend the membership is static; there is no elasticity at all. You must passc10d(or etcd) explicitly. - "Multi-node won't start" — in static mode each machine needs a distinct
--node-rank; in c10d mode every node runs the same command and the node behind--rdzv-endpointacts as coordinator.
4. What rendezvous actually does: four invariants
Rendezvous is not "connect to the master once"; it combines peer discovery, distributed synchronization, and a consistent decision. It has four invariants, and they double as a diagnostic taxonomy:
- Barrier: all nodes block until at least
minnodes have joined; after reachingminthere is an additional small last-call window so that nearly simultaneous arrivals are not excluded; reachingmaxcompletes immediately; there is an overall timeout that fails ifminis never reached, explicitly defined as non-retryable. - Exclusivity: while a completed rendezvous has not been destroyed, late nodes can only register as waiting — they cannot form a parallel worker group. This is why two jobs sharing one endpoint either deadlock or get merged.
- Consistency: once complete, all members agree on the membership set and their ranks; ranks are not stable across rounds.
- Shared store: completion yields a
torch.distributed.Store(TCPStore by default for the c10d backend), used to exchange control-plane information.
The four exception types map onto four diagnostic paths: RendezvousClosedError (closed — retrying is pointless), RendezvousConnectionError (cannot reach the backend), RendezvousStateError (corrupt state), RendezvousTimeoutError (did not assemble in time). Whichever you see tells you whether to check the network, the id, or the node count.
Only three backends exist: c10d (a C10d store, TCPStore by default, no third-party dependency, recommended), etcd-v2, and etcd (legacy, in maintenance mode, slated for removal). The etcd variants require the v2 API enabled; the official test baseline is etcd v3.4.3. Pick a high-bandwidth node for the endpoint; when no port is given it defaults to 29400.
5. The full sequence of a restart
Chaining the mechanisms above, one elastic event looks like this:
- A worker exits, or a node's agent disappears.
- Agents on surviving nodes notice (polling every
monitor-interval, 0.1 s by default). - All surviving workers are killed immediately — no graceful exit, no waiting for the current step to finish.
- Surviving nodes (plus newly joined nodes under elasticity) re-enter rendezvous; once
minis reached a new membership forms and the rest register as waiting. - The new WorkerGroup starts with a new RANK and WORLD_SIZE, and your script re-executes from
main. - The first thing
mainmust do is load a checkpoint; otherwise it is equivalent to restarting from step 0.
Step 5 is the most overlooked: a restart is not "resume the stack at the breakpoint", it is "start processes again, import again, build the model again". Recovery logic therefore belongs outside the training loop, and must handle "no checkpoint yet" gracefully.
Steps 3 and 6 together give a quantitative relation: checkpoint interval = the maximum work you accept losing per failure. Asynchronous saves keep the write off the critical path, but the upload in flight at kill time may be incomplete — deciding whether that checkpoint is usable means waiting for the upload future and having .metadata on disk (see Distributed Checkpointing in Practice).
6. Four contracts on the script side
First, decorate the entry point with @record. The documented behavior: uncaught exceptions in workers are written to TORCHELASTIC_ERROR_FILE, the agent aggregates error files across children and reports the one with the smallest timestamp (i.e. the first error) as the root cause. Without the decorator, the summary contains only an exit code and no traceback — multiplying debugging cost.
import os
import torch
import torch.distributed as dist
from torch.distributed.elastic.multiprocessing.errors import record
@record
def main():
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank) # must precede init_process_group
dist.init_process_group(backend="nccl") # RANK/WORLD_SIZE/MASTER_* injected by torchrun
train()
if __name__ == "__main__":
main()Second, do not parse ranks yourself. torchrun injects RANK/WORLD_SIZE/MASTER_ADDR/MASTER_PORT; init_process_group reads them from the environment by default.
Third, mind the --local-rank spelling change. Since PyTorch 2.0, torchrun passes --local-rank=<rank> (hyphen); the old --local_rank (underscore) is only kept for compatibility. A parser that registers only the underscore version fails with error: unrecognized arguments: --local-rank=<rank>. Register both:
parser.add_argument("--local-rank", "--local_rank", type=int)Fourth, fix the construction order. Every restart builds objects from scratch: set_device → init_process_group → model → wrap in DDP/FSDP → optimizer → load state → dataloader (call set_epoch before iterating). A wrong order produces silent drift such as "weights loaded but optimizer momentum is fresh" or "sampler shuffle does not match the checkpoint".
Also, side effects in a rank==0 branch must be idempotent: deleting old checkpoints, clearing shared temp directories, writing a "training started" row to an external system — all of these run again after a restart.
7. What actually has to be recovered
"Saving model weights" is far from enough. A resumable checkpoint carries at least six classes of state; miss one and you get a matching symptom:
| State | Symptom if missing | Note |
|---|---|---|
| Model weights | Starts over | Save sharded via DCP, which supports cross-topology resharding |
| Optimizer state (incl. step) | Adam momentum resets, loss steps up | Optimizer step also locates the scheduler |
| LR scheduler | LR jumps back to initial | Can be omitted if derived from step |
| Data position (epoch + intra-epoch offset) | Samples replayed or skipped | DistributedSampler.set_epoch only fixes epoch-level shuffle |
| RNG states (torch/cuda/python/numpy) | Dropout and augmentation irreproducible | Determines whether a rerun can be aligned |
| Gradient-accumulation boundary | Accumulation window misaligned, effective batch changes | An unfinished window must be discarded on restart |
Data position deserves its own paragraph: DistributedSampler.set_epoch(epoch) only guarantees that shuffle order for the N-th epoch is identical; it does not offer "batch 137 in the middle of an epoch". If checkpoints are taken per step, you must record samples consumed and skip that many after rebuilding the dataloader. Not skipping replays data (loss curve shows a repeated segment); skipping incorrectly drops samples (loss degrades suddenly).
A sufficient state dict looks like this (field names are yours to choose; the point is that all of them are present):
ckpt = {
"model": model.state_dict(), # or DCP sharded save
"optim": optim.state_dict(),
"sched": sched.state_dict(),
"step": global_step,
"epoch": epoch,
"samples_consumed": samples_seen, # for intra-epoch resume
"rng": {
"torch": torch.get_rng_state(),
"cuda": torch.cuda.get_rng_state_all(),
},
}8. max_restarts counts restarts, not failures
This directly affects capacity planning. --max-restarts=3 means "the worker group may be restarted at most 3 times", not "at most 3 GPUs may fail". One restart may be triggered by one worker failing or by eight failing at once; the counter still increments by one. Observe it via TORCHELASTIC_RESTART_COUNT and TORCHELASTIC_MAX_RESTARTS.
The docs classify errors into four categories, and retry policy should follow them:
| Category | Typical source | Retry? |
|---|---|---|
| User Error | Invalid arguments (e.g. min>max), bad config | No — retrying just fails again |
| Worker Failure | Exceptions in the training process, OOM, NCCL errors | Yes, but bounded |
| Platform Error | Failures of the agent itself | Hand to the job manager |
| Infra Error | Host, network, storage failures | Hand to the job manager / alerting |
Setting max_restarts very high turns a deterministic bug into a restart storm: each restart rebuilds process groups, re-rendezvouses, re-reads checkpoints, burns tens of GPU-minutes, and fails anyway. Bounded retries + error classification + persisting the failing sample is closer to production-grade fault tolerance than "infinite elasticity".
9. Parameter templates for three common shapes
Single-node multi-GPU debugging — --standalone picks a random port and a generated id:
torchrun --standalone --nnodes=1 --nproc-per-node=gpu train.pyFixed-size fault tolerance (no elasticity, tolerates 3 restarts) — explicit c10d and id:
torchrun \
--nnodes=$NUM_NODES --nproc-per-node=8 \
--max-restarts=3 \
--rdzv-backend=c10d --rdzv-id=$JOB_ID \
--rdzv-endpoint=node1.example.com:29400 \
train.pyElastic scaling (1 to 4 nodes, tolerates 3 membership changes or failures):
torchrun \
--nnodes=1:4 --nproc-per-node=8 \
--max-restarts=3 \
--rdzv-backend=c10d --rdzv-id=$JOB_ID \
--rdzv-endpoint=node1.example.com:29400 \
train.pyIn multi-node settings every node runs the same command (static mode is the exception and needs a distinct --node-rank). Let the job manager launch torchrun as the entry point; do not start one torchrun per node competing for the same rdzv-id — that is not distributed training, it is two sets of processes waiting on the same barrier.
When running several independent jobs on one host, use different ports (or --rdzv-endpoint=localhost:0 to let the OS pick a free one). Otherwise the two jobs get merged into a single worker group, with symptoms of doubled world size and hanging collectives.
10. Troubleshooting by symptom
- Stuck in rendezvous: verify
--rdzv-endpointis reachable from every node,--rdzv-idis identical everywhere, and--nnodes'mindoes not exceed the actual node count. ARendezvousTimeoutErroris documented as non-retryable — release the resources and resubmit rather than waiting in place. Address already in use, or two jobs mysteriously merged: same-host jobs sharing a port, or reusing onerdzv-id. Change both the port and the id.- Restart hangs in
init_process_group: commonly stale processes holdingMASTER_PORT, or a network policy blocking the new connections. Check local leftovers and port usage first, then follow whateverRendezvousConnectionErrorpoints at. - Loss jumps after a restart: first check whether only weights were restored (missing optimizer/scheduler) — that gives a step-shaped jump; then check whether data position was replayed — that gives a repeated curve segment.
- Restart storm:
TORCHELASTIC_RESTART_COUNTclimbing while every failure lands near the same step is almost certainly a deterministic bug (corrupt data, wrong shape, checkpoint version mismatch). Stop the job instead of raising the restart ceiling. unrecognized arguments: --local-rank: the parser only registers--local_rank; register both names as in Sections 3 and 6.
11. Reproducible acceptance checks
None of this needs a large cluster. Three checks:
- Verify the default: omit
--max-restarts,killone worker manually, and observe the whole group exit with noTORCHELASTIC_RESTART_COUNT(the processes never restarted). - Verify restart semantics: add
--max-restarts=3,killone worker again, and observe all workers killed together, the script re-executing frommain, the restart count incrementing, and training resuming from the checkpointed step. - Verify isolation: start two
--standalonejobs (random ports) and two c10d jobs on the same port, then compare world size and collective behavior to see what a "merged job" looks like.
By now the boundary should be clear: torchrun owns membership detection, whole-group restart, and re-rendezvous; your script owns recovering state from a checkpoint. The former is a framework capability, the latter an engineering responsibility, and the seam between them — checkpoint integrity and data position — is what decides whether a failure costs minutes or days.
Further reading: Distributed Checkpointing in Practice: DCP Shard Layout, Async Save and Resharding, FSDP2 in Practice, Data Pipeline Tuning.