Skip to content

Elastic Training and Fault Tolerance: torchrun Rendezvous, Restart Semantics, and State Recovery ​

Three days into a run, one GPU out of 64 throws an ECC error and its process dies. The intuitive expectation is "torchrun will swap the machine out and resume from the last checkpoint". What actually happens: the entire worker group is killed immediately and the job terminates, because the default value of --max-restarts is 0.

Elastic training is widely mistaken for "automatic resume". It is not. What it does is three things: detect membership changes, restart every surviving worker together, and re-form a worker group. Where training resumes from is not the framework's job — your script has to recover it from a checkpoint. This article draws that boundary: who restarts, which quantities change after a restart, what must be in the state, and what to look at when things break.

1. The restart unit is the worker group, not a single process ​

The official description of failure modes has only three entries, and each one dictates an operational decision:

Failure typeDocumented behaviorWhat it means for you
Worker failureOf n workers, if k≤n fail, all workers are stopped and restarted up to max_restarts timesA single GPU OOM restarts the whole group; this is not hot replacement
Agent failureThe local worker group fails; the job manager decides between gang termination and node replacementtorchrun does not decide for you — Slurm or K8s does
Node failureIsomorphic to agent failureA node disappearing equals its agent disappearing

There is one more easily missed hard rule: on failures or membership changes, all surviving workers are killed immediately, with no graceful shutdown. So a checkpoint being written at the moment of failure may be only half-written — which is exactly why Section 6 insists on a success criterion.

2. After a restart, neither rank nor world size can be trusted ​

In torchrun's vocabulary, a node runs LOCAL_WORLD_SIZE workers forming a LocalWorkerGroup; the union of all nodes' LocalWorkerGroups is the WorkerGroup. Under elasticity the docs give two explicit warnings:

  • RANK is not stable: between restarts, workers on the same node can be assigned a completely different range of ranks. "Rank 7 is always on node 3" is a false assumption.
  • WORLD_SIZE can change: with --nnodes=min:max and min≠max, nodes may leave and join, so world size changes and must not be hardcoded.
VariableMeaningWhen it changes
RANKGlobal index of the worker within the worker groupEvery re-rendezvous
LOCAL_RANKIndex within the local group, usually the GPU indexMostly stable, but never a global identity
WORLD_SIZETotal workers in the worker groupOn elastic scaling
GROUP_RANKRank of the worker group (0..max_nnodes)With multiple groups or nodes
TORCHELASTIC_RESTART_COUNTWorker-group restarts so far+1 per restart
TORCHELASTIC_MAX_RESTARTSConfigured ceilingSet by --max-restarts
TORCHELASTIC_RUN_IDThe rendezvous run id of this jobMatches --rdzv-id

Practical consequence: any identity written to logs, filenames, or an external system should be TORCHELASTIC_RUN_ID plus step — not rank. Rank is only for picking a device within the current run.

3. The real defaults of eight flags explain most "it didn't work" ​

These defaults come from the argument parser in torch/distributed/run.py of PyTorch v2.11.0, not from memory:

FlagDefaultWhat matters
--nnodes1:1Only min:max gives elasticity; min must be >0 and ≤max, otherwise an assertion fires immediately
--nproc-per-node1Accepts auto/cpu/gpu/xpu or an integer; only gpu means one process per GPU
--rdzv-backendstaticThe default is not c10d! Static forms fixed membership by --node-rank, with no elasticity
--rdzv-endpointemptyUnder static it falls back to master_addr:master_port; under c10d you must pass host:port
--rdzv-idnoneOnly nodes with the same id join the same worker group
--max-restarts0No restarts by default — the direct cause of "the whole job exited"
--monitor-interval0.1 (seconds)How often the agent polls worker state
--start-methodspawnfork/forkserver also allowed; relevant to CUDA context safety
--master-addr / --master-port127.0.0.1 / 29500Only meaningful for static rendezvous; ignored with a warning otherwise
--standaloneoffWhen set, forces rdzv-backend=c10d, rdzv-endpoint=localhost:0, and rdzv-id=<uuid>, overriding explicit values

Three symptoms follow directly:

  1. "I configured 4 restarts and it still exits" — if you used --standalone, rdzv-id and the endpoint are overridden but max_restarts is not; if you never wrote --max-restarts, it is 0.
  2. "I wrote --nnodes=1:4 and it still cannot scale" — with the default static backend the membership is static; there is no elasticity at all. You must pass c10d (or etcd) explicitly.
  3. "Multi-node won't start" — in static mode each machine needs a distinct --node-rank; in c10d mode every node runs the same command and the node behind --rdzv-endpoint acts as coordinator.

4. What rendezvous actually does: four invariants ​

Rendezvous is not "connect to the master once"; it combines peer discovery, distributed synchronization, and a consistent decision. It has four invariants, and they double as a diagnostic taxonomy:

  • Barrier: all nodes block until at least min nodes have joined; after reaching min there is an additional small last-call window so that nearly simultaneous arrivals are not excluded; reaching max completes immediately; there is an overall timeout that fails if min is never reached, explicitly defined as non-retryable.
  • Exclusivity: while a completed rendezvous has not been destroyed, late nodes can only register as waiting — they cannot form a parallel worker group. This is why two jobs sharing one endpoint either deadlock or get merged.
  • Consistency: once complete, all members agree on the membership set and their ranks; ranks are not stable across rounds.
  • Shared store: completion yields a torch.distributed.Store (TCPStore by default for the c10d backend), used to exchange control-plane information.

The four exception types map onto four diagnostic paths: RendezvousClosedError (closed — retrying is pointless), RendezvousConnectionError (cannot reach the backend), RendezvousStateError (corrupt state), RendezvousTimeoutError (did not assemble in time). Whichever you see tells you whether to check the network, the id, or the node count.

Only three backends exist: c10d (a C10d store, TCPStore by default, no third-party dependency, recommended), etcd-v2, and etcd (legacy, in maintenance mode, slated for removal). The etcd variants require the v2 API enabled; the official test baseline is etcd v3.4.3. Pick a high-bandwidth node for the endpoint; when no port is given it defaults to 29400.

5. The full sequence of a restart ​

How three failures converge, the sequence of an elastic event, and four rendezvous invariants

Chaining the mechanisms above, one elastic event looks like this:

  1. A worker exits, or a node's agent disappears.
  2. Agents on surviving nodes notice (polling every monitor-interval, 0.1 s by default).
  3. All surviving workers are killed immediately — no graceful exit, no waiting for the current step to finish.
  4. Surviving nodes (plus newly joined nodes under elasticity) re-enter rendezvous; once min is reached a new membership forms and the rest register as waiting.
  5. The new WorkerGroup starts with a new RANK and WORLD_SIZE, and your script re-executes from main.
  6. The first thing main must do is load a checkpoint; otherwise it is equivalent to restarting from step 0.

Step 5 is the most overlooked: a restart is not "resume the stack at the breakpoint", it is "start processes again, import again, build the model again". Recovery logic therefore belongs outside the training loop, and must handle "no checkpoint yet" gracefully.

Steps 3 and 6 together give a quantitative relation: checkpoint interval = the maximum work you accept losing per failure. Asynchronous saves keep the write off the critical path, but the upload in flight at kill time may be incomplete — deciding whether that checkpoint is usable means waiting for the upload future and having .metadata on disk (see Distributed Checkpointing in Practice).

6. Four contracts on the script side ​

First, decorate the entry point with @record. The documented behavior: uncaught exceptions in workers are written to TORCHELASTIC_ERROR_FILE, the agent aggregates error files across children and reports the one with the smallest timestamp (i.e. the first error) as the root cause. Without the decorator, the summary contains only an exit code and no traceback — multiplying debugging cost.

python
import os
import torch
import torch.distributed as dist
from torch.distributed.elastic.multiprocessing.errors import record

@record
def main():
    local_rank = int(os.environ["LOCAL_RANK"])
    torch.cuda.set_device(local_rank)          # must precede init_process_group
    dist.init_process_group(backend="nccl")    # RANK/WORLD_SIZE/MASTER_* injected by torchrun
    train()

if __name__ == "__main__":
    main()

Second, do not parse ranks yourself. torchrun injects RANK/WORLD_SIZE/MASTER_ADDR/MASTER_PORT; init_process_group reads them from the environment by default.

Third, mind the --local-rank spelling change. Since PyTorch 2.0, torchrun passes --local-rank=<rank> (hyphen); the old --local_rank (underscore) is only kept for compatibility. A parser that registers only the underscore version fails with error: unrecognized arguments: --local-rank=<rank>. Register both:

python
parser.add_argument("--local-rank", "--local_rank", type=int)

Fourth, fix the construction order. Every restart builds objects from scratch: set_device → init_process_group → model → wrap in DDP/FSDP → optimizer → load state → dataloader (call set_epoch before iterating). A wrong order produces silent drift such as "weights loaded but optimizer momentum is fresh" or "sampler shuffle does not match the checkpoint".

Also, side effects in a rank==0 branch must be idempotent: deleting old checkpoints, clearing shared temp directories, writing a "training started" row to an external system — all of these run again after a restart.

7. What actually has to be recovered ​

"Saving model weights" is far from enough. A resumable checkpoint carries at least six classes of state; miss one and you get a matching symptom:

StateSymptom if missingNote
Model weightsStarts overSave sharded via DCP, which supports cross-topology resharding
Optimizer state (incl. step)Adam momentum resets, loss steps upOptimizer step also locates the scheduler
LR schedulerLR jumps back to initialCan be omitted if derived from step
Data position (epoch + intra-epoch offset)Samples replayed or skippedDistributedSampler.set_epoch only fixes epoch-level shuffle
RNG states (torch/cuda/python/numpy)Dropout and augmentation irreproducibleDetermines whether a rerun can be aligned
Gradient-accumulation boundaryAccumulation window misaligned, effective batch changesAn unfinished window must be discarded on restart

Data position deserves its own paragraph: DistributedSampler.set_epoch(epoch) only guarantees that shuffle order for the N-th epoch is identical; it does not offer "batch 137 in the middle of an epoch". If checkpoints are taken per step, you must record samples consumed and skip that many after rebuilding the dataloader. Not skipping replays data (loss curve shows a repeated segment); skipping incorrectly drops samples (loss degrades suddenly).

A sufficient state dict looks like this (field names are yours to choose; the point is that all of them are present):

python
ckpt = {
    "model": model.state_dict(),          # or DCP sharded save
    "optim": optim.state_dict(),
    "sched": sched.state_dict(),
    "step": global_step,
    "epoch": epoch,
    "samples_consumed": samples_seen,     # for intra-epoch resume
    "rng": {
        "torch": torch.get_rng_state(),
        "cuda": torch.cuda.get_rng_state_all(),
    },
}

8. max_restarts counts restarts, not failures ​

This directly affects capacity planning. --max-restarts=3 means "the worker group may be restarted at most 3 times", not "at most 3 GPUs may fail". One restart may be triggered by one worker failing or by eight failing at once; the counter still increments by one. Observe it via TORCHELASTIC_RESTART_COUNT and TORCHELASTIC_MAX_RESTARTS.

The docs classify errors into four categories, and retry policy should follow them:

CategoryTypical sourceRetry?
User ErrorInvalid arguments (e.g. min>max), bad configNo — retrying just fails again
Worker FailureExceptions in the training process, OOM, NCCL errorsYes, but bounded
Platform ErrorFailures of the agent itselfHand to the job manager
Infra ErrorHost, network, storage failuresHand to the job manager / alerting

Setting max_restarts very high turns a deterministic bug into a restart storm: each restart rebuilds process groups, re-rendezvouses, re-reads checkpoints, burns tens of GPU-minutes, and fails anyway. Bounded retries + error classification + persisting the failing sample is closer to production-grade fault tolerance than "infinite elasticity".

9. Parameter templates for three common shapes ​

Single-node multi-GPU debugging — --standalone picks a random port and a generated id:

bash
torchrun --standalone --nnodes=1 --nproc-per-node=gpu train.py

Fixed-size fault tolerance (no elasticity, tolerates 3 restarts) — explicit c10d and id:

bash
torchrun \
  --nnodes=$NUM_NODES --nproc-per-node=8 \
  --max-restarts=3 \
  --rdzv-backend=c10d --rdzv-id=$JOB_ID \
  --rdzv-endpoint=node1.example.com:29400 \
  train.py

Elastic scaling (1 to 4 nodes, tolerates 3 membership changes or failures):

bash
torchrun \
  --nnodes=1:4 --nproc-per-node=8 \
  --max-restarts=3 \
  --rdzv-backend=c10d --rdzv-id=$JOB_ID \
  --rdzv-endpoint=node1.example.com:29400 \
  train.py

In multi-node settings every node runs the same command (static mode is the exception and needs a distinct --node-rank). Let the job manager launch torchrun as the entry point; do not start one torchrun per node competing for the same rdzv-id — that is not distributed training, it is two sets of processes waiting on the same barrier.

When running several independent jobs on one host, use different ports (or --rdzv-endpoint=localhost:0 to let the OS pick a free one). Otherwise the two jobs get merged into a single worker group, with symptoms of doubled world size and hanging collectives.

10. Troubleshooting by symptom ​

  1. Stuck in rendezvous: verify --rdzv-endpoint is reachable from every node, --rdzv-id is identical everywhere, and --nnodes' min does not exceed the actual node count. A RendezvousTimeoutError is documented as non-retryable — release the resources and resubmit rather than waiting in place.
  2. Address already in use, or two jobs mysteriously merged: same-host jobs sharing a port, or reusing one rdzv-id. Change both the port and the id.
  3. Restart hangs in init_process_group: commonly stale processes holding MASTER_PORT, or a network policy blocking the new connections. Check local leftovers and port usage first, then follow whatever RendezvousConnectionError points at.
  4. Loss jumps after a restart: first check whether only weights were restored (missing optimizer/scheduler) — that gives a step-shaped jump; then check whether data position was replayed — that gives a repeated curve segment.
  5. Restart storm: TORCHELASTIC_RESTART_COUNT climbing while every failure lands near the same step is almost certainly a deterministic bug (corrupt data, wrong shape, checkpoint version mismatch). Stop the job instead of raising the restart ceiling.
  6. unrecognized arguments: --local-rank: the parser only registers --local_rank; register both names as in Sections 3 and 6.

11. Reproducible acceptance checks ​

None of this needs a large cluster. Three checks:

  1. Verify the default: omit --max-restarts, kill one worker manually, and observe the whole group exit with no TORCHELASTIC_RESTART_COUNT (the processes never restarted).
  2. Verify restart semantics: add --max-restarts=3, kill one worker again, and observe all workers killed together, the script re-executing from main, the restart count incrementing, and training resuming from the checkpointed step.
  3. Verify isolation: start two --standalone jobs (random ports) and two c10d jobs on the same port, then compare world size and collective behavior to see what a "merged job" looks like.

By now the boundary should be clear: torchrun owns membership detection, whole-group restart, and re-rendezvous; your script owns recovering state from a checkpoint. The former is a framework capability, the latter an engineering responsibility, and the seam between them — checkpoint integrity and data position — is what decides whether a failure costs minutes or days.

Further reading: Distributed Checkpointing in Practice: DCP Shard Layout, Async Save and Resharding, FSDP2 in Practice, Data Pipeline Tuning.

AI-HPC Expert · Contact: openaihpc@gmail.com