FSDP2 in Practice: the fully_shard Contract, the Memory Bill, and Combining with torch.compile
Summary: FSDP2 is not "FSDP1 under a new name". It replaces the flat parameter with per-parameter dim-0 DTensor sharding, and that single change cascades: state dicts no longer need all-gathers, constraints around frozen parameters relax, and memory management no longer needs
record_streamorlimit_all_gathers. Meanwhilefully_shardcomes with six usage rules, and breaking any of them does not raise an error — it just quietly costs throughput. This article covers the contract, the memory bill, and how to combine withtorch.compile, with acceptance checks you can actually run.
1. Why FSDP1 habits fail on FSDP2
The official docs describe FSDP2 as an implementation that targets performant eager-mode execution while using per-parameter sharding for improved usability. The differences from FSDP1 cluster in three places:
- Sharding representation: FSDP2 chunks each parameter on dim-0 across data-parallel workers (using
torch.chunk(dim=0)), whereas FSDP1 flattens, concatenates and chunks a group of tensors together (the flat parameter). The consequence is that "what data sits on which worker" becomes reason-able, and resharding to other parallelisms gets simpler; - State dict: FSDP2 does not support full state dicts directly; the sharded state dict is already made of DTensors and is communication-free. FSDP1 needed all-gathers to produce a full one. If you want a full tensor, call
DTensor.full_tensor()yourself, or use the distributed checkpoint state dict APIs; - Memory management: FSDP2 uses a different approach for multi-stream usages that avoids
torch.Tensor.record_stream, and does not require blocking the CPU withlimit_all_gathers=Trueas FSDP1 did.
Two more practical consequences: constraints around frozen parameters relax (you no longer fight over how frozen parameters fit into a flat buffer), and fully_shard exposes APIs for manual control of prefetching and collective scheduling for teams that want to tune.
So the first habit to drop when migrating is "use limit_all_gathers to smooth memory spikes" — on FSDP2 it buys nothing, and memory has to be computed from reshard_after_forward and prefetch depth instead.
2. The six rules of fully_shard
Signature, verbatim from the docs:
torch.distributed.fsdp.fully_shard(
*module*, *,
mesh=None,
reshard_after_forward=None,
shard_placement_fn=None,
mp_policy=MixedPrecisionPolicy(param_dtype=None, reduce_dtype=None,
output_dtype=None, cast_forward_inputs=True),
offload_policy=OffloadPolicy(),
ignored_params=None,
)Six rules, each of the "no error, just lost performance" kind:
- Parameters become DTensors in place. At initialization
fully_shardconvertsmodel.parameters()from plaintorch.Tensorto DTensor in place and moves them to the appropriate device per the device mesh. Therefore the optimizer must be initialized with the DTensor parameters and the step must run on them. Create the optimizer beforefully_shardand it will hold the old tensors. - Hooks do the all-gather and the freeing. A pre-forward hook all-gathers the parameters into plain tensors; a post-forward hook frees them (no communication needed); post-backward hooks free parameters and reduce-scatter gradients.
- Call
model(input), notmodel.forward(input). Only__call__triggers the pre-forward hook. If you really must callforward, either callmodel.unshard()explicitly or useregister_fsdp_forward_method(model, "forward")to register that method for hooking. - Apply it bottom-up.
fully_shardbuilds one group frommodule.parameters()excluding those already assigned by an earlier call on a submodule; applied to the root, it excludes each layer's parameters and groups only the remainder (embeddings, output projection). Calling it only on the root gives up per-layer overlap. - The type is unioned in place with FSDPModule. An
nn.LinearbecomesFSDPLinear, an instance of bothnn.LinearandFSDPModule, so you gainreshard()andunshard()— note both are not recursive. - FQNs do not change.
fully_sharddoes not wrap the module; it only registers hooks, somodel.state_dict()keys are identical before and after. That is the precondition for a seamless checkpoint handoff.
3. Grouping granularity decides whether communication overlaps
One group equals one all-gather and one reduce-scatter. The docs say it plainly: partitioning the model into multiple groups ("layer by layer") is what allows peak memory savings and communication/computation overlap, and users generally should not call fully_shard() only on the topmost root module.
The engineering meaning: the number of layers is the number of overlap opportunities and the granularity of prefetch scheduling. Too coarse (say four layers per group) makes each all-gather larger and harder to hide behind compute; too fine (several groups per layer) multiplies collectives and shrinks each message until launch overhead eats the gain. One group per Transformer block is the starting point, then tune from measurements.
4. The three settings of reshard_after_forward
| Value | After forward | Backward | Typical use |
|---|---|---|---|
True | frees unsharded parameters | re-all-gathers | default for non-root modules |
False | keeps unsharded parameters | skips that all-gather | default for the root module (backward needs them immediately) |
None (default) | True for non-root, False for root | as above | leave it alone |
int | reshards to a smaller world size | all-gathers over that smaller world | trade memory for communication |
Details that bite:
- An
intmust be a non-trivial divisor of the mesh shard dim size (excluding 1 and the dim size itself); the intra-node size (e.g. GPUs per node) is a common choice. It narrows the backward all-gather to intra-node traffic at the cost of higher memory thanTrue; - To modify parameters between forward and backward, the registered parameters must be the sharded ones. With
Falseor anint, callreshard()manually; - You can flip it at runtime:
set_reshard_after_forward(False)for evaluation and back toTruefor training, which beats rebuilding the model.
5. Mixed precision: three dtypes, three jobs
MixedPrecisionPolicy(param_dtype, reduce_dtype, output_dtype, cast_forward_inputs) applies mixed precision at the module level, not the op level, and the documented division of labour is:
param_dtype: the dtype of the unsharded parameter, hence of forward/backward compute and of the parameter all-gather.Nonekeeps the original dtype; the optimizer step always uses the sharded parameter in the original dtype;reduce_dtype: the dtype of gradient reduction (reduce-scatter or all-reduce).Nonewith a non-Noneparam_dtypemeans reduction follows the compute dtype. This is the door to "low-precision compute, full-precision reduction":param_dtype=bf16, reduce_dtype=fp32;output_dtype: the dtype to cast floating-point forward outputs to, useful when different modules use different policies;cast_forward_inputs: whether to cast floating-point forward inputs toparam_dtype.
The docs also give the memory argument: FSDP keeps the high-precision sharded parameters in memory anyway, so module-level mixed precision needs no extra high-precision copy of the parameters for the optimizer — a different path from "autocast plus an fp32 master weight".
One detail not to miss: reduce_dtype interacts with accumulation when gradient sync is off — the docs state that if gradient reduction is also disabled via set_requires_gradient_sync(), FSDP accumulates gradients using reduce_dtype. If you do gradient accumulation and low-precision reduction together, think through the numerics.
6. Two knobs for accumulation and backward
FSDP1's no_sync() maps to set_requires_gradient_sync(False) in FSDP2; the docs call it "the equivalence of no_sync in FSDP1". It suppresses reduction during microbatches and syncs on the last one — the standard way to implement gradient accumulation without communication.
The two companions:
set_reshard_after_backward(False): do not reshard after backward, trading memory for avoiding a re-all-gather before the next forward. Very useful inside an accumulation window, but the longer the window, the tighter the memory;set_is_last_backward(True): tells FSDP the next backward is the last one; it waits on pending gradient reduction and clears the internal structures used for backward prefetch. Mandatory with microbatching, or prefetch state leaks across microbatches.
7. Communication tuning: prefetch, shared streams, zero-copy
- Prefetch depth:
set_modules_to_forward_prefetch(list)/set_modules_to_backward_prefetch(list). A singleton list containing the adjacent module reproduces the default overlap; a list of length ≥2 is required for more aggressive overlap and reserves more memory. This is the first knob to try when squeezing out communication stalls. - Shared communication streams:
share_comm_ctx([...])shares CUDA streams across multiple FSDPModules. The documented use case is pipeline parallelism, where each model chunk is an FSDP root; sharing streams avoids inter-stream memory fragmentation. - Zero-copy collectives:
set_allocate_memory_from_process_group_for_comm(True)allocates communication staging buffers from the ProcessGroup's own allocator, letting NCCL exploit zero-copy transfers over SHARP (NVLink and/or InfiniBand). Two hard constraints: it cannot be combined withset_custom_all_gather()orset_custom_reduce_scatter()(those allow finer-grained control that this method cannot account for), and it requires sum-type reductions only, so pair it withset_force_sum_reduction_for_comms(True)— NCCL currently supports zero-copy only for that kind of collective. set_unshard_in_backward(False): for groups whose parameters are provably not needed in backward (embeddings, for example), skipping a backward all-gather.
8. The real cost of CPU offload
CPUOffloadPolicy(pin_memory=True) offloads parameters, gradients and optimizer states to CPU: sharded parameters are copied host-to-device before all-gather, unsharded parameters are freed per reshard_after_forward, sharded gradients are copied device-to-host in backward, and the optimizer step runs on CPU with CPU optimizer states.
State the cost honestly: pinned memory makes copies more efficient and lets them overlap with compute, but pinned memory cannot be used by other processes; if CPU memory is short, set pin_memory=False and accept slower, less overlapped copies. There is also the structural cost: the optimizer step runs on CPU and gets noticeably longer, so the end-to-end trade must be computed as "memory saved versus step time added", not as peak memory alone.
9. Combining with torch.compile
The official signature and defaults of torch.compile: fullgraph=False, dynamic=None, backend='inductor', mode defaulting to "default". Points that directly affect an FSDP2 combination:
fullgraph=False(default): Dynamo looks for compilable regions inside the function, i.e. graph breaks are allowed;Truerequires the whole function in one graph and raises on any break. FSDP2's hooks, collectives and DTensor redistributions are natural break sources, so always start with the defaultFalse, inspect what got compiled, and only then consider tightening.- Recompile ceiling: a single frame can be associated with up to
torch._dynamo.config.recompile_limitcompiled results, which defaults to 8; beyond that it falls back to eager. Guard failures cause repeat compiles — debug withTORCH_LOGS=guards. FSDP2's parameters alternating between DTensor and plain tensor is a common source of guard mismatch, so when "it compiled but did not get faster", check recompile counts before blaming kernels. dynamic=None(default): specialize first, then compile a more dynamic kernel on recompile once dynamism is detected. Variable-length batches will keep recompiling; useTORCH_LOGS=dynamicto catch overspecialization. For stability setdynamic=Trueexplicitly, noting the docs' caveat that some ops and optimizations still force specialization.mode="reduce-overhead": uses CUDA graphs to cut Python overhead, but caches the workspace memory required for the invocation, so memory use is higher, and it only applies to CUDA graphs that do not mutate inputs. FSDP2 is already memory-hungry, so budget separately for this mode;max-autotunealso enables CUDA graphs by default on GPU, with the same accounting.torch.nn.Module.compile(): compiles a module in place without changing its structure — a meaningful entry point for a module that has already been throughfully_shard.
A step order you can roll back at any point:
- Establish an uncompiled baseline:
fully_shardper layer, record step time and peak memory; - Compile per block, not the whole model: compiling a Transformer block keeps the region small, compile time short and the graph easier to keep pure; whole-model compilation raises both compile time and recompile risk;
- Start from defaults:
fullgraph=False,mode="default", then readTORCH_LOGS=guardsandTORCH_LOGS=dynamicto confirm recompile counts converge; - Then try
reduce-overhead: only if the previous step left memory headroom, and record the peak-memory delta separately; - Accept on three numbers: step time (median, not mean), peak memory, recompiles per unit time. If any one regresses, the compile placement or parameters were wrong.
An honest caveat: the interaction between torch.compile and FSDP2 changes quickly across releases (hook capture scope, DTensor decompositions and CUDA graph support all move). Conclusions must come from your own version's TORCH_LOGS output and measurements — never copy a parameter combination from a blog post.
10. A one-page checklist and five pitfalls
Pre-flight checklist
- [ ] The optimizer is built after
fully_shard, and its parameters really are DTensors - [ ] Forward calls go through
model(x), nevermodel.forward(x) - [ ]
fully_shardis applied bottom-up per layer, not only on the root - [ ] Gradient accumulation uses
set_requires_gradient_sync(False)and restoresTrueon the last microbatch - [ ] Microbatching calls
set_is_last_backward(True) - [ ] Any
intforreshard_after_forwardis a non-trivial divisor of the mesh shard dim - [ ] Mixed precision uses
MixedPrecisionPolicywithout stacking autocast on top and double-casting - [ ] Checkpoints use the sharded state dict (DTensor), calling
full_tensor()only when a full copy is required - [ ] With compilation enabled, recompile counts and peak memory are recorded
Five pitfalls
- Calling
fully_shardonly on the root: it runs, but you lose per-layer overlap and peak memory never drops. - Building the optimizer before
fully_shard: it holds replaced tensors, updates silently do nothing. - Using
model.forward(x)for convenience: hooks never fire, parameters are never all-gathered, and the forward pass computes garbage. - Applying FSDP1's
limit_all_gatherslesson to FSDP2: the memory path differs and that knob has no equivalent payoff. - Watching throughput but not recompiles after enabling compile: at
recompile_limit(8) it falls back to eager, so it looks "compiled" while most steps run eager.
To close: correctness in FSDP2 comes from the contract (what to build first, how to call, in what order); performance comes from three measurable knobs (grouping granularity, reshard setting, prefetch depth). torch.compile multiplies a correct contract — it cannot rescue a broken one.