Skip to content

FSDP2 in Practice: the fully_shard Contract, the Memory Bill, and Combining with torch.compile ​

Summary: FSDP2 is not "FSDP1 under a new name". It replaces the flat parameter with per-parameter dim-0 DTensor sharding, and that single change cascades: state dicts no longer need all-gathers, constraints around frozen parameters relax, and memory management no longer needs record_stream or limit_all_gathers. Meanwhile fully_shard comes with six usage rules, and breaking any of them does not raise an error — it just quietly costs throughput. This article covers the contract, the memory bill, and how to combine with torch.compile, with acceptance checks you can actually run.

1. Why FSDP1 habits fail on FSDP2 ​

The official docs describe FSDP2 as an implementation that targets performant eager-mode execution while using per-parameter sharding for improved usability. The differences from FSDP1 cluster in three places:

  1. Sharding representation: FSDP2 chunks each parameter on dim-0 across data-parallel workers (using torch.chunk(dim=0)), whereas FSDP1 flattens, concatenates and chunks a group of tensors together (the flat parameter). The consequence is that "what data sits on which worker" becomes reason-able, and resharding to other parallelisms gets simpler;
  2. State dict: FSDP2 does not support full state dicts directly; the sharded state dict is already made of DTensors and is communication-free. FSDP1 needed all-gathers to produce a full one. If you want a full tensor, call DTensor.full_tensor() yourself, or use the distributed checkpoint state dict APIs;
  3. Memory management: FSDP2 uses a different approach for multi-stream usages that avoids torch.Tensor.record_stream, and does not require blocking the CPU with limit_all_gathers=True as FSDP1 did.

Two more practical consequences: constraints around frozen parameters relax (you no longer fight over how frozen parameters fit into a flat buffer), and fully_shard exposes APIs for manual control of prefetching and collective scheduling for teams that want to tune.

So the first habit to drop when migrating is "use limit_all_gathers to smooth memory spikes" — on FSDP2 it buys nothing, and memory has to be computed from reshard_after_forward and prefetch depth instead.

2. The six rules of fully_shard ​

Signature, verbatim from the docs:

python
torch.distributed.fsdp.fully_shard(
    *module*, *,
    mesh=None,
    reshard_after_forward=None,
    shard_placement_fn=None,
    mp_policy=MixedPrecisionPolicy(param_dtype=None, reduce_dtype=None,
                                   output_dtype=None, cast_forward_inputs=True),
    offload_policy=OffloadPolicy(),
    ignored_params=None,
)

Six rules, each of the "no error, just lost performance" kind:

  1. Parameters become DTensors in place. At initialization fully_shard converts model.parameters() from plain torch.Tensor to DTensor in place and moves them to the appropriate device per the device mesh. Therefore the optimizer must be initialized with the DTensor parameters and the step must run on them. Create the optimizer before fully_shard and it will hold the old tensors.
  2. Hooks do the all-gather and the freeing. A pre-forward hook all-gathers the parameters into plain tensors; a post-forward hook frees them (no communication needed); post-backward hooks free parameters and reduce-scatter gradients.
  3. Call model(input), not model.forward(input). Only __call__ triggers the pre-forward hook. If you really must call forward, either call model.unshard() explicitly or use register_fsdp_forward_method(model, "forward") to register that method for hooking.
  4. Apply it bottom-up. fully_shard builds one group from module.parameters() excluding those already assigned by an earlier call on a submodule; applied to the root, it excludes each layer's parameters and groups only the remainder (embeddings, output projection). Calling it only on the root gives up per-layer overlap.
  5. The type is unioned in place with FSDPModule. An nn.Linear becomes FSDPLinear, an instance of both nn.Linear and FSDPModule, so you gain reshard() and unshard() — note both are not recursive.
  6. FQNs do not change. fully_shard does not wrap the module; it only registers hooks, so model.state_dict() keys are identical before and after. That is the precondition for a seamless checkpoint handoff.

3. Grouping granularity decides whether communication overlaps ​

One group equals one all-gather and one reduce-scatter. The docs say it plainly: partitioning the model into multiple groups ("layer by layer") is what allows peak memory savings and communication/computation overlap, and users generally should not call fully_shard() only on the topmost root module.

The engineering meaning: the number of layers is the number of overlap opportunities and the granularity of prefetch scheduling. Too coarse (say four layers per group) makes each all-gather larger and harder to hide behind compute; too fine (several groups per layer) multiplies collectives and shrinks each message until launch overhead eats the gain. One group per Transformer block is the starting point, then tune from measurements.

4. The three settings of reshard_after_forward ​

ValueAfter forwardBackwardTypical use
Truefrees unsharded parametersre-all-gathersdefault for non-root modules
Falsekeeps unsharded parametersskips that all-gatherdefault for the root module (backward needs them immediately)
None (default)True for non-root, False for rootas aboveleave it alone
intreshards to a smaller world sizeall-gathers over that smaller worldtrade memory for communication

Details that bite:

  • An int must be a non-trivial divisor of the mesh shard dim size (excluding 1 and the dim size itself); the intra-node size (e.g. GPUs per node) is a common choice. It narrows the backward all-gather to intra-node traffic at the cost of higher memory than True;
  • To modify parameters between forward and backward, the registered parameters must be the sharded ones. With False or an int, call reshard() manually;
  • You can flip it at runtime: set_reshard_after_forward(False) for evaluation and back to True for training, which beats rebuilding the model.

5. Mixed precision: three dtypes, three jobs ​

MixedPrecisionPolicy(param_dtype, reduce_dtype, output_dtype, cast_forward_inputs) applies mixed precision at the module level, not the op level, and the documented division of labour is:

  • param_dtype: the dtype of the unsharded parameter, hence of forward/backward compute and of the parameter all-gather. None keeps the original dtype; the optimizer step always uses the sharded parameter in the original dtype;
  • reduce_dtype: the dtype of gradient reduction (reduce-scatter or all-reduce). None with a non-None param_dtype means reduction follows the compute dtype. This is the door to "low-precision compute, full-precision reduction": param_dtype=bf16, reduce_dtype=fp32;
  • output_dtype: the dtype to cast floating-point forward outputs to, useful when different modules use different policies;
  • cast_forward_inputs: whether to cast floating-point forward inputs to param_dtype.

The docs also give the memory argument: FSDP keeps the high-precision sharded parameters in memory anyway, so module-level mixed precision needs no extra high-precision copy of the parameters for the optimizer — a different path from "autocast plus an fp32 master weight".

One detail not to miss: reduce_dtype interacts with accumulation when gradient sync is off — the docs state that if gradient reduction is also disabled via set_requires_gradient_sync(), FSDP accumulates gradients using reduce_dtype. If you do gradient accumulation and low-precision reduction together, think through the numerics.

6. Two knobs for accumulation and backward ​

FSDP1's no_sync() maps to set_requires_gradient_sync(False) in FSDP2; the docs call it "the equivalence of no_sync in FSDP1". It suppresses reduction during microbatches and syncs on the last one — the standard way to implement gradient accumulation without communication.

The two companions:

  • set_reshard_after_backward(False): do not reshard after backward, trading memory for avoiding a re-all-gather before the next forward. Very useful inside an accumulation window, but the longer the window, the tighter the memory;
  • set_is_last_backward(True): tells FSDP the next backward is the last one; it waits on pending gradient reduction and clears the internal structures used for backward prefetch. Mandatory with microbatching, or prefetch state leaks across microbatches.

7. Communication tuning: prefetch, shared streams, zero-copy ​

  • Prefetch depth: set_modules_to_forward_prefetch(list) / set_modules_to_backward_prefetch(list). A singleton list containing the adjacent module reproduces the default overlap; a list of length ≥2 is required for more aggressive overlap and reserves more memory. This is the first knob to try when squeezing out communication stalls.
  • Shared communication streams: share_comm_ctx([...]) shares CUDA streams across multiple FSDPModules. The documented use case is pipeline parallelism, where each model chunk is an FSDP root; sharing streams avoids inter-stream memory fragmentation.
  • Zero-copy collectives: set_allocate_memory_from_process_group_for_comm(True) allocates communication staging buffers from the ProcessGroup's own allocator, letting NCCL exploit zero-copy transfers over SHARP (NVLink and/or InfiniBand). Two hard constraints: it cannot be combined with set_custom_all_gather() or set_custom_reduce_scatter() (those allow finer-grained control that this method cannot account for), and it requires sum-type reductions only, so pair it with set_force_sum_reduction_for_comms(True) — NCCL currently supports zero-copy only for that kind of collective.
  • set_unshard_in_backward(False): for groups whose parameters are provably not needed in backward (embeddings, for example), skipping a backward all-gather.

8. The real cost of CPU offload ​

CPUOffloadPolicy(pin_memory=True) offloads parameters, gradients and optimizer states to CPU: sharded parameters are copied host-to-device before all-gather, unsharded parameters are freed per reshard_after_forward, sharded gradients are copied device-to-host in backward, and the optimizer step runs on CPU with CPU optimizer states.

State the cost honestly: pinned memory makes copies more efficient and lets them overlap with compute, but pinned memory cannot be used by other processes; if CPU memory is short, set pin_memory=False and accept slower, less overlapped copies. There is also the structural cost: the optimizer step runs on CPU and gets noticeably longer, so the end-to-end trade must be computed as "memory saved versus step time added", not as peak memory alone.

9. Combining with torch.compile ​

The official signature and defaults of torch.compile: fullgraph=False, dynamic=None, backend='inductor', mode defaulting to "default". Points that directly affect an FSDP2 combination:

  • fullgraph=False (default): Dynamo looks for compilable regions inside the function, i.e. graph breaks are allowed; True requires the whole function in one graph and raises on any break. FSDP2's hooks, collectives and DTensor redistributions are natural break sources, so always start with the default False, inspect what got compiled, and only then consider tightening.
  • Recompile ceiling: a single frame can be associated with up to torch._dynamo.config.recompile_limit compiled results, which defaults to 8; beyond that it falls back to eager. Guard failures cause repeat compiles — debug with TORCH_LOGS=guards. FSDP2's parameters alternating between DTensor and plain tensor is a common source of guard mismatch, so when "it compiled but did not get faster", check recompile counts before blaming kernels.
  • dynamic=None (default): specialize first, then compile a more dynamic kernel on recompile once dynamism is detected. Variable-length batches will keep recompiling; use TORCH_LOGS=dynamic to catch overspecialization. For stability set dynamic=True explicitly, noting the docs' caveat that some ops and optimizations still force specialization.
  • mode="reduce-overhead": uses CUDA graphs to cut Python overhead, but caches the workspace memory required for the invocation, so memory use is higher, and it only applies to CUDA graphs that do not mutate inputs. FSDP2 is already memory-hungry, so budget separately for this mode; max-autotune also enables CUDA graphs by default on GPU, with the same accounting.
  • torch.nn.Module.compile(): compiles a module in place without changing its structure — a meaningful entry point for a module that has already been through fully_shard.

A step order you can roll back at any point:

  1. Establish an uncompiled baseline: fully_shard per layer, record step time and peak memory;
  2. Compile per block, not the whole model: compiling a Transformer block keeps the region small, compile time short and the graph easier to keep pure; whole-model compilation raises both compile time and recompile risk;
  3. Start from defaults: fullgraph=False, mode="default", then read TORCH_LOGS=guards and TORCH_LOGS=dynamic to confirm recompile counts converge;
  4. Then try reduce-overhead: only if the previous step left memory headroom, and record the peak-memory delta separately;
  5. Accept on three numbers: step time (median, not mean), peak memory, recompiles per unit time. If any one regresses, the compile placement or parameters were wrong.

An honest caveat: the interaction between torch.compile and FSDP2 changes quickly across releases (hook capture scope, DTensor decompositions and CUDA graph support all move). Conclusions must come from your own version's TORCH_LOGS output and measurements — never copy a parameter combination from a blog post.

10. A one-page checklist and five pitfalls ​

Pre-flight checklist

  • [ ] The optimizer is built after fully_shard, and its parameters really are DTensors
  • [ ] Forward calls go through model(x), never model.forward(x)
  • [ ] fully_shard is applied bottom-up per layer, not only on the root
  • [ ] Gradient accumulation uses set_requires_gradient_sync(False) and restores True on the last microbatch
  • [ ] Microbatching calls set_is_last_backward(True)
  • [ ] Any int for reshard_after_forward is a non-trivial divisor of the mesh shard dim
  • [ ] Mixed precision uses MixedPrecisionPolicy without stacking autocast on top and double-casting
  • [ ] Checkpoints use the sharded state dict (DTensor), calling full_tensor() only when a full copy is required
  • [ ] With compilation enabled, recompile counts and peak memory are recorded

Five pitfalls

  • Calling fully_shard only on the root: it runs, but you lose per-layer overlap and peak memory never drops.
  • Building the optimizer before fully_shard: it holds replaced tensors, updates silently do nothing.
  • Using model.forward(x) for convenience: hooks never fire, parameters are never all-gathered, and the forward pass computes garbage.
  • Applying FSDP1's limit_all_gathers lesson to FSDP2: the memory path differs and that knob has no equivalent payoff.
  • Watching throughput but not recompiles after enabling compile: at recompile_limit (8) it falls back to eager, so it looks "compiled" while most steps run eager.

To close: correctness in FSDP2 comes from the contract (what to build first, how to call, in what order); performance comes from three measurable knobs (grouping granularity, reshard setting, prefetch depth). torch.compile multiplies a correct contract — it cannot rescue a broken one.

AI-HPC Expert · Contact: openaihpc@gmail.com