JAX Sharding in Practice: Mesh, PartitionSpec and the Three Sharding Modes
On the same 8-GPU node with the same parameter count, one implementation stores a full replica on every device and another stores one eighth per device. In JAX this is not controlled by "installing a parallelism policy object": it is controlled by the sharding property of arrays plus compiler propagation. You write a global-view computation; the compiler turns it into per-device programs. That framing explains three very different failure modes — silent replication, trace-time errors, and results that simply disagree.
This article stays on engineering semantics and makes no framework value judgments: the official three-mode comparison, the four forms of PartitionSpec, the committed conflict caused by device_put, the parameter defaults of jax.distributed.initialize, and finally a structural comparison with PyTorch plus a self-check list.
1. Separate the three sharding modes first
JAX's documentation splits this capability into three styles, and the difference compresses into one table (from the explicit-sharding tutorial's mode summary):
| Mode | The view you write | Sharding explicit? | Collectives explicit? |
|---|---|---|---|
| Auto (automatic sharding) | Global view | No | No |
| Explicit (sharding in types) | Global view | Yes | No |
Manual (shard_map) | Per-device view | Yes | Yes |
What matters on the job is that the source of truth for sharding differs, and so does the failure signature:
- Auto: the compiler decides everything. A mistake raises no exception; it shows up as "some axis got replicated" or "an extra all-gather appeared" — a bandwidth and memory cost, not an error.
- Explicit: sharding lives in the type, queryable at trace time. Ops whose result cannot be inferred fail immediately at trace time and demand an
out_sharding. - Manual: you write sharding and collectives from a single device's viewpoint.
check_repcatchesout_specspromises that do not hold.
2. Mesh: arranging devices into a named grid
import jax
from jax.sharding import PartitionSpec as P, NamedSharding
mesh = jax.make_mesh((4, 2), ('x', 'y'))
sharding = NamedSharding(mesh, P('x', 'y'))
arr = jax.device_put(data, sharding)Three points:
make_meshchooses a better device order taking hardware topology into account (the docs state it "takes hardware topology into account for theDeviceorder").Mesh(devices, axis_names)also works, but the documentation recommendsmake_mesh.- The axis names (
'x','y') become the vocabulary for every laterPartitionSpec. A misspelled or omitted axis name raises no Python error; it silently becomes replication. - Sharding mode can be set per axis:
jax.make_mesh(shape, names, axis_types=(AxisType.Auto, AxisType.Auto)). The explicit-sharding tutorial passes this argument explicitly, which tells you the parameter decides the mode for that axis — state it when it matters instead of relying on the default.
There is also a scope form that saves passing the mesh around:
with jax.set_mesh(mesh):
... # P(...) inside the scope resolves against this mesh3. PartitionSpec: four forms and one silent cost
The tutorial lays the forms out clearly; here they are in operational language:
| Form | Meaning | Typical misuse |
|---|---|---|
P('a', 'b') | Axis 0 split over a, axis 1 over b | rank of the array does not match |
P('a', None) | Axis 0 split over a, axis 1 not split | assuming None means "excluded" |
P() | Fully replicated | leaving it out, causing full replication |
P(('a', 'b'), None) | One logical axis split across two mesh axes | assuming a 2-D mesh allows only one axis per dimension |
Two documented rules deserve separate mention:
Noneis a placeholder, not a wildcard.P('a', None)means the second dimension is not split; and because mesh axis'b'is never mentioned, the value is replicated along'b'.- Trailing
Nones can be omitted:P('a', None)is the same asP('a').
The silent cost lives here. On a (4, 2) mesh, P('x') replicates data twice along y. With the parameter count unchanged, memory footprint and collective traffic can both double, and no exception is raised. It is the first place to look when "my JAX training uses twice the memory I budgeted".
4. Committed versus uncommitted: the root cause of a ValueError
JAX tracks two array states, and this rule explains most crashes that appear after "adding one device_put line":
- Arrays explicitly placed or sharded with
jax.device_putare committed; they will not be moved to other devices automatically. - Arrays produced by
jnp.zeros,jnp.arange,jnp.arrayare uncommitted; they can be resharded automatically to accommodate other arguments.
So this snippet raises ValueError outright:
sharding1 = NamedSharding(Mesh(jax.devices()[:4], 'x'), P('x'))
sharding2 = NamedSharding(Mesh(jax.devices()[4:], 'x'), P('x'))
y = jax.device_put(x, sharding1)
z = jax.device_put(x, sharding2)
y + z # ValueError: the two committed shardings are incompatibleDiagnostic rule: on an incompatible-sharding error, first look for "two explicit device_put results used directly in one operation". If one operand was produced by jnp on the fly, it is uncommitted, JAX moves it itself, and no error appears.
5. Auto mode: computation follows sharding
The core sentence for Auto mode is computation follows the data sharding. Inside jit, the compiler does not copy sharded arrays back to a single device; it derives shardings for intermediates and outputs from the inputs' sharding, inserts communication where necessary, and parallelizes evaluation accordingly.
Two consequences:
- You only worry about the math and the compiler derives sharding — at the cost of unpredictability. Whole-program optimization (GSPMD) means one local edit can change sharding decisions for the entire graph.
- The hint mechanism is
jax.lax.with_sharding_constraint, and it is a constraint or hint, not a command:
@jax.jit
def f(x):
x = x + 1
y = jax.lax.with_sharding_constraint(x, NamedSharding(mesh, P('y', 'x')))
return yThe documentation explicitly recommends annotating at least the outputs of computations, based on how the value is consumed downstream — a wrong output sharding makes every later step pay for it.
The real Auto-mode workflow is "add annotation → inspect what the compiler chose → add another annotation", a slow loop. Explicit mode exists to end that loop.
6. Explicit mode: sharding enters the type, assertable at trace time
The only difference in Explicit mode is this: each array's sharding is part of its JAX-level type, queryable during tracing.
print(jax.typeof(x)) # float32[4@X,4@Y]: the type records a 4-way split along X and along Y
mesh_types = jax.sharding.get_abstract_mesh() # current mesh and per-axis modesThe per-op rules are defined one by one (summarized from the tutorial's rule list):
| Op category | Output sharding rule |
|---|---|
nullary (jnp.zeros, jnp.arange) | unsharded by default unless overridden by out_sharding |
unary elementwise (sin, exp) | same sharding as the input |
binary (+, -, *) | shardings on "zipped" dimensions must match or be None; "outer product" dimensions appearing in only one argument follow that input; an error if the result mentions the same mesh axis twice |
reshape | singleton axes are stripped, then the op is classified as split or merge; if a sharded axis is involved, an error demands out_sharding |
The value of these rules: instead of guessing the compiler, anything that cannot be inferred fails immediately at trace time rather than surfacing later as a bandwidth surprise.
Modes can be mixed, and the granularity is the mesh axis, not the mesh: axis_types sets it per axis; auto_axes returns selected axes to Auto while explicit_axes pushes selected axes into Explicit. Two hard constraints follow:
- Sharding annotations (the
@Xmarks in the type) may only mention explicit mesh axes; - Collective operations may only mention manual mesh axes.
7. Manual mode: shard_map from a single device's viewpoint
shard_map maps a function over shards, and each instance sees only its own shard:
from jax.experimental.shard_map import shard_map
f_sharded = shard_map(f, mesh=mesh,
in_specs=(P('x'), P('x', None)),
out_specs=P('x', None),
check_rep=True)Parameter semantics (from the API reference):
in_specs: a pytree ofPartitionSpecleaves whose structure is a tree prefix of the args tuple. A mesh axis name at a position means that array axis is split along that mesh axis; an unmentioned axis means replication; a spec ofNonefor an argument means that argument is not sharded at all.out_specs: a mesh axis name at a position means shards are concatenated along that axis; an unmentioned axis is a promise that the shards are equal along it, so a single value is produced rather than a concatenation.check_rep=True(default): enables extra validity checks and autodiff optimizations. The checks are exactly "are shards really identical along mesh axes thatout_specsdoes not mention".auto=frozenset({}): an experimental argument listing axes that are left to the compiler instead of manual sharding; those axes may not appear inin_specs,out_specs, or collectives inside the function.
Two prerequisites worth knowing:
shard_mapis documented as an experimental API subject to change; using it in a production training loop warrants regression tests.- If the function uses a Pallas kernel,
check_repmust be set toFalse.
The easiest mistake is that half-sentence promise in out_specs: people read "not written" as "don't care", while it actually means "must guarantee the shards are identical". check_rep catches violations at runtime — the biggest return of Manual mode over hand-written collectives.
8. Multi-process: parameters and defaults of jax.distributed.initialize
At multi-host scale, JAX's model is multi-controller: each Python process (controller) owns its local devices, the union across processes forms the global devices, and the local sets are disjoint.
jax.distributed.initialize(coordinator_address='192.168.0.1:8000',
num_processes=4, process_id=0)
print(jax.process_index(), jax.devices(), jax.local_devices())Hard constraints and defaults (from the API page):
| Parameter | Semantics |
|---|---|
coordinator_address | address and port of process 0; localhost / 127.0.0.1 are not usable in a multi-host run |
num_processes / process_id | process_id must be a dense range 0..num_processes-1 |
local_device_ids | restricts visible devices; defaults to all local devices visible, but Slurm / Open MPI on GPUs default to one device per process |
initialization_timeout | default 300 s; initialization errors if exceeded |
heartbeat_timeout_seconds | default 100 s; a process that misses heartbeats is considered dead |
shutdown_timeout_seconds | default 300 s; how long a terminating process waits for the others |
coordinator_bind_address | required on multi-NIC nodes; otherwise the coordination service may listen on only one interface and other nodes cannot connect |
Four field-level traps:
- Ordering:
initialize()must precede any device access (jax.devices(),jax.local_devices(), andjnpcomputation all count). Calling it after accessing devices raises an error. - Proxy variables: in environments that reach the network only through
HTTP_PROXY/HTTPS_PROXY,initializemay time out; unset those variables before launch. - Auto-detection: on TPU, Slurm and Open MPI all arguments are optional; other MPI installs can use
cluster_detection_method="mpi4py"(Open MPI 5 and newer require a workingmpi4py); when all four key arguments are supplied, auto-detection is skipped. - Failure semantics: the coordination service performs health checking — if any process dies, all processes shut down; it is also used for distributed checkpointing. This mirrors
torchrun elastic, where a failure kills all surviving workers; the difference is that on the JAX side restart is initiated by an external scheduler, not by the framework itself.
9. Structural comparison with PyTorch sharding
Structure only: no performance verdicts and no accelerator-ecosystem specifics.
| Dimension | JAX | PyTorch |
|---|---|---|
| Device grid | jax.make_mesh with named axes | torch.distributed.device_mesh.init_device_mesh |
| Sharding description | PartitionSpec / NamedSharding, visible in jax.typeof | DTensor placements |
| Who decides sharding | compiler in Auto, types plus user in Explicit, user in Manual | user via fully_shard, recorded by DTensor |
| Explicit collectives | written inside shard_map (psum etc.) | dist.all_reduce by hand, or delegated to the comm library |
| Checkpointing | coordination service plus ecosystem checkpointing | torch.distributed.checkpoint |
One sentence for the essential difference: in JAX sharding is part of the compilation unit — queryable and assertable; in PyTorch it is a runtime property attached to tensors. That changes the debugging feel: JAX catches mistakes at trace time more easily, PyTorch makes intermediate state easier to observe at runtime with hooks. For the PyTorch-side module wrapping and checkpoint idioms, see FSDP2 in practice and Distributed checkpoint in practice.
10. Memory arithmetic and six frequent traps
A recomputable figure first: 70B parameters in bf16 (2 bytes) is 70 × 10⁹ × 2 = 140 GB (decimal) of pure weights. Split across 8 devices that is 17.5 GB each — provided the sharding really splits along 8 devices. If a PartitionSpec omits an axis and triggers replication, that figure doubles immediately, on top of activations.
Six traps, as symptom → root cause:
| Symptom | Root cause |
|---|---|
Adding one device_put line raises ValueError | two committed shardings are incompatible (section 4) |
| Mesh construction rejects the size | product of axis sizes does not match available devices |
check_rep reports inconsistent sharding at runtime | shards along axes unmentioned in out_specs are not actually equal (section 7) |
jax.devices() only shows local devices | initialize ordering: a jnp op or devices() call happened first |
| Multi-host initialization keeps failing | localhost was used, or proxy variables were not unset |
| Training runs on only some nodes | process_id is not dense, or local_device_ids does not match the mesh size |
11. Go-live checklist
Print these before writing code, and compare them when something breaks:
print('global devices =', len(jax.devices()))
print('local devices =', len(jax.local_devices()))
print('abstract mesh =', jax.sharding.get_abstract_mesh())- Devices match the grid:
prod(axis_sizes) == len(jax.devices()). If not, fix the mesh before going further. - Inspect sharding instead of guessing:
jax.debug.visualize_array_sharding(x)per layer, watching specifically for parameters replicated along one axis. - Annotate outputs:
jit(..., out_shardings=NamedSharding(mesh, P(...))), chosen by how the value is consumed downstream. - Check
shard_map'sout_specsposition by position: each array the function actually returns must line up with its spec under "mentioned = concatenate, unmentioned = promise of equality". - Make
process_iddense across hosts: map the scheduler's rank to a dense integer; do not pass a node index through. - Design checkpointing together with sharding: sharding metadata must be saved with the checkpoint, or a differently shaped mesh cannot restore it. JAX-side checkpointing deserves its own article; on the PyTorch side compare the DCP sharded layout.
Related Articles
- Training framework ledger: the parallelism abstractions of Megatron-LM and DeepSpeed
- FSDP2 in practice: the fully_shard contract, memory arithmetic and torch.compile
- Distributed checkpoint in practice: DCP's sharded layout, async saving and re-sharding
Related Articles
- PyTorch Internals: Tensor Storage, Views, and the Dispatch Mechanism
- Data Pipeline Tuning: Choosing Between DataLoader, DALI and WebDataset
- Elastic Training and Fault Tolerance: torchrun Rendezvous, Restart Semantics, and State Recovery
- PyTorch Autograd Deep Dive: From Dynamic Computation Graphs to Backpropagation