Skip to content

JAX Sharding in Practice: Mesh, PartitionSpec and the Three Sharding Modes ​

On the same 8-GPU node with the same parameter count, one implementation stores a full replica on every device and another stores one eighth per device. In JAX this is not controlled by "installing a parallelism policy object": it is controlled by the sharding property of arrays plus compiler propagation. You write a global-view computation; the compiler turns it into per-device programs. That framing explains three very different failure modes — silent replication, trace-time errors, and results that simply disagree.

This article stays on engineering semantics and makes no framework value judgments: the official three-mode comparison, the four forms of PartitionSpec, the committed conflict caused by device_put, the parameter defaults of jax.distributed.initialize, and finally a structural comparison with PyTorch plus a self-check list.

1. Separate the three sharding modes first ​

JAX's documentation splits this capability into three styles, and the difference compresses into one table (from the explicit-sharding tutorial's mode summary):

ModeThe view you writeSharding explicit?Collectives explicit?
Auto (automatic sharding)Global viewNoNo
Explicit (sharding in types)Global viewYesNo
Manual (shard_map)Per-device viewYesYes

What matters on the job is that the source of truth for sharding differs, and so does the failure signature:

  • Auto: the compiler decides everything. A mistake raises no exception; it shows up as "some axis got replicated" or "an extra all-gather appeared" — a bandwidth and memory cost, not an error.
  • Explicit: sharding lives in the type, queryable at trace time. Ops whose result cannot be inferred fail immediately at trace time and demand an out_sharding.
  • Manual: you write sharding and collectives from a single device's viewpoint. check_rep catches out_specs promises that do not hold.

Three sharding modes, three sources of truth, and the silent cost of P('x') on a (4,2) mesh

2. Mesh: arranging devices into a named grid ​

python
import jax
from jax.sharding import PartitionSpec as P, NamedSharding

mesh = jax.make_mesh((4, 2), ('x', 'y'))
sharding = NamedSharding(mesh, P('x', 'y'))
arr = jax.device_put(data, sharding)

Three points:

  1. make_mesh chooses a better device order taking hardware topology into account (the docs state it "takes hardware topology into account for the Device order"). Mesh(devices, axis_names) also works, but the documentation recommends make_mesh.
  2. The axis names ('x', 'y') become the vocabulary for every later PartitionSpec. A misspelled or omitted axis name raises no Python error; it silently becomes replication.
  3. Sharding mode can be set per axis: jax.make_mesh(shape, names, axis_types=(AxisType.Auto, AxisType.Auto)). The explicit-sharding tutorial passes this argument explicitly, which tells you the parameter decides the mode for that axis — state it when it matters instead of relying on the default.

There is also a scope form that saves passing the mesh around:

python
with jax.set_mesh(mesh):
    ...  # P(...) inside the scope resolves against this mesh

3. PartitionSpec: four forms and one silent cost ​

The tutorial lays the forms out clearly; here they are in operational language:

FormMeaningTypical misuse
P('a', 'b')Axis 0 split over a, axis 1 over brank of the array does not match
P('a', None)Axis 0 split over a, axis 1 not splitassuming None means "excluded"
P()Fully replicatedleaving it out, causing full replication
P(('a', 'b'), None)One logical axis split across two mesh axesassuming a 2-D mesh allows only one axis per dimension

Two documented rules deserve separate mention:

  • None is a placeholder, not a wildcard. P('a', None) means the second dimension is not split; and because mesh axis 'b' is never mentioned, the value is replicated along 'b'.
  • Trailing Nones can be omitted: P('a', None) is the same as P('a').

The silent cost lives here. On a (4, 2) mesh, P('x') replicates data twice along y. With the parameter count unchanged, memory footprint and collective traffic can both double, and no exception is raised. It is the first place to look when "my JAX training uses twice the memory I budgeted".

4. Committed versus uncommitted: the root cause of a ValueError ​

JAX tracks two array states, and this rule explains most crashes that appear after "adding one device_put line":

  • Arrays explicitly placed or sharded with jax.device_put are committed; they will not be moved to other devices automatically.
  • Arrays produced by jnp.zeros, jnp.arange, jnp.array are uncommitted; they can be resharded automatically to accommodate other arguments.

So this snippet raises ValueError outright:

python
sharding1 = NamedSharding(Mesh(jax.devices()[:4], 'x'), P('x'))
sharding2 = NamedSharding(Mesh(jax.devices()[4:], 'x'), P('x'))
y = jax.device_put(x, sharding1)
z = jax.device_put(x, sharding2)
y + z  # ValueError: the two committed shardings are incompatible

Diagnostic rule: on an incompatible-sharding error, first look for "two explicit device_put results used directly in one operation". If one operand was produced by jnp on the fly, it is uncommitted, JAX moves it itself, and no error appears.

5. Auto mode: computation follows sharding ​

The core sentence for Auto mode is computation follows the data sharding. Inside jit, the compiler does not copy sharded arrays back to a single device; it derives shardings for intermediates and outputs from the inputs' sharding, inserts communication where necessary, and parallelizes evaluation accordingly.

Two consequences:

  1. You only worry about the math and the compiler derives sharding — at the cost of unpredictability. Whole-program optimization (GSPMD) means one local edit can change sharding decisions for the entire graph.
  2. The hint mechanism is jax.lax.with_sharding_constraint, and it is a constraint or hint, not a command:
python
@jax.jit
def f(x):
    x = x + 1
    y = jax.lax.with_sharding_constraint(x, NamedSharding(mesh, P('y', 'x')))
    return y

The documentation explicitly recommends annotating at least the outputs of computations, based on how the value is consumed downstream — a wrong output sharding makes every later step pay for it.

The real Auto-mode workflow is "add annotation → inspect what the compiler chose → add another annotation", a slow loop. Explicit mode exists to end that loop.

6. Explicit mode: sharding enters the type, assertable at trace time ​

The only difference in Explicit mode is this: each array's sharding is part of its JAX-level type, queryable during tracing.

python
print(jax.typeof(x))     # float32[4@X,4@Y]: the type records a 4-way split along X and along Y
mesh_types = jax.sharding.get_abstract_mesh()  # current mesh and per-axis modes

The per-op rules are defined one by one (summarized from the tutorial's rule list):

Op categoryOutput sharding rule
nullary (jnp.zeros, jnp.arange)unsharded by default unless overridden by out_sharding
unary elementwise (sin, exp)same sharding as the input
binary (+, -, *)shardings on "zipped" dimensions must match or be None; "outer product" dimensions appearing in only one argument follow that input; an error if the result mentions the same mesh axis twice
reshapesingleton axes are stripped, then the op is classified as split or merge; if a sharded axis is involved, an error demands out_sharding

The value of these rules: instead of guessing the compiler, anything that cannot be inferred fails immediately at trace time rather than surfacing later as a bandwidth surprise.

Modes can be mixed, and the granularity is the mesh axis, not the mesh: axis_types sets it per axis; auto_axes returns selected axes to Auto while explicit_axes pushes selected axes into Explicit. Two hard constraints follow:

  • Sharding annotations (the @X marks in the type) may only mention explicit mesh axes;
  • Collective operations may only mention manual mesh axes.

7. Manual mode: shard_map from a single device's viewpoint ​

shard_map maps a function over shards, and each instance sees only its own shard:

python
from jax.experimental.shard_map import shard_map

f_sharded = shard_map(f, mesh=mesh,
                      in_specs=(P('x'), P('x', None)),
                      out_specs=P('x', None),
                      check_rep=True)

Parameter semantics (from the API reference):

  • in_specs: a pytree of PartitionSpec leaves whose structure is a tree prefix of the args tuple. A mesh axis name at a position means that array axis is split along that mesh axis; an unmentioned axis means replication; a spec of None for an argument means that argument is not sharded at all.
  • out_specs: a mesh axis name at a position means shards are concatenated along that axis; an unmentioned axis is a promise that the shards are equal along it, so a single value is produced rather than a concatenation.
  • check_rep=True (default): enables extra validity checks and autodiff optimizations. The checks are exactly "are shards really identical along mesh axes that out_specs does not mention".
  • auto=frozenset({}): an experimental argument listing axes that are left to the compiler instead of manual sharding; those axes may not appear in in_specs, out_specs, or collectives inside the function.

Two prerequisites worth knowing:

  1. shard_map is documented as an experimental API subject to change; using it in a production training loop warrants regression tests.
  2. If the function uses a Pallas kernel, check_rep must be set to False.

The easiest mistake is that half-sentence promise in out_specs: people read "not written" as "don't care", while it actually means "must guarantee the shards are identical". check_rep catches violations at runtime — the biggest return of Manual mode over hand-written collectives.

8. Multi-process: parameters and defaults of jax.distributed.initialize ​

At multi-host scale, JAX's model is multi-controller: each Python process (controller) owns its local devices, the union across processes forms the global devices, and the local sets are disjoint.

python
jax.distributed.initialize(coordinator_address='192.168.0.1:8000',
                           num_processes=4, process_id=0)
print(jax.process_index(), jax.devices(), jax.local_devices())

Hard constraints and defaults (from the API page):

ParameterSemantics
coordinator_addressaddress and port of process 0; localhost / 127.0.0.1 are not usable in a multi-host run
num_processes / process_idprocess_id must be a dense range 0..num_processes-1
local_device_idsrestricts visible devices; defaults to all local devices visible, but Slurm / Open MPI on GPUs default to one device per process
initialization_timeoutdefault 300 s; initialization errors if exceeded
heartbeat_timeout_secondsdefault 100 s; a process that misses heartbeats is considered dead
shutdown_timeout_secondsdefault 300 s; how long a terminating process waits for the others
coordinator_bind_addressrequired on multi-NIC nodes; otherwise the coordination service may listen on only one interface and other nodes cannot connect

Four field-level traps:

  1. Ordering: initialize() must precede any device access (jax.devices(), jax.local_devices(), and jnp computation all count). Calling it after accessing devices raises an error.
  2. Proxy variables: in environments that reach the network only through HTTP_PROXY / HTTPS_PROXY, initialize may time out; unset those variables before launch.
  3. Auto-detection: on TPU, Slurm and Open MPI all arguments are optional; other MPI installs can use cluster_detection_method="mpi4py" (Open MPI 5 and newer require a working mpi4py); when all four key arguments are supplied, auto-detection is skipped.
  4. Failure semantics: the coordination service performs health checking — if any process dies, all processes shut down; it is also used for distributed checkpointing. This mirrors torchrun elastic, where a failure kills all surviving workers; the difference is that on the JAX side restart is initiated by an external scheduler, not by the framework itself.

9. Structural comparison with PyTorch sharding ​

Structure only: no performance verdicts and no accelerator-ecosystem specifics.

DimensionJAXPyTorch
Device gridjax.make_mesh with named axestorch.distributed.device_mesh.init_device_mesh
Sharding descriptionPartitionSpec / NamedSharding, visible in jax.typeofDTensor placements
Who decides shardingcompiler in Auto, types plus user in Explicit, user in Manualuser via fully_shard, recorded by DTensor
Explicit collectiveswritten inside shard_map (psum etc.)dist.all_reduce by hand, or delegated to the comm library
Checkpointingcoordination service plus ecosystem checkpointingtorch.distributed.checkpoint

One sentence for the essential difference: in JAX sharding is part of the compilation unit — queryable and assertable; in PyTorch it is a runtime property attached to tensors. That changes the debugging feel: JAX catches mistakes at trace time more easily, PyTorch makes intermediate state easier to observe at runtime with hooks. For the PyTorch-side module wrapping and checkpoint idioms, see FSDP2 in practice and Distributed checkpoint in practice.

10. Memory arithmetic and six frequent traps ​

A recomputable figure first: 70B parameters in bf16 (2 bytes) is 70 × 10⁹ × 2 = 140 GB (decimal) of pure weights. Split across 8 devices that is 17.5 GB each — provided the sharding really splits along 8 devices. If a PartitionSpec omits an axis and triggers replication, that figure doubles immediately, on top of activations.

Six traps, as symptom → root cause:

SymptomRoot cause
Adding one device_put line raises ValueErrortwo committed shardings are incompatible (section 4)
Mesh construction rejects the sizeproduct of axis sizes does not match available devices
check_rep reports inconsistent sharding at runtimeshards along axes unmentioned in out_specs are not actually equal (section 7)
jax.devices() only shows local devicesinitialize ordering: a jnp op or devices() call happened first
Multi-host initialization keeps failinglocalhost was used, or proxy variables were not unset
Training runs on only some nodesprocess_id is not dense, or local_device_ids does not match the mesh size

11. Go-live checklist ​

Print these before writing code, and compare them when something breaks:

python
print('global devices =', len(jax.devices()))
print('local  devices =', len(jax.local_devices()))
print('abstract mesh =', jax.sharding.get_abstract_mesh())
  1. Devices match the grid: prod(axis_sizes) == len(jax.devices()). If not, fix the mesh before going further.
  2. Inspect sharding instead of guessing: jax.debug.visualize_array_sharding(x) per layer, watching specifically for parameters replicated along one axis.
  3. Annotate outputs: jit(..., out_shardings=NamedSharding(mesh, P(...))), chosen by how the value is consumed downstream.
  4. Check shard_map's out_specs position by position: each array the function actually returns must line up with its spec under "mentioned = concatenate, unmentioned = promise of equality".
  5. Make process_id dense across hosts: map the scheduler's rank to a dense integer; do not pass a node index through.
  6. Design checkpointing together with sharding: sharding metadata must be saved with the checkpoint, or a differently shaped mesh cannot restore it. JAX-side checkpointing deserves its own article; on the PyTorch side compare the DCP sharded layout.

AI-HPC Expert · Contact: openaihpc@gmail.com