Distributed Parallelism Overview: From Single GPU to 10K

Abstract: Training a model that does not fit in a single GPU's memory boils down to three questions: what to partition (weights / gradients / activations / experts), into how many pieces (degree of parallelism), and how to synchronize after partitioning (communication primitives and network topology). Starting from a single GPU, this article derives mixed-parallel configurations step by step and provides a selection decision tree you can use directly.

1. The Starting Point: Why a Single GPU Is Not Enough

Take a 70B-parameter model in BF16: the weights alone need about 140 GB (70 × 10⁹ × 2 bytes). Adding the Adam optimizer states (2 momentum + 1 variance, ~420 GB), gradients (140 GB) and activations, a single H100's 80 GB is nowhere near enough.

Single-GPU memory = weights 140G + gradients 140G + optimizer 420G + activations
                 ≈ 700+ GB (70B BF16 full-parameter training)

This is where distributed training starts: when it doesn't fit in memory, you must partition the state across GPUs.

2. Parallelism Strategies: Six Ways to Partition

2.1 Data Parallelism (DP) — Partition the Data

Each GPU holds a full model replica and consumes a different batch; after forward/backward, gradients are synchronized with a single AllReduce.

• Communication: 2 × model parameter count per step (gradients), independent of data size.
• Best for: models that fit on a single GPU where data volume / throughput is the bottleneck.
• Bottleneck: no memory savings; communication ratio shrinks as batch grows.

2.2 ZeRO (Zero Redundancy Optimizer) — Partition Optimizer States

A memory-saving variant of data parallelism: optimizer states, gradients, even weights are sharded across DP ranks; each rank holds only its shard and fetches via AllGather when needed.

• ZeRO-1 shards optimizer states only; ZeRO-2 adds gradients; ZeRO-3 shards all three (activations aside).
• Cost: communication grows from 2× to roughly 2.7×–3× parameters (extra weight AllGather).
• Best for: small-scale (≤32 GPUs) fine-tuning, long-context, low-cost scenarios. DeepSpeed ZeRO / FSDP are the engineering implementations.

2.3 Tensor Parallelism (TP) — Partition Weight Matrices

Split a linear layer's weight matrix by row/column across GPUs; each forward requires 2 AllReduces (one forward, one backward).

• Communication: 2 × layer activations per layer per pass, independent of parameter count, growing with sequence length and batch.
• Extremely frequent synchronization (every Transformer layer), requiring low latency and high bandwidth: keep TP inside a node (NVLink) or under the same leaf switch.
• Best for: the natural choice for single-node multi-GPU; for 1000+ GPUs, TP degree should generally stay ≤ 8 to avoid cross-switch amplification.

2.4 Pipeline Parallelism (PP) — Partition Layers

Split Transformer layers into contiguous stages; each GPU computes one stage, and data flows through like an assembly line.

• Communication: only 2 × inter-stage activations per step (one send/recv) — the cheapest of all strategies.
• Cost: pipeline bubble; with microbatch count = 2×(PP−1) the bubble is theoretically minimal (GPipe); PP ≥ 4 typically uses 1F1B scheduling.
• Best for: memory expansion when memory is tight and nodes must be crossed; complements TP (TP handles inside a layer, PP handles between layers).

2.5 Sequence Parallelism (SP) — Partition the Sequence

Split the sequence dimension and shard token-wise ops (LayerNorm / Dropout) together with TP, saving activation memory with almost no extra communication.

• Best for: ultra-long context (e.g. 128K/1M tokens) training where activation memory grows linearly with sequence.
• In practice often combined as "TP + SP" (e.g. Megatron-LM sequence parallelism).

2.6 MoE Expert Parallelism (EP) — Partition Experts

MoE expert networks are sharded across GPUs; tokens are routed via All-to-All to the expert's GPU, computed, and routed back.

• Communication: 2 × routed token volume, growing with batch, and it is All-to-All (global pattern) — the most topology-dependent.
• Best for: MoE models (Mixtral, DeepSeek-MoE) training and inference; typically needs high bandwidth and low-congestion networks (fully-connected topology / high-radix switches).

3. Mixed Parallelism: How to Combine

Real training almost never uses a single strategy. A typical LLM training run looks like:

LLM training = Data Parallel (outermost) × Tensor Parallel (within node) × Pipeline Parallel (across nodes) × Sequence/Expert Parallel (optional)

Combination principles:

1. TP first as the foundation: with 8-GPU H100 nodes fully interconnected via NVLink, TP=8 is affordable; TP should not cross nodes (unless the node has fewer than 8 GPUs).
2. PP sized by the memory gap: for 70B on 8×80G nodes, TP=8 is still not enough — PP=2+ is needed; PP may cross nodes (its communication volume is small).
3. DP fills the remaining compute: use DP (or ZeRO/FSDP) for whatever compute is left.
4. EP only for MoE: dense models have no need for All-to-All.

Typical Configurations (BF16 full-parameter training)

Model	Per-GPU memory	Typical parallelism	Min GPUs	Notes
7B	80 GB	DP/ZeRO-3	8–32	Single node is enough; main bottleneck is communication ratio
13B	80 GB	TP=8 + DP	16–64	TP=8 within node as the base
70B	80 GB	TP=8 + PP=2~4 + DP	64–256	TP within node, PP across nodes
175B+	80 GB	TP=8 × PP=8~16 × DP	512+	Needs 3D hybrid parallelism + dense network
MoE 1000-GPU	80 GB	TP=4~8 × EP + DP	1000+	Heavy All-to-All traffic; needs high-radix topology

4. Communication Cost Estimation: Is the Network Enough?

Estimate the communication ratio before selecting hardware:

Per-step communication ≈ 2×params (gradient sync) + 2×TP layer activations (TP) + 2×inter-stage activations (PP) + 2×routed tokens (EP)
Network demand ≈ communication volume ÷ (aggregate bandwidth × steps × utilization)

Engineering rules of thumb:

• AllReduce gradient sync (DP/ZeRO): ~14 GB per step for 7B; on a 1000-GPU cluster with NDR 400G IB (~50 GB/s per GPU), the ratio is roughly 5–15%.
• TP=8 layer AllReduce: activations grow with sequence; at 4K sequence and batch 512, a single layer can move GBs; TP communication is typically 10–25%.
• If the estimated communication ratio exceeds 30%, reduce TP or increase batch instead of adding GPUs — with a high communication ratio, adding GPUs can be net-negative (communication time exceeds compute savings).

5. Selection Decision Tree

Model doesn't fit on one GPU?
├─ Small memory gap (≤2×): ZeRO-3 / FSDP, simplest
├─ Large gap but fits within a node: TP=8 → add PP if still short
├─ Must cross nodes: PP across nodes (cheap) + TP within nodes
└─ MoE model: EP + high-radix network; verify All-to-All bandwidth first

Network-side checklist:
├─ Within node: NVLink (required for TP)
├─ Across nodes: IB/RoCE full-mesh or high-radix (required for PP/EP)
└─ Rack/cluster level: even ECMP hashing + PFC/ECN (required for AllReduce/All-to-All)

6. Common Misconceptions

• "More GPUs = faster": once communication exceeds compute, scaling efficiency collapses; for 7B above 512 GPUs, a larger batch usually beats more GPUs.
• "Bigger TP is better": TP has the densest communication; TP=16 across nodes can lose all its gains to AllReduce latency.
• "ZeRO-3 is a silver bullet": ZeRO-3 communicates ~50% more than true DP; for 1000-GPU training, prefer true DP with full model replicas over ZeRO-3.
• "Only model parallelism matters, not topology": EP / large-DP AllReduce needs uniform cluster bandwidth; Fat-Tree and Dragonfly hash very differently — always measure with nccl-tests first.

Next: once the parallelism strategy is set, see NCCL Tuning in Practice for whether the network can back it up.