Collective Communication Primitives & Libraries: From MPI to NCCL

Large-model training can spend >60% of time on communication. Communication is abstracted into a set of primitives (broadcast, all-reduce, all-gather, …), and libraries like MPI and NCCL run those primitives at the highest bandwidth on specific hardware. This article explains "what primitives are, how much data each moves, and how to choose the library," complementing the MPI series and Communication Ledger.

1. Communicator: The Precondition of All Communication

Communication isn't "any two machines sending to each other" — it happens inside a pre-established process group (communicator):

MPI_COMM_WORLD = {rank0, rank1, ..., rankN-1}   # global communicator
sub-communicator = sub-groups (e.g. one per node)

• rank: the unique id of a process within the communicator — the addressing scheme;
• The communicator decides who can talk to whom and over which path — this is the starting point of every collective library, corresponding to ncclComm in NCCL.

2. Primitive Classification: Point-to-Point vs Collective

Type	Primitive	Semantics	Data per rank
Point-to-point	Send / Recv	one-to-one	one message
Collective	Broadcast	1 rank sends to all	receives 1 copy
Collective	Scatter	root distributes different data	receives 1/N
Collective	Gather	collect all ranks' data at root	sends 1/N
Collective	All-Gather	gather then broadcast to all	sends 1/N, receives N
Collective	Reduce	reduce to root (sum/max)	sends 1
Collective	All-Reduce	reduce then broadcast result	sends 1, receives 1
Collective	All-to-All	everyone sends N-1 different chunks	sends (N-1)/N

Takeaway: the core primitives for LLM training are All-Reduce (gradient sync) and All-Gather (ZeRO/TP weight aggregation); MoE routing is All-to-All. Different workloads use different primitives with completely different bandwidth ledgers.

3. How Much Data Each Primitive Moves (N cards, message M)

Primitive	Total traffic (theoretical)	Note
Broadcast	N×M	root broadcasts to N ranks
All-Gather	(N-1)×M	each contributes M/N, assembled to M, redistributed
All-Reduce (ring)	2(N-1)/N × M	reduce + broadcast share one path
All-to-All	(N-1)/N × M	each sends N-1 chunks

Conclusion: for the same message, All-Reduce moves ~half of All-Gather's total traffic — the quantitative reason ZeRO training prefers All-Reduce over gather+broadcast.

4. MPI vs NCCL: Positioning Two Kinds of Libraries

	MPI	NCCL
Origin	general HPC message-passing standard	GPU-dedicated collectives
Coverage	p2p + collectives + process mgmt	collectives-focused
Hardware awareness	traditional networks	topology-aware (rail/ring/tree auto)
Typical use	scientific computing, CPU clusters	multi-GPU training/inference
GPU memory	via CUDA-aware support	native GPU memory

Takeaway: GPU large-model training prefers NCCL (topology-aware, highest bandwidth); CPU/mixed HPC workloads use MPI. They're not substitutes but a division of labor — many clusters use MPI for launch/data plane and NCCL for GPU collectives.

5. NCCL Architecture: Topology-Aware Algorithm Selection

ncclComm (communicator)
   ├── topology discovery (PCIe/NVLink/IB positions)
   ├── algorithm selection: ring / tree / (NVLS full-mesh)
   └── channel tuning: multi-channel, rail alignment, SHARP offload

• ring: highest bandwidth, default (see Communication Ledger);
• tree: low latency, for large N (see Collective Topology);
• NVLS: with NVLink full-mesh, one-shot full exchange beats ring.

This is why the same cards can run 2× faster under NCCL than hand-written MPI: it probes topology and picks the best algorithm instead of running one fixed scheme.

6. Reproducible Python: Primitive Traffic Comparison

def collective_traffic(kind, n, msg_mb):
    """Total traffic of each collective primitive (MB, theoretical)"""
    m = msg_mb
    return {
        "broadcast": n * m,
        "all_gather": (n - 1) * m,
        "all_reduce": 2 * (n - 1) / n * m,
        "all_to_all": (n - 1) / n * m,
    }[kind]

collective_traffic("all_reduce", n=8, msg_mb=140e3)
# ≈ 245,000 MB; vs all_gather 980,000 MB → saves 75%

7. Checklist

1. Identify the primitive first: gradient sync→All-Reduce, weight aggregation→All-Gather, MoE routing→All-to-All;
2. For GPU training use NCCL directly — don't let MPI handle GPU collectives;
3. Measure per-primitive busbw with nccl-tests (see nccl-test) and compare with the model;
4. If topology isn't aligned (e.g. cross-node rails), debug NCCL env vars and topology first (see IB Three-Plane).

---

Organized and written by this site for learning purposes; specific versions and measurements are subject to on-site conditions.