NCCL Tuning in Practice: Locating Communication Bottlenecks in a 10K-GPU Cluster

Abstract: NCCL is NVIDIA's multi-GPU communication library and the "throat" of LLM training throughput. This article follows a reproducible order — baseline first → trace the communication path → NIC-side tuning → network-side tuning → case review. The core idea is one sentence: quantify first, then act, change one variable at a time.

1. Baseline First: Know Your Network's Ceiling

Before any tuning, establish a baseline with nccl-tests — the industry-standard "health check".

# Build
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests && make MPI=1 MPI_HOME=/usr/mpi/gcc/openmpi-4.1.5a1

# 8-GPU single-node baseline (-g 1 = 1 GPU/process)
mpirun -np 8 -hostfile hosts ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

# Cross-node: 256 GPUs (8 per node × 32 nodes)
mpirun -np 256 -hostfile hosts32 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

Key metrics:

Metric	Meaning	Healthy reference (NDR 400G)
busbw	Bus bandwidth (effective per-GPU)	Single node 8 GPUs ≥ 350 GB/s (NVLink); cross-node ≥ 45 GB/s
algbw	Algorithmic bandwidth	busbw ≈ algbw × 2 (AllReduce)
Scale ratio	256-GPU vs 8-GPU busbw	≥ 80% healthy; < 60% means a network problem

WARNING

Testing only small messages (< 1MB) does not represent training load. Measure 64MB–8GB large messages — training gradient sync is all large messages; small messages test latency, not bandwidth.

2. Trace the Communication Path: What an AllReduce Passes Through

GPU memory → (NVLink) → in-node aggregation → (PCIe/IB HCA) → NIC
   → switch (leaf) → (spine) → remote NIC → (PCIe/NVLink) → remote GPU

Each stage has typical bottlenecks. Fix the inspection order: software before hardware, single points before topology.

Stage	Typical bottleneck	Verification
GPU→GPU (in-node)	NVLink downgrade/degradation	nvidia-smi nvlink -s, nvidia-smi topo -m
PCIe	PCIe Gen downgrade, shared switch bandwidth	lspci -vvv (LinkCap/LinkSta)
HCA	Outdated firmware/driver, port rate negotiation	ibstat / ibstatus (Rate: 400 Gb/s)
Switch	Congestion, drops, CRC errors	ethtool -S / ibswtraceroute
Topology layer	ECMP hash imbalance, cross-pod aggregation, PFC deadlock	nvidia-smi topo -m + communication graph

3. NIC-Side Tuning: NCCL Environment Variables

3.1 Enable RDMA and Pick the Right Transport

export NCCL_IB_DISABLE=0          # default; explicit confirmation to use IB
export NCCL_IB_TIMEOUT=22         # retransmission timeout; higher = more stable (default 22)
export NCCL_IB_RETRY_CNT=7        # retry count
export NCCL_IB_QPS_PER_CONNECTION=8   # 400G NICs: ≥4 recommended
export NCCL_IB_GID_INDEX=3        # mandatory for RoCEv2; leave default for IB

3.2 Protocol Selection

export NCCL_PROTO=LL/LL128/Simple  # pick one by message size

• LL (Low Latency): 8-byte granularity, lowest latency, for small messages.
• LL128: 128-byte aligned, the default recommendation for LLM training (gradients are typically 4B-aligned multiples of 128).
• Simple: full reliable transport, best throughput for very large messages, but higher latency.

Rule of thumb: LL for small communications like embeddings, LL128 for main gradient sync, Simple only for IO-heavy sync.

3.3 Topology Awareness and Algorithm Selection

export NCCL_TOPO_FILE=/path/to/custom_topology.xml  # specify manually when topology file missing
export NCCL_ALGO=Tree/Ring/CollnetDirect/CollnetChain  # pick by topology

• Small scale (≤8 nodes): Ring is usually enough.
• Large Fat-Tree: Tree is significantly better than Ring — Ring underutilizes links at scale.
• With SHARP (in-switch aggregation): CollnetDirect offloads AllReduce to switches, doubling throughput — requires switch support and SHARP enabled.
• Run once with NCCL_DEBUG=INFO to see the actual algorithm and topology:

export NCCL_DEBUG=INFO
mpirun -np 256 ... ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 2>&1 | grep -E "NCCL INFO.*(Ring|Tree|CollNet|topo)"

3.4 The Right Way to Tune Parameters

Change one variable at a time and re-run the identical baseline after each change. Recommended order:

1. NCCL_PROTO=LL128 (benefits most scenarios immediately)
2. NCCL_ALGO=Tree (compare against Ring when ≥64 GPUs)
3. NCCL_IB_QPS_PER_CONNECTION 4→8
4. NCCL_BUFFSIZE (default 4MB; 8MB for large messages — watch memory overhead)
5. NCCL_IB_TIMEOUT 22→23/24 (only when retransmissions are observed)

4. Network-Side Tuning: Bottlenecks Live in Switches and Topology

4.1 PFC (Priority Flow Control) and ECN

RDMA relies on lossless networks: a drop means retransmission, and retransmission collapses throughput.

# Mellanox switches: enable PFC and ECN (400G example)
mlnx_qos -i eth2 --pfc=0,0,0,0,1,0,0,0        # PFC for the RoCE priority
mlnx_qos -i eth2 --ecn=1 --trust=dscp          # enable ECN marking

• PFC deadlock loops are a classic 10K-cluster failure: an overloaded link triggers PFC back-pressure that propagates into a loop, showing as "overall throughput plunge + one link's counters climbing". Check: ethtool -S <if> | grep prio for continuously increasing rx_prioX_pause.
• ECN thresholds: too low causes frequent marking and throttling; too high loses the congestion signal. Start around 100KB/queue and tune.

4.2 ECMP Hashing: The Invisible Cause of Imbalanced Traffic

In Fat-Tree, AllReduce aggregation traffic is distributed by ECMP across equal-cost paths. Default 5-tuple hashing distributes same-source-same-destination large flows poorly, leaving one link saturated while others idle.

# Mellanox switches: enable enhanced hash
mlnx_ecmp -i eth2 --enhanced_hash en,eth,l4_src_port,l4_dst_port

Verify even hashing: perftest endpoint pairs + switch show telemetry, or compare nccl-tests result variance across different rank combinations.

4.3 Topology Selection: Fat-Tree vs Dragonfly

Topology	AllReduce behavior	Best for
Fat-Tree	Even, predictable, Tree-algorithm friendly	Mainstream 10K-GPU training
Dragonfly	Fewer hops, lower latency, but global links congest easily	MPI mixed loads in scientific computing
Fully-connected (high radix)	Best All-to-All (EP)	MoE training

10K-GPU MoE training has the highest network demands: 2 full-cluster All-to-All per step — Fat-Tree + even ECMP hashing + PFC/ECN, all required.

5. Real Case: AllReduce Stutter in a 10K-GPU Cluster

Symptom: 256 GPUs across 32 nodes; after a driver upgrade, throughput dropped from 5200 to 3100 tokens/s, and nccl-tests busbw fell from 46 to 22 GB/s.

Investigation (in order, one variable at a time):

Step	Action	Finding
1	nccl-tests 8 GPUs single node	Normal (350 GB/s), NVLink excluded
2	2 nodes, 16 GPUs cross-node	Normal (45 GB/s)
3	32 nodes, 256 GPUs	22 GB/s — problem confirmed at topology layer
4	NCCL_DEBUG=INFO for algorithm	Ring selected (default)
5	Set NCCL_ALGO=Tree	Recovered to 44 GB/s ✅
6	Add NCCL_IB_QPS_PER_CONNECTION=8	46 GB/s, back at baseline

Root cause: the driver upgrade changed NCCL's default algorithm from Tree to Ring (version behavior difference). A 32-node Ring chain serializes AllReduce through 256 ranks; Tree aggregates at switches and bandwidth recovered immediately.

Lesson: after any driver/firmware/NCCL upgrade, re-run the baselines — NCCL defaults change between versions; upgrades are never transparent.

6. Tuning Checklist

• [ ] nccl-tests large-message (≥1GB) baseline established; cross-node busbw ≥ 40 GB/s (400G IB)
• [ ] In-node NVLink healthy (nvidia-smi nvlink -s no degradation)
• [ ] HCA rate negotiated correctly (ibstatus shows 400 Gb/s)
• [ ] NCCL_PROTO=LL128 for large-message scenarios
• [ ] NCCL_ALGO=Tree vs Ring compared when ≥64 GPUs
• [ ] NCCL_IB_GID_INDEX=3 set for RoCE environments
• [ ] Switch PFC enabled, ECN thresholds tuned, rx_prioX_pause not growing
• [ ] ECMP enhanced hash enabled, hashing even
• [ ] Baselines re-run after driver/firmware upgrades