Communication Ledger: All-Reduce Bandwidth Scaling

In large-model training, the most underestimated ledger is communication: gradient sync happens every step, and its time grows with scale while compute only grows linearly with cards — whoever wins this race decides scaling efficiency. This article derives the all-reduce bandwidth model. Method framework references the open-source book Understanding AI Infra (Apache-2.0); formulas and examples are derived by this site and are reproducible.

1. How Much Data One All-Reduce Moves

Every training step needs an All-Reduce over gradients: N cards sum their per-card gradients and broadcast the result. Data = params × bytes per param:

$$Grad\_bytes = N_{params} \times bytes\_per\_param \times 2 \quad(\text{depending on algorithm, sum + broadcast})$$

70B FP16: gradients alone are 140 GB, and the all-reduce exchanges this 140 GB across all cards (in terms of total message traffic).

2. Ring Algorithm Bandwidth Model

NCCL's default ring algorithm chains N cards into a ring; each card talks only to its neighbors. Key result:

$$T_{allreduce} = \frac{2 \times (N-1)}{N} \times \frac{Msg}{BW}, \quad busbw = BW \times \frac{2(N-1)}{N}$$

• N cards, per-link bandwidth BW, message Msg;
• The larger the ring, the closer busbw approaches 2×BW (theoretical upper bound);
• So an 8-card in-node ring gives busbw ≈ 1.75× one link — not 8×. This is the most commonly miscalculated ledger.

Cards N	busbw/BW	Note
2	1.0×	duplex pair
4	1.5×
8	1.75×	single-node ceiling
16 (2 nodes)	1.875×	capped by slowest link (inter-node IB)

3. Why Large-Model Training Needs IB

70B × 2 bytes = 140 GB/step gradients. If the parameter plane is 25G Ethernet (≈3 GB/s):

$$T \approx \frac{2 \times (N-1)}{N} \times \frac{140\ GB}{3\ GB/s} \approx 90\ s/step$$

But one training step usually takes seconds to tens of seconds — communication eats most of it and scaling efficiency approaches zero. Switch to 4×200G HDR IB (single-node aggregate 800Gbps ≈ 100 GB/s):

$$T \approx 1.75 \times \frac{140\ GB}{100\ GB/s} \approx 2.5\ s$$

Takeaway: the quantitative criterion for network selection is not "how fast" but comm time vs compute time — if comm ratio >30%, optimize topology or switch networks.

4. L40S Without NVLink: Intra-Node Ceiling

L40S in-node P2P goes over PCIe Gen4×16 (≈32 GB/s one-way). 8-card all-reduce:

$$busbw_{L40S} \approx 1.75 \times 32\ GB/s \approx 56\ GB/s$$

Compared with NVLink-equipped H-series in-node (≈300-900 GB/s), that is an order of magnitude. This is the quantitative origin of the L40S Without NVLink article: gradient all-reduce on L40S is intra-node communication-bound.

5. Reproducible Python

def allreduce_time(msg_gb, bw_gbps, n, comm_overhead=0.1):
    """Ring all-reduce time: busbw = 2(n-1)/n × BW; returns seconds (example incl. overhead)"""
    busbw = bw_gbps * (2 * (n - 1) / n) / 8        # GB/s
    t = 2 * (n - 1) / n * msg_gb / busbw * (1 + comm_overhead)
    return t

allreduce_time(msg_gb=140, bw_gbps=800, n=48)      # 4×200G IB aggregate
# ≈ 0.45 s (with 10% protocol overhead; actual depends on SM/topology, see Ch.3 measurements)

6. Checklist

1. Do the communication ledger before choosing a network: comm/compute < 30% acceptable, otherwise use IB or change parallelism;
2. Use actual PCIe/NVLink bandwidth for intra-node comm (L40S ≈ 32 GB/s, not spec marketing);
3. Compare measured busbw with the model: deviation >20% — check topology locality (see Rail Topology) and whether SHARP is active;
4. Combine this ledger with the Training Resource Ledger for scaling estimates.

---

Derived and organized by this site; method framework references the open-source book Understanding AI Infra (Apache-2.0). Version numbers and measurements are subject to on-site conditions.