Supernode Ledger: Capacity, Throughput, and Sync Cost

When one card's VRAM can't hold the model and one card's bandwidth can't feed it, the industry answer is the supernode — wrapping 8~64 GPUs into a single "big GPU" abstraction via high-speed in-node interconnect. But the wrap has a price: capacity adds linearly, bandwidth is topology-limited, and sync cost grows with scale. This article settles the supernode's three ledgers: aggregated capacity, aggregated throughput, and sync cost. Method framework references the open-source book Understanding AI Infra (Apache-2.0); formulas and examples are derived by this site and are reproducible.

1. Why a Supernode

Two reasons "more cards" is needed yet one card isn't enough:

1. VRAM wall: 700B FP16 weights are 1.4 TB; an 80G card can't hold them — needs N cards;
2. Bandwidth wall: decode reads all weights per token (see Accelerator Arithmetic); single-card bandwidth caps single-card throughput.

The essence of a supernode: add N cards' capacity, and treat the card-to-card interconnect as "the new memory bandwidth." The design question becomes: how fast must the interconnect be so sync cost doesn't eat the parallelism gains?

2. Capacity Ledger: Linear Addition, Minus the Memory Wall

$$Cap_{node} = N \times (VRAM\_per\_card), \quad Peak_{node} = N \times Peak\_per\_card$$

Capacity and compute grow linearly — but "deliverable" capacity must deduct three items:

Deduction	Note
Parallel-sharding redundancy	TP/PP keeps overlapping state per card; aggregate VRAM utilization is 70-90%
KV/activation residency	inference supernodes spread KV across cards; that VRAM isn't available for weights
Communication buffers	NCCL reserves buffers; share grows with scale

Takeaway: plan supernode size with "aggregate VRAM × 0.8" as usable capacity, not N×VRAM.

3. Throughput Ledger: In-Node Interconnect Is the Soul

The difference between a supernode and "a pile of servers" is entirely in-node interconnect bandwidth. Order-of-magnitude comparison (one-way, approximate):

Interconnect	Bandwidth	vs 200G IB
NVLink 3 (H-series)	~900 GB/s	~36×
NVLink 4 (newer gen)	~1800 GB/s	~72×
PCIe Gen4×16 (L40S in-node)	~32 GB/s	~1.3×
200G HDR IB (inter-node)	~25 GB/s	1×
400G NDR IB (inter-node)	~50 GB/s	2×

This is the supernode's core ledger: move communication from "inter-node IB 25GB/s" to "in-node 900GB/s" and the same gradient sync runs 36× faster. Conversely for L40S: in-node PCIe is only ~32GB/s, roughly the same order as inter-node IB — an L40S "supernode" experience is almost no supernode (see L40S Without NVLink); at scale you must compensate with inter-node topology.

4. Sync Cost Ledger: Comm Time vs Compute Time

Per-step collective time (ring model, see Communication Ledger):

$$T_{sync} = \frac{2(N-1)}{N} \times \frac{Msg}{BW_{node}}, \quad sync\_ratio = \frac{T_{sync}}{T_{compute}+T_{sync}}$$

Supernode sizing criterion: sync ratio < 30%. Two corollaries:

• The bigger the supernode, the closer busbw approaches 2×BW for a fixed gradient Msg — in-node interconnect busbw advantage is the premise for scaling a supernode;
• Put comm-frequent parallelism inside the supernode (TP/PP sync every layer), and comm-sparse parallelism across supernodes (DP once per step) — this is the standard "TP inside, DP across" architecture.

5. Reproducible Python

def supernode_sync(msg_gb, bw_gbps, n, compute_s):
    """Ring sync time & ratio inside a supernode; bw = aggregate in-node bandwidth (Gbps)"""
    busbw = bw_gbps * (2 * (n - 1) / n) / 8          # GB/s
    t_sync = 2 * (n - 1) / n * msg_gb / busbw
    ratio = t_sync / (compute_s + t_sync)
    return t_sync, ratio

supernode_sync(msg_gb=140, bw_gbps=7200, n=8, compute_s=5)   # NVLink aggregate ~7.2Tbps
# (0.155 s, 3.0%)  → in-node sync ~3%, negligible
supernode_sync(msg_gb=140, bw_gbps=256, n=8, compute_s=5)    # PCIe aggregate ~256Gbps
# (4.38 s, 46.7%) → 47% — comm eats half the time, scaling collapses

6. Checklist

1. Size the supernode first: aggregate VRAM × 0.8 ≥ model + KV, else grow scale or change parallelism;
2. Check the interconnect order of magnitude: if in-node BW ≤ 2× inter-node BW, the "supernode" is nominal — design as ordinary nodes;
3. Grow supernode scale only while sync ratio <30%; otherwise fix interconnect or parallelism first;
4. Place parallelism: TP/PP inside the supernode, DP across — don't mix;
5. Compare measured per-step comm time with the model (see Acceptance & Baselines).

---

Derived and organized by this site; method framework references the open-source book Understanding AI Infra (Apache-2.0). Version numbers and measurements are subject to on-site conditions.