Rail Topology + 1.5:1 Oversubscription: Design a 48-Node Cluster Like a 10K-GPU One

Someone will ask: 48 nodes, 384 GPUs — why not a "simple" single-layer Leaf instead of a 4-Leaf + 4-Spine Fat-Tree? The answer lives in the collective communication pattern: NCCL allreduce is network-wide broadcast-style traffic, and the moment oversubscription exists, the communication wall appears. This article walks through the ledger behind a rail-optimized topology.

1. The Topology Ledger: Oversubscription Is Not Just "Bandwidth Discounted"

Option	Structure	Ratio	Consequence
Single-layer Leaf	All nodes on one switch	1:1	Fine at small scale; 48×200G into one 51.2T box is unrealistic on ports and fault domains
Classic Fat-Tree	1:1 full mesh uplinks	1:1 (non-blocking)	Switch cost doubles
Rail-optimized Fat-Tree	4 Leaf + 4 Spine, 1.5:1	48×200G down : 16×400G up	Oversubscribed, but mitigated by topology locality + SHARP

The plan picks the third. Why "oversubscribed" is correct: training traffic is not uniform random flow — it is patterned collective communication (between same-indexed GPUs, between neighboring nodes). Rail topology keeps the hot flows within one rail, one hop. The share of truly cross-rail traffic is far below the naive 1/1.5 intuition.

2. Rail Optimization: Same-Indexed GPUs One Hop Apart

Topology rules:

• Each node's 4 compute ports connect to the Leaf of rail0-3 respectively;
• Compute traffic of GPU N lands entirely in rail N — collectives between same-indexed GPUs complete in one hop within the rail;
• Cross-rail traffic forwards through the Spine (IB has no L3; subnet routing by the SM's LID tables). With Adaptive Routing enabled, packets pick the shortest available path per message and the multiple spines load-balance naturally.

Judgment: rail optimization is, in essence, mapping the communication pattern onto the topology — the hottest communication paths get the shortest paths. This is the standard play of 10K-GPU clusters; adopting it early at 384 GPUs costs only four extra spines.

3. Breakout and the 1.5:1 Tradeoff

• Leaf uses the Quantum-2 QM9700 (64×400G NDR ports, supporting 2×200G breakout):
  • 48×200G downlink occupies only 24 physical 400G ports (breakout 2×200G);
  • The remaining 16×400G uplinks reach the 4 spines;
  • Ratio = 48×200G : 16×400G = 1.5:1;
• Mitigations: SHARP (in-network aggregation on the switch) offloads the allreduce tree, so the reduce phase no longer saturates host bandwidth; Adaptive Routing flattens uplink load across paths.

Engineering judgment: 1.5:1 oversubscription is an acceptable engineering compromise for a training cluster — it buys "half the switches + half the cabling." What you actually monitor is not the ratio number but whether measured NCCL busbw grows linearly with node count; if not, check topology locality first, then whether SHARP is active.

4. Single-Leaf Redundancy and Phase-2 Expansion

• One Leaf per rail: this phase keeps one cold-spare whole unit — IB switches need no per-port configuration; plug in and the SM discovers it automatically;
• Phase-2 expansion: after adding Leaf/Spine, the SM auto-discovers, LID routing tables recompute; no manual tables;
• Growth boundary: 48 → 96 nodes: expand each rail with one more Leaf plus uplink capacity; beyond that (10K-GPU class), each rail gets multiple Leaves with a larger ratio or a 2:1 internal compromise — the Fat-Tree hierarchy must be re-engineered.

Scale	Leaf/Spine shape	Suggested ratio
≤32 nodes	Single-layer Leaf (acceptable)	1:1
48-96 nodes	4 Leaf + 4 Spine	1.5:1
10K-GPU class	Multi-rail, multi-Leaf + full-mesh Spine	≤2:1 (compensated by SHARP/adaptive routing)

5. Delivery Checklist

1. Each node's 4 ports land in rail0-3; verify per-port rail membership (wrong rail = dead topology);
2. After breakout configuration, ibstat confirms 200G per port;
3. Enable SHARP: confirm the switch aggregation engine is online, and NCCL_DEBUG=INFO shows in-network aggregation;
4. Baseline: run allreduce at 2 → 4 → 8 nodes; busbw must scale linearly for the topology to be correct;
5. One cold-spare Leaf power-on drill of SM auto-discovery.

---

Based on field measurements from a 384-GPU L40S IB cluster delivery; exact versions and commands follow on-site validation.