GPU Server Hardware Topology: PCIe, NVLink and NVSwitch Interconnect

Summary: Multi-GPU training performance depends not only on GPU compute but on the interconnect between GPUs, GPU↔CPU, and GPU↔NIC. PCIe lanes, NVLink bandwidth, NVSwitch full-mesh—these hardware details decide whether NCCL communication takes seconds or minutes. This post measures the real topology with commands and does the math.

0. Experiment Goals

• Probe the real hardware topology with lspci / nvidia-smi topo
• Compute the bandwidth ledger of PCIe vs NVLink (why NVLink is an order of magnitude faster)
• Understand how NVSwitch full-mesh eliminates the multi-GPU communication bottleneck
• Derive training-deployment and hardware-selection decisions from topology constraints

1. See the Real Topology First: Three Commands

# 1) GPU models and count
nvidia-smi -L

# 2) GPU interconnect topology (PIX/PXB/NODE levels)
nvidia-smi topo -m

# 3) PCIe link width (x16/x8 sets the bandwidth ceiling)
lspci -vvv -s $(nvidia-smi --query-gpu=pci.bus_id --format=csv,noheader | head -1) | grep -i "LnkSta"

Results (example · 8×H100 server):

GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7
GPU0  X    NV18  NV18  NV18  SYS   SYS   SYS   SYS
GPU1  NV18  X    NV18  NV18  SYS   SYS   SYS   SYS
...
GPU4  SYS   SYS   SYS   SYS   X    NV18  NV18  NV18

Step-by-step:

• NV18: two GPUs are connected by 18 NVLink lanes (H100 full-mesh); communication goes over NVLink, not PCIe.
• SYS: communication across NUMA domains goes through PCIe switches/CPU links—an order of magnitude worse in bandwidth and latency. This is why NCCL must route topology-aware (hardware foundation for the NCCL Tuning post).
• x16 in LnkSta: the PCIe link width from GPU to CPU. If it is x8, bandwidth halves—must be checked during acceptance.

2. The Bandwidth Ledger: PCIe vs NVLink vs NVSwitch

Do the math first:

Interconnect	One-way bandwidth (example)	Notes
PCIe Gen5 x16	~63 GB/s	generic; CPU↔GPU, cross-NUMA GPU
NVLink 4 (18 lanes)	~450 GB/s	GPU↔GPU direct, H100 full-mesh
NVSwitch (H100 platform)	7 NVSwitches, full-mesh	any pair of 8 GPUs at full bandwidth
NIC (IB NDR 400G)	~50 GB/s	cross-node communication

Key conclusion: intra-node NVLink bandwidth is 7× PCIe; cross-node relies on NICs (400G IB ≈ 50GB/s), about 1/9 of NVLink. So the communication bottleneck is layered: intra-node over NVLink (fast), cross-node over IB (slow)—parallelism strategies (tensor parallelism eats NVLink, data parallelism eats IB) are designed along this topology.

3. NVSwitch: How Full-Mesh Is Achieved

How it works:

• With only PCIe switches, 8-GPU servers route GPU↔GPU traffic through the CPU/PCIe switch—concurrent pairwise communication contends for bandwidth.
• NVSwitch: a "GPU-level switch" inside the server, connecting the 8 GPUs into full-mesh at full pairwise bandwidth (H100 platform uses 7 redundant NVSwitches). GPU↔GPU traffic no longer passes through CPU/PCIe; NCCL AllReduce intra-node scales near-linearly.
• How to tell: if nvidia-smi topo -m shows all-NV rows (no SYS), NVSwitch full-mesh is in effect.

Engineering notes:

• Read training as "compute × interconnect": 8-GPU NVLink full-mesh runs tensor parallelism (TP=8) sensibly; if the server is PCIe-interconnected 8-GPU (e.g., some 4090 platforms), TP=8 is dragged down by the PCIe bottleneck—prefer data parallelism + large batch.
• AI/HPC hybrid: HPC jobs (MPI) also eat topology—bind processes by NUMA/NVLink (mpirun --map-by / --bind-to), or cross-NUMA communication silently costs 20-50%.

4. Topology-Aware Deployment Decisions

# Check GPU/NIC NUMA affinity (critical for cross-node training)
# Ideal: GPU and its IB NIC in the same NUMA domain, avoiding detours
lspci | grep -E "NVIDIA|Mellanox" | head -12
nvidia-smi topo -m | grep -i "PIX\|NODE"

Engineering takeaways (copy into selection/acceptance specs):

1. Acceptance must check three things: all-NV in nvidia-smi topo -m (NVSwitch works), PCIe link x16, GPU and IB NIC in the same NUMA domain.
2. 8-GPU NVLink platforms run TP: tensor parallelism eats NVLink bandwidth; TP=8 is a sane config.
3. PCIe platforms avoid large TP: prefer DP/PP, or correct routing with NCCL topology awareness (NCCL_TOPO_FILE).
4. Budget for IB in cross-node: 400G IB ≈ 50GB/s ≈ 1/9 of NVLink; where AllReduce crosses nodes heavily, the network is the real bottleneck.
5. Cooling does not change topology: but power walls throttle clocks; before sustained high load lock power with nvidia-smi -pl so throttling does not corrupt benchmarks (see Liquid Cooling).

5. Engineering Takeaways

1. See the topology before talking performance: nvidia-smi topo -m + lspci are acceptance standard.
2. Layered bandwidth memory: NVLink (~450GB/s) ≫ PCIe (~63GB/s) ≫ IB cross-node (~50GB/s).
3. NVSwitch = GPU-level full-mesh: full config only when topo shows no SYS.
4. TP eats NVLink, DP eats IB: design training strategy along the topology.
5. PCIe x8 is an acceptance trap: halved link, performance directly discounted.
6. NUMA affinity: keep GPU/NIC in the same domain; cross-NUMA silently costs 20-50%.
7. Lock the power wall first: nvidia-smi -pl before benchmarks so throttling does not skew comparisons.

6. Further Reading

• NVIDIA NVLink/NVSwitch whitepaper
• NCCL topology awareness docs
• KB: 01. Hardware & Chips · Liquid Cooling · NCCL Tuning · Selection & Acceptance