L40S Without NVLink: NCCL and the Per-Node Communication Ledger on a PCIe Topology

An 8×L40S node has no NVLink between GPUs. That is not just "one missing high-speed interconnect" — it resets the per-node communication model from an NVLink mesh back to a PCIe tree plus CPU bridges. NCCL path selection, core binding and cross-socket overhead all need to be re-derived. This article records the per-node communication facts and engineering judgments from delivering a 384-GPU L40S IB cluster.

1. Conclusions First: Three Implicit Constraints of No NVLink

The L40S is a PCIe Gen4 x16, 350 W passively cooled card; there are no NVLink bridges in an 8-GPU node. Three physical constraints follow — not configuration items, but physics:

Constraint	What It Means	Engineering Consequence
P2P over PCIe	Inter-GPU traffic traverses PCIe switches/CPU bridges, not direct GPU links	Single-direction bandwidth ≈ 32 GB/s (Gen4 x16 theoretical), an order of magnitude below NVLink
GDDR6 memory	48 GB GDDR6 at ≈ 864 GB/s, no HBM capacity/bandwidth combination	Memory-bound kernels must be budgeted on GDDR6 bandwidth
Cross-NUMA must cross the CPU bridge	GPU0-3 on NUMA0, GPU4-7 on NUMA1; inter-socket traffic goes over UPI/QPI	Wrong core binding costs 20-30% of busbw

2. Node Topology: Draw It Before Tuning

The first thing after racking is not training — it is pinning down the topology with nvidia-smi topo -m and lspci:

Key facts (confirm by measurement; vendor layout prevails):

1. Four GPUs per socket: GPU0-3 share a socket with CPU0 and NIC0; GPU4-7 with CPU1 and NIC1;
2. NICs span CPUs: the two ConnectX-7 adapters sit on different NUMA nodes, so a single NIC failure only affects two rails;
3. P2P availability: same-NUMA GPUs can P2P (PIX/PXB); cross-NUMA GPUs go over the CPU bridge (PHB) — cudaDeviceCanAccessPeer may pass, but bandwidth is an order of magnitude lower.

Judgment: "GPU and NIC must share the same NUMA node" is not pedantry; it is a hard precondition for the NCCL remote-memory (RDMA) path. If data must cross the CPU bridge between GPU and NIC, every collective operation pays the penalty directly in busbw.

3. How NCCL Sees This Node

Without NVLink, NCCL does not error out — it degrades to a PCIe tree topology:

• Discovery: NCCL infers hierarchy from NVML/PCIe location; no NVLink means grouping by PCIe switch;
• Algorithm: with 8 GPUs and no NVLink, ring is usually the default and most stable; tree can help for large cross-node messages, but intra-node gains are capped by the PCIe bandwidth ceiling;
• Key knobs: NCCL_P2P_LEVEL, NCCL_NET_GDR_LEVEL decide the path. For PCIe-class cards like the L40S, NCCL_P2P_LEVEL=PHB (allow P2P within the same PCIe domain, host memory for cross-socket) is often more stable than full P2P.

The right way to take a baseline:

# Per-node 8-GPU allreduce baseline (must be measured with core binding)
mpirun --bind-to core -x NCCL_DEBUG=INFO \
  /opt/nccl-tests/build/all_reduce_perf -b 8M -e 512M -f 2 -g 8
# Watch busbw, not algbw (algbw includes algorithmic overhead)

4. Three Variables That Move busbw

Variable	Correct	Wrong	Typical Gap
Core binding	Bind to the CPU socket owning the GPU (taskset/cgroup)	Random process scheduling	-20-30% busbw
GDR	Enable GPUDirect RDMA (NCCL_NET_GDR_LEVEL=PXB to start)	Bounce buffer path	Cross-node throughput halved
Message size	Evaluate at large sizes (≥8 MB) for saturated bandwidth	Judge the node by small messages	Misleading conclusions

Judgment: on a no-NVLink node, intra-node bandwidth is not the selling point. The real lever is GDR + NUMA affinity — it decides how fast each GPU pushes data outward, which is the foundation of cluster-wide collectives.

5. The Gap vs. NVLink Cards: Quantify, Don't Panic

Dimension	8×L40S (no NVLink)	8×NVLink GPU node
GPU-to-GPU single hop	PCIe Gen4 x16 ≈ 32 GB/s (one-way)	NVLink ≈ 300-900 GB/s
Per-node allreduce	Capped by PCIe ceiling	Near-linear scaling
Fit	Data-parallel/large-batch training and inference	Tensor/expert-parallel with high-frequency communication

Engineering conclusion: L40S clusters suit data-parallel, large-batch training and inference. They are not for cutting each node into 8 tensor-parallel ranks chasing peak throughput — the communication wall eats the gains. Match the workload's communication pattern to the platform before counting single-GPU compute.

6. Delivery Checklist

1. Confirm GPU/NIC-to-NUMA mapping with nvidia-smi topo -m; rebuild affinity scripts on any deviation;
2. Bind processes: mpirun --bind-to core or cgroup, GPU and CPU on the same socket;
3. NCCL baseline: record 8M-512M busbw with all_reduce_perf as the reference for future changes;
4. Verify GDR: NCCL_DEBUG=INFO shows RDMA paths, not bounce-buffer fallback;
5. Re-test cross-node: once the node baseline passes, validate with a 2-node IB run to confirm GDR is effective end to end.

---

Based on field measurements from a 384-GPU L40S IB cluster delivery; exact versions and commands follow on-site validation.