Acceptance & Performance Baselines for a 384-GPU Cluster: From BOM Checks to NCCL busbw

The worst acceptance habit is "it runs, so it passes" — running only proves function exists, not that performance meets the bar, nor where the boundaries are. This article is the acceptance path of a 384-GPU L40S IB cluster: layered verification, baseline archiving, layer-by-layer debugging — every step falsifies rather than confirms.

1. Acceptance Philosophy: Falsify Layer by Layer, Not "Runs = Pass"

Acceptance is not one big inspection; it is layered falsification:

Layer	What is verified	Pass criterion
Hardware	BOM vs. reality	24×32GB@4800, PCIe 16GT/s x16, 7 NIC functions
Network	IB links & topology correct	ibstat Active, ib_write_bw near line rate, topology locality working
System	Version consistency + unified params	Package versions identical, Ansible --check zero changes
Performance	Baselines met	STREAM/HPL/NCCL in expected ranges
Stability	Long run without blowups	72h stress: no OOM/hang/derating

Judgment: each layer passing only proves "this layer is fine" — it does not certify the layer above. Hence acceptance must go bottom-up, archiving each layer independently.

2. Hardware Verification: BOM Is Not "Correct by Default"

• Power-on checks per node: dmidecode (memory), lspci (PCIe rates, NIC count), nvidia-smi (8 GPUs recognized);
• Topology check: nvidia-smi topo -m confirms GPU/NIC-to-NUMA ownership; any deviation → rebuild the affinity script;
• RAID check: system volume RAID1, write cache Enabled (storcli).

3. Network & Performance Baselines: Archive Three Tables

IB network baseline (per node):

ibstat                  # Port Active, Rate 200
ib_write_bw -a          # near 200G line rate per port
ib_read_lat             # latency in expected range

Compute baselines (sampled + full):

Benchmark	Target	Purpose
STREAM	85%+ of theoretical	Memory subsystem health
HPL (CPU/GPU)	Expected efficiency range	Compute delivery
NCCL allreduce	busbw linear in node count	Topology correctness
FIO	Storage bandwidth met	Data plane

Key: baselines are not "measure once" — they are archived with an environment snapshot (versions, params, topology). Otherwise in three months you cannot tell which environment the baseline came from.

4. When Performance Falls Short: Debug Layer by Layer, Don't Tune Blind

When NCCL busbw does not scale linearly, debug in order:

1. Topology: are the 4 ports really in the 4 rails? Wrong port = dead topology (verify with ibstat/switch port mapping);
2. Driver/mode: LINK_TYPE in IB? MLNX_OFED version matched?
3. System: C-states disabling deep sleep? IRQ affinity pinned right?
4. Params: are NCCL_GDR/P2P effective (NCCL_DEBUG=INFO shows the path)?
5. Application: core binding correct? Message size saturated?

Principle: change one variable at a time, re-measure, record, compare. Performance "found" by mixing three params cannot be rolled back, reproduced, or archived.

5. Delivery Checklist

1. Hardware: BOM, topology, RAID verified; deviations logged;
2. Network: ibstat/ib_write_bw/ib_read_lat per node;
3. Performance: STREAM/HPL/NCCL/FIO baselines + environment snapshot archived;
4. Stability: 72h stress passed;
5. Debug path documented: the layer-by-layer checklist fixed into the delivery doc.

---

Based on field measurements from a 384-GPU L40S IB cluster delivery; exact versions and commands follow on-site validation.