10K-GPU Cluster Hardware Selection & Acceptance Checklist

Abstract: A 10K-GPU cluster is "data-center-level" engineering; selection errors are nearly impossible to reverse afterwards (racks, power, and cooling are all fixed). This article gives a five-dimension selection framework and a four-step acceptance process, all from real large AI/HPC cluster delivery experience — every item corresponds to a pit that was actually stepped in.

1. Five-Dimension Selection Framework

① Compute  —— TFLOPS demand of training/scientific workloads
② Memory   —— capacity + bandwidth (can the model fit? is it fed fast enough?)
③ Interconnect —— in-node NVLink + cross-node IB/RoCE topology
④ Storage  —— parallel filesystem throughput (checkpoint & dataset loading)
⑤ Power & cooling —— rack power density and cooling method (often underestimated)

1.1 Compute & Memory: Reverse-Engineer from the Target Workload

• Define the target model + precision + sequence length, and reverse-engineer the GPU count with the estimation formulas in Distributed Parallelism Overview.
• Leave 20–30% memory headroom: activations, intermediates, and communication buffers (NCCL_BUFFSIZE) all consume memory.
• Bandwidth priority: training > long-context inference > low-concurrency inference.

1.2 Interconnect: Determines Scaling Efficiency

• In-node: 8-GPU NVLink full mesh is the floor; check NVSwitch health (nvidia-smi topo -m).
• Cross-node: IB NDR 400G as the starting point (10K scale); RoCE requires strict PFC/ECN configuration (see NCCL Tuning).
• Topology: Fat-Tree for training; evaluate high-radix/full-mesh for MoE mixed loads.
• Acceptance: nccl-tests with 256 GPUs cross-node busbw ≥ 40 GB/s (400G IB); below that, topology or hashing has a problem.

1.3 Storage: The Most Underestimated Dimension

• Training reads datasets/checkpoints and periodically writes checkpoints; checkpoint write speed determines failure recovery time.
• Reference: parallel filesystem aggregate read ≥ 50 GB/s (1000-GPU scale); checkpoint writes in minutes.
• Common pitfall: slow data loading leaves GPUs idle (IO wait) — it looks like insufficient compute but is actually insufficient storage.

1.4 Power & Cooling: The New 10K-Scale Constraint

• H100 at 700 W/card; full-rack power density reaches 100+ kW/rack, while air-cooling limits around 50–60 kW/rack — beyond that, liquid cooling is mandatory.
• Cold-plate liquid cooling is mainstream, typically 1:1 (servers:CDU); immersion for ultra-high density.
• Power: design for peak (not average), reserve 20% for expansion; UPS + generator transfer time must cover IB switch restart duration.

2. Four-Step Acceptance Process

Principle: single points first, then local, then the whole cluster; every step has a quantified threshold — do not proceed without meeting it.

2.1 Step 1: Single-Node Compute Acceptance (HPL / Peak Test)

# Compute: HPL measures FP64 (HPC view); training scenarios can add BF16 matmul peak
# 8 GPUs per node, take the median of 3 runs, compare against spec ≥ 95%
mpirun -np 8 ./xhpl          # HPL: A100 node ~60+ TFLOPS (FP64)

• Pass line: measured ≥ spec × 95%; > 5% deviation → check drivers, clocks, cooling (throttling).
• Also run nvidia-smi -q -d CLOCK to confirm no abnormal throttling (temperature/power wall).

2.2 Step 2: Memory Bandwidth Acceptance (Stream)

# Stream: measure real HBM bandwidth
./stream_c.exe   # reference: H100 node measured ≥ 3.0 TB/s (spec 3.35)

• Run the multithreaded build to saturate all channels; ≥ 90% of spec is healthy.
• Both HPL and Stream passing proves "can compute + can be fed".

2.3 Step 3: Interconnect Acceptance (nccl-tests + Endpoint Pairs)

# In-node NVLink
mpirun -np 8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1   # 8-GPU busbw ≥ 350 GB/s (H100)

# Cross-node: pair-by-pair + full cluster
mpirun -np 256 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1  # 256-GPU busbw ≥ 40 GB/s

• Pair-by-pair (--hostfile 2 nodes) finds bad links; full-cluster finds topology issues.
• Combine ibstatus (rate negotiation) and ethtool -S (pause frames) for network health.

2.4 Step 4: 72-Hour Stability Burn-In

• Use real training loads (not pure benchmarks) for 72h: mix 2–3 jobs covering training + inference + data loading.
• Monitor: GPU temp/power, memory ECC errors (nvidia-smi -q -d ECC), IB retransmits/drops, filesystem latency.
• Criteria: no fatal errors in 72h, performance degradation < 5%, correctable ECC not growing continuously.

3. Checklist (Ready to Use)

Selection Phase

• [ ] GPU count/memory reverse-engineered from target workload, 20–30% memory headroom
• [ ] Training scenarios prioritize memory bandwidth and interconnect, not just TFLOPS
• [ ] Cross-node IB NDR 400G floor; topology selection (Fat-Tree/high-radix) done
• [ ] Parallel filesystem aggregate bandwidth meets minute-level checkpoint writes
• [ ] Rack power density assessed; liquid cooling defined if beyond air limits
• [ ] Power designed for peak + 20% expansion reserve
• [ ] Ecosystem validated: framework/NCCL/inference engine versions for the chosen architecture

Acceptance Phase

• [ ] HPL: single-node FP64 measured ≥ 95% of spec
• [ ] Stream: memory bandwidth measured ≥ 90% of spec
• [ ] nccl-tests: 8-GPU busbw ≥ 350 GB/s, 256-GPU cross-node ≥ 40 GB/s
• [ ] Pair-by-pair link tests passed; no bad links/rate downgrades
• [ ] 72h real-load burn-in passed; ECC/retransmits/drops not growing
• [ ] Performance degradation < 5%; no fatal errors

4. Common Pitfalls & Handling

Pit	Symptom	Handling
Insufficient power	Random node reboots at peak	Design for peak + 20%; recheck UPS capacity
Air cooling exceeded	Persistent GPU throttling, HPL down to 80%	Switch to liquid cooling or reduce density; don't push through
IB hash imbalance	High nccl-tests variance, one link saturated	Enable ECMP enhanced hash; re-measure
Storage IO bottleneck	Low GPU utilization, data-loading waits	Expand filesystem clients/bandwidth
Driver mismatch	New card + old driver, halved performance	Lock versions; validate on small cluster first
Growing ECC	Memory chip aging	Log and locate; replace under warranty

Further reading: Distributed Parallelism Overview (how to use the cards after selection), NCCL Tuning in Practice (tuning path after interconnect acceptance).