QA Highlight: How to Design an InfiniBand Network for a 10K-GPU AI Training Cluster?

This post is distilled from a real Q&A session with the AI-HPC Assistant (chat.ai-hpc.org) — the answer is grounded in the 14-chapter engineering knowledge base (mainly Chapter 03-network) and then curated and extended by hand. The assistant works bilingually; just ask.

Question

How to design an InfiniBand network for a 10K-scale AI training cluster?

Core Principle

Distributed training performance is no longer compute-bound — it is network-bound. For large-scale LLM training, inefficient RDMA communication can reduce cluster efficiency (MFU) by more than 40% (KB · Chapter 03-network). The fabric must therefore be treated as a first-class performance domain, not an afterthought.

I. Topology: Fat-Tree as the Mainstream Choice

• Two-tier Fat-Tree (Leaf-Spine): 10K-scale clusters typically use non-blocking (1:1 oversubscription) or low-oversubscription (e.g., 1:2) fat-trees to guarantee bandwidth for collective operations.
• Multi-Rail design: Each GPU server carries multiple HCAs (e.g., 8 NICs for 8 GPUs, 400G each), with each NIC on its own uplink to avoid single-point congestion.
• Rail-optimized topology: Align the same GPU ordinal (e.g., GPU 0 on every node) onto the same rail plane, so NCCL can prefer same-plane communication and reduce cross-Spine hops.

II. Rate and Port Planning

• Mainstream choice: NDR 400G (HDR 200G as the runner-up), with GPU:NIC ratios such as 1:1 or 1:2.
• Switch port count × tiers must be derived backward from cluster scale; reserve 20–30% spare ports for expansion and failover.

III. Key Tuning Practices (Chapter 03-network Methodology)

1. Enable GPUDirect RDMA: GPUs DMA directly to/from NICs, bypassing host-memory copies.
2. Adaptive Routing: enable on Spine switches to dynamically bypass congested links.
3. Congestion control: configure DCQCN / timestamp-based congestion control to suppress incast.
4. NCCL tuning: set NCCL_IB_HCA, NCCL_IB_GID_INDEX, NCCL_ALGO (Ring/Tree/CollNet), NCCL_NET_GDR_LEVEL etc. to match the topology.
5. MTU & QoS: set a uniform 4096 MTU and partition SL/VL priorities by traffic class.

IV. Supporting Design

• Liquid cooling: with H100 at ~700W per GPU and 100kW+ per rack, liquid cooling is becoming mandatory; high-density switches must also be included in cooling planning (KB · Chapter 01-hardware).
• Monitoring & acceptance: deploy ibstat, perfquery, ibdiagnet for link health checks; run NCCL AllReduce benchmarks before production to validate effective bandwidth.

V. Field-Experience Reference

The site owner (Zhang Donghao) provided InfiniBand high-speed networking and cluster support for a 10K-GPU AI training cluster of a Shanghai telecom operator, and led full-cycle delivery of large AI/HPC clusters at Tsinghua University and Peking University. For finer tuning parameters and acceptance checklists, see Chapter 03-network.

Note: The topology and tuning above are engineering-practice summaries. The exact oversubscription ratio, port counts, and NCCL parameters must be determined by benchmarking against your actual cluster scale and communication patterns; the knowledge base does not yet contain a full 10K-scale port-planning table — combine with NVIDIA/Mellanox design guides for detailed simulation.

---

Follow-ups (try the assistant): How to choose the fat-tree oversubscription ratio? What's the difference between NCCL Ring and Tree algorithms? How to tune DCQCN? → Chat with the AI-HPC Assistant