03. Cluster Architecture

Chapter focus: This chapter covers key steps from AI-HPC cluster design to benchmarking and tuning.

Outcomes

• baseline benchmark methods (HPL, STREAM)
• CPU/GPU cluster tuning patterns
• practical benchmark toolkit workflows
• MPI environment setup and parallel programming

🧭 Reading Map

• Prerequisites: 02. Hardware & Chips — benchmarks and topology here build on hardware knowledge
• Role: core chapter of Part 1, L1 — from capacity planning and CPU/GPU benchmarks to MPI parallelism and the L40S cluster delivery series; the longest, most delivery-oriented chapter
• Order: CPU theory→estimation→measurement (three steps) → MPI practice (4 articles) → GPU HPL → L40S series (10 articles incl. network/system/software stack) → benchmark toolkit
• Next: 03. Network — the ceiling of cluster performance lives in the network

Articles

CPU Performance Testing in Practice (Theory → Measurement)

CPU Theoretical Peak (Rpeak) Calculation in Practice

compute Rpeak in three steps with real machine specs: AVX-512 32 FLOP/cycle, 104-core node, 80-node cluster; includes a lookup table and verification tools.

阅读文章 →

Estimating HPL Performance: from Theoretical Peak to Deliverable Efficiency

a 91-node per-node dataset: 93.4% single-node average efficiency, 78.3% at 80 nodes, 74.2% memory bandwidth, with a scaling-band estimation formula.

阅读文章 →

CPU HPL in Practice: from Single Node to 80 Nodes

full post-mortem of single-node tuning → 36 nodes → 80 nodes: HPL.dat configs, commands, real results, and pitfalls.

阅读文章 →

CPU Linpack Performance Benchmarking Guide (HPL)

a comprehensive HPL guide on Intel MKL: environment setup, HPL.dat tuning, multi-node testing, and performance analysis.

阅读文章 →

MPI Parallel Programming in Practice

MPI Getting Started: Installing oneAPI and Your First Parallel Environment

offline install of Intel oneAPI Base + HPC Toolkit, silent commands, environment loading, and common pitfalls.

阅读文章 →

MPI Hello World in Practice

compile, single-node multi-rank, and multi-node runs from hello.c, with rank output and fabric flags explained.

阅读文章 →

MPI Programming in Practice: from Hello to Parallel Reduce

compilable code from point-to-point Send/Recv to MPI_Reduce, implementing parallel summation with verification.

阅读文章 →

MPI Point-to-Point in Practice

ring relay with MPI_Sendrecv and a ping-pong latency/bandwidth probe, with cross-node fabric notes.

阅读文章 →

MPI Collectives in Practice: from Bcast to Allreduce

broadcast, parallel sum, and Allreduce global statistics with compilable code, mapped to the HPL P×Q grid.

阅读文章 →

MPI + Slurm in Practice: Scheduling and Performance Tuning

srun/sbatch submission, process layout and pinning, and an HPL tuning retrospective from 1 to 80 nodes.

阅读文章 →

GPU / Heterogeneous & Tools

NVIDIA GPU HPL Benchmarking Guide (Docker)

running HPL on NVIDIA GPUs via Docker: startup parameters, MPI commands, and result analysis.

阅读文章 →

AMD CPU Linpack (HPL) Performance Guide

HPL testing and tuning for AMD EPYC (Genoa/Zen 4): OpenMPI binding, BIOS settings, and multi-node scripts.

阅读文章 →

Memory Bandwidth Testing & STREAM Guide

server memory bandwidth evaluation with STREAM: DDR5 theoretical calculation, compiler optimizations, and case studies.

阅读文章 →

Building an All-in-One HPC Benchmark Toolkit

a portable, containerized toolkit integrating HPL, Intel oneAPI, STREAM, FIO, and OSU benchmarks.

阅读文章 →

GPU Cluster Capacity Planning: From Business Requirements to Racks & Power Budget

total compute, rack layout, power & cooling, cost & cadence.

阅读文章 →

384-GPU L40S IB Cluster: Engineering Thinking

48 nodes × 8×L40S + 4×200G HDR IB: three-plane separation, rail topology, lossless IB and the acceptance philosophy.

Read →

L40S Without NVLink: NCCL & Node Communication

Three implicit constraints, NUMA affinity, busbw baselines and an actionable tuning checklist.

Read →

IB Three-Plane Separation: PKey, SL/VL & IPoIB

Physical isolation of compute/storage/management, PKey 0x100/0x200 isolation, OpenSM failover and verification commands.

Read →

Rail Topology + 1.5:1 Oversubscription

Design a 48-node cluster like a 10K-GPU one: rail-optimized Fat-Tree, 2×200G breakout, SM auto-discovery and growth.

Read →

MLNX_OFED & VPI LINK_TYPE in Practice

Switch dual-mode adapters to IB: mstconfig burning, driver stack, ibstat/ibv_devinfo verification and traps.

Read →

Kickstart + Mirrors + Ansible Delivery

Version consistency across 48 nodes: bulk install, internal YUM/pip mirrors, Ansible rollout and acceptance.

Read →

openEuler Kernel & System Tuning

The BIOS checklist, kernel parameters, NUMA and IRQ affinity — each step verified.

Read →

CUDA Stack Pinning & Bulk Install

Driver/CUDA/cuDNN/NCCL version matrix, internal-mirror installs, NCCL knobs and verification.

Read →

Acceptance & Performance Baselines

Layered verification, archived baselines and layer-by-layer debugging — falsify, don't just run.

Read →

Power & Cooling: The 4.6kW Ledger

Per-node power math, rack density, power sequencing and full-load verification.

Read →