PKU HPC Project Delivery Retrospective: Cluster Design for Scientific Computing

Abstract: Unlike AI training clusters, HPC workloads are mixed: MPI scientific computing (CFD, materials, biology) alongside AI training and inference. The delivery challenge is not single-point performance — it is making different workloads coexist on one system without fighting. This article reviews the core decisions of the PKU HPC project delivery.

1. Essential Differences from AI Training Clusters

Dimension	AI training cluster	HPC scientific computing cluster
Primary load	Single large model training	Multi-user, multi-discipline, mixed jobs
Compute type	FP16/BF16 matmul	FP64 simulation + some AI
Communication	AllReduce/All-to-All sync	MPI point-to-point/collective mix
Scheduling	Large jobs exclusive across nodes	Small jobs high-concurrency; priority and fairness
Software ecosystem	PyTorch family	MPI + discipline software (VASP/GROMACS/OpenFOAM etc.)

Core tension: an HPC cluster must split resources between "large training jobs" and "many small simulation jobs"; a wrong scheduling strategy satisfies neither side.

2. Architecture Decisions

2.1 Compute Resources: Heterogeneous Layers

• GPU partition: AI training and GPU-accelerated scientific computing (e.g. GROMACS GPU builds).
• CPU high-memory partition: traditional MPI simulation, pre/post-processing.
• Fat nodes (high core + high memory): memory-intensive apps (genome assembly, sparse solvers).
• Separated by Slurm partitions + QOS for priority control — physical partitions for isolation, QOS for fairness.

2.2 Network: MPI Has Lower Tolerance for Network Imperfection

• MPI scientific computing is latency-sensitive (many small messages) and less bandwidth-hungry than training.
• IB network (low latency) + Fat-Tree; MPI libraries (OpenMPI/Intel MPI) tuned to hardware topology (--mca pml ucx --mca btl ^openib etc.).
• UCX configuration is mandatory for HPC delivery: ucx.conf centrally manages transport selection to stop MPI from taking the wrong channel.

2.3 Storage: Home + Parallel FS + Cache, Three Layers

• Home: NFS-class, light IO, shared cluster-wide.
• Parallel filesystem: big datasets and checkpoints, high-performance concurrent.
• Tiering: read-only scientific data (crystal structure libraries) cached to the high-speed tier to cut repeated compute reads.

2.4 Software Stack: Modularity Is the Lifeline

• Environment Modules (Lmod): multiple compiler/MPI/discipline-software versions coexisting, users load on demand — without modularity, multi-version conflicts eat all operations time.
• Prebuilt common software: VASP, GROMACS, OpenFOAM, LAMMPS per user requirement list, documented.
• Containers (Singularity) as a supplement: user-customized environments; read-only image mounts for security.

3. Acceptance Focus: Differences from AI Clusters

Step 1  HPL (FP64) single-node acceptance   → scientific compute floor
Step 2  OSU MPI benchmarks                  → point-to-point latency/bandwidth (core HPC metric)
Step 3  nccl-tests (GPU partition)          → AI-load communication acceptance
Step 4  72h mixed-load burn-in              → large training + small simulations concurrent; verify scheduling and fairness

• OSU benchmarks (osu_latency / osu_bw) are mandatory for HPC acceptance: IB point-to-point latency target < 1.5 µs (400G), bandwidth ≥ 370 Gb/s.
• Mixed-load burn-in matters most: verify small jobs don't starve when large jobs can't grab all resources (QOS fairness).

4. Troubleshooting Records

4.1 MPI Jobs Halved Across Nodes

• Symptom: multi-node MPI bandwidth at 40% of single-node.
• Locate: mpirun --mca pml ob1 --mca btl self,sm,ib layer by layer → UCX transport chose shared memory over IB.
• Fix: unified ucx.conf (UCX_TLS=rc,sm,self); re-test recovered. Lesson: 60% of MPI performance problems live in transport selection — check UCX/PMI before hardware.

4.2 Scheduling Fairness Out of Control

• Symptom: after a 512-core job started, many 16-core jobs queued for hours.
• Locate: Slurm QOS had no priority weighting; large jobs occupied indefinitely.
• Fix: per-queue QOS (backfill + priority decay); small jobs backfill large-job gaps. Lesson: HPC acceptance must include "small jobs are not starved."

4.3 Software Version Conflicts

• Symptom: two research groups needed different MPI/compiler combinations, breaking each other's environments.
• Fix: full Lmod modularity; environments load as combinations; conflicts isolated by the module system. Lesson: modularity is not a nice-to-have; it is a requirement for HPC delivery.

5. Checklist (HPC-specific)

• [ ] HPL FP64 passed (≥ 95% of spec)
• [ ] OSU point-to-point latency/bandwidth passed (IB < 1.5 µs / ≥ 370 Gb/s)
• [ ] nccl-tests passed (GPU partition)
• [ ] 72h mixed-load burn-in: large + small jobs concurrent; fairness verified
• [ ] UCX/MPI transport layer unified; no cross-node performance halving
• [ ] Lmod modularity ready; common discipline software prebuilt and documented
• [ ] Queue/QOS policy documented (priority, backfill, limits)
• [ ] Three-layer storage (home/parallel/cache) in place

Further reading: Tsinghua GPU Cluster Delivery Retrospective (the AI-training comparison), NVIDIA Technical Certification Guide (capability validation path).