Tsinghua GPU Cluster Delivery Retrospective: From Architecture to Acceptance

Abstract: A large GPU cluster delivery is essentially translating "business requirements" into "architecture decisions," then realizing those decisions into an "acceptance-ready system." This article reviews the complete path of leading the Tsinghua GPU cluster delivery — methodology at every stage from requirements, architecture, and acceptance to troubleshooting. Details are anonymized; the method is reusable.

1. Requirement Decomposition: Ask "Who Uses It, What Runs on It" First

The worst case in delivery is "the customer said 100 GPUs, but it's 100 GPUs running 3 completely different workloads." Four things must be locked during requirements:

Dimension	Questions that must be asked	Impact on decisions
Workload type	Training/inference/scientific computing? Ratio?	Compute vs bandwidth vs FP64 mix
Concurrency	How many jobs at once? Max GPUs per job?	Scheduling strategy, queue partitioning
Data scale	Dataset/checkpoint size? Read/write pattern?	Storage capacity and bandwidth
User level	Researchers/engineers/students?	Software stack complexity, docs & training

This case's conclusion: LLM training dominant, multi-user concurrent, scientific computing secondary → compute-first, high-end interconnect, storage designed for training loads.

2. Architecture Decisions: Three Pillars + Storage + Scheduling

2.1 Compute and Interconnect

• Node spec: 8-GPU nodes (NVLink full mesh), TP=8 within a node as the base — cross-node only for PP/DP, keeping the communication ratio controlled.
• Cross-node network: IB high-speed network + Fat-Tree topology for even AllReduce hashing; RoCE requires strict PFC/ECN (see NCCL Tuning).
• Basis: reverse-engineer from the target model (e.g. 70B) + parallelism strategy (see Distributed Parallelism Overview), never guess GPU counts.

2.2 Storage

• Parallel filesystem (Lustre/GPFS-class) for datasets and checkpoints; aggregate bandwidth reverse-engineered from "minute-level checkpoint writes" (see Acceptance Checklist).
• Home/shared directories separated from high-speed storage — small IO must not saturate big bandwidth.

2.3 Scheduling and Software Stack

• Slurm heterogeneous scheduling: GPU + CPU nodes managed together, gres for GPU resources, queues by job type (training/test/interactive).
• Container runtime (Singularity/Apptainer or Enroot) + image registry for reproducible environments — reproducibility is the baseline of delivery quality.
• Monitoring: GPU utilization, temperature, ECC, IB traffic unified — monitoring is handed over at delivery; no operational vacuum.

3. Four-Step Acceptance: No Sign-Off Without Meeting Criteria

Following the Acceptance Checklist strictly:

Step 1  HPL single-node compute     → measured ≥ 95% of spec
Step 2  Stream memory bandwidth    → measured ≥ 90% of spec
Step 3  nccl-tests cross-node      → 256-GPU busbw ≥ 40 GB/s (400G IB)
Step 4  72h real-load burn-in      → mixed training+inference+loading, no fatal errors

This case's key data: single-node HPL all passed (no throttling); cross-node nccl-tests initially 38 GB/s (below line), traced to ECMP hash imbalance; after enabling enhanced hash, 45 GB/s passed.

4. Troubleshooting Records: Three Representative Issues

4.1 ECMP Hash Imbalance (interconnect layer)

• Symptom: 256-GPU nccl-tests variance high; one link saturated.
• Locate: 2-node pair tests normal → full-cluster abnormal → switch telemetry for traffic distribution.
• Fix: enabled ECMP enhanced hash (L4 port hashing); re-test normalized. Lesson: large-scale baselines must be accepted at the topology layer — single-point normality does not imply cluster normality.

4.2 Data-Loading IO Bottleneck (storage layer)

• Symptom: training GPU utilization 60%; data-loading wait prominent.
• Locate: nvidia-smi idle GPUs + storage-side IOPS saturated.
• Fix: data prefetch/cache layer + more parallel-filesystem clients; utilization back to 90%+. Lesson: compute acceptance passing ≠ training runs fast — the data pipeline is a hidden bottleneck.

4.3 Driver/Container Compatibility (software layer)

• Symptom: some user containers fail to start with CUDA version errors.
• Locate: driver-supported version vs in-image CUDA runtime mismatch.
• Fix: documented "driver/CUDA/image compatibility matrix" + centrally maintained base images. Lesson: software-stack standardization is the core asset of post-delivery operations (see Driver Lifecycle).

5. Delivery Checklist (Reusable)

• [ ] Four requirement questions (workload/concurrency/data/users) locked
• [ ] Compute/interconnect/storage reverse-engineered from target load, with evidence
• [ ] Four-step acceptance all passed (HPL/Stream/nccl-tests/72h)
• [ ] Topology-layer baseline passed (256-GPU cross-node), not single-point only
• [ ] Data pipeline verified: training GPU utilization ≥ 85%
• [ ] Software stack standardized: image/driver/CUDA compatibility matrix documented
• [ ] Monitoring and alerting handed over; operations runbook ready
• [ ] Backup and checkpoint strategy in place (recovery drill passed)

Further reading: PKU HPC Project Delivery Retrospective (the scientific-computing complement), NVIDIA Technical Certification Guide (capability validation path).