AI-HPC Knowledge Overview

This documentation aims to build a vertically integrated knowledge graph, connecting the underlying High Performance Computing (HPC) infrastructure with upper-layer Artificial Intelligence (AI) applications. The whole library is organized into six levels (L0 → L5) with 15 numbered chapters and 150+ bilingual engineering articles.

🧭 Learning Paths (pick your starting level)

BeginnerStart with the math, build full-stack awarenessL0 Quantitative Methods (01) → L1 Hardware & Infrastructure (02→05)Settle the memory/compute/bandwidth ledgers first, then understand every building block of the compute base: GPUs/chips, cluster topology, IB networking, parallel storage.IntermediateMaster system software, compilers & frameworksL2 System Software & Scheduling (06→07) → L3 Compilers, Frameworks & Drivers (08→10)Independently deliver clusters: automated provisioning, Slurm/K8s scheduling, drivers and heterogeneous adaptation, then compilers and deep learning frameworks.ExpertTackle model engineering & AI×ScienceL4 Model Engineering (11→13) → L5 Applications & Science (14→15)Pretraining, distributed training, inference optimization, industry adoption and AI-for-Science frontier exploration.

🧮 L0 · Quantitative Methods (Calculate First)

Goal: Settle the accounts with order-of-magnitude estimates — what is moved, how much, how many times, through where, and who must wait for it.

• 01. AI Infra Quantitative Analysis: Inference resource accounting (memory/compute/bandwidth), model architecture to resources, inference optimization boundaries, training resource accounting (memory/communication/recompute).

🏗️ L1 · Hardware & Infrastructure

Goal: Build a computing base with high bandwidth, low latency, and massive parallelism.

• 02. Hardware & Chips: GPU (H100/A100), NPU, Heterogeneous Architectures, Liquid Cooling & Firmware.
• 03. Cluster Architecture: 10K-GPU network architecture, IB/RoCE, MPI in practice, 384×L40S delivery playbook.
• 04. Performance Benchmarking: HPL/STREAM/NCCL-test baselines, CPU/GPU measurement methodology.
• 05. Parallel Storage: Lustre, BeeGFS, GPFS, High Concurrency IO.

🖥️ L2 · System Software & Scheduling

Goal: Improve resource utilization and abstract underlying hardware differences.

• 06. Operating System: PXE, Ubuntu Autoinstall, Diskless Boot, Cluster Delivery Automation.
• 07. Cloud & Scheduling: Kubernetes, Slurm, Containerization.

🔧 L3 · Compilers, Frameworks & Drivers

Goal: From model graphs to high-performance kernels, take control of framework runtimes and heterogeneous drivers.

• 08. AI Compiler: OpenAI Triton, MLIR, PyTorch Compile Stack.
• 09. Deep Learning Frameworks: PyTorch 2.x Autograd/Dispatch, DDP/FSDP, Triton Kernels.
• 10. Heterogeneous Computing: NVIDIA Driver, CUDA Toolkit, MIG, Domestic Accelerator Porting.

🧠 L4 · Model Engineering

Goal: Efficient training and inference on massive clusters.

• 11. Pre-trained Models: Transformer, MoE/MLA, Pre-training Data Pipeline & Engineering.
• 12. Distributed Training: 3D Parallelism, LLM Lab Series, Mixed Precision, NCCL Tuning.
• 13. Inference Engines: vLLM (PagedAttention), TensorRT-LLM, Quantization, Long Context.

🚀 L5 · Applications & Science

Goal: Empower industries and explore new scientific paradigms.

• 14. Industry Applications: RAG, Agents, Private Deployment, Scientific Surrogate Models.
• 15. AI×Science:
  • Life Science
  • Weather Forecasting
  • Fluid Dynamics (CFD)
  • Materials Science
  • AI4S Clusters & Training Engineering

🗺️ Architecture Overview

For the full AI-HPC full-stack architecture diagram, see Core Architecture. All layers are presented as L0 → L5 groups in the sidebar; read through the layers in order.