10. Heterogeneous Computing: Multi-Chip Scheduling

Chapter focus: In the era of multi-chip, multi-stack coexistence, the difficulty of heterogeneous computing lies less in "installing" than in "scheduling smoothly". This chapter covers four high-frequency engineering problems: driver/CUDA environments, GPU partitioning (MIG/MPS), CUDA-ecosystem migration for domestic accelerators, and Slurm heterogeneous scheduling.

🧭 Reading Map

• Prerequisites: 09. Deep Learning Frameworks — above framework abstractions, heterogeneous hardware differences
• Role: closing chapter of Part 2, L2 — CUDA toolchain, MIG partitioning, multi-vendor mixing, domestic accelerator porting
• Order: CUDA toolchain (install/MPS) → resource partitioning (MIG) → heterogeneous scheduling → domestic porting
• Next: 11. Pre-trained Models — heterogeneous base ready, build large models

Articles

GPU Driver & CUDA Installation Guide

A step-by-step guide for installing NVIDIA GPU Drivers and CUDA Toolkit on Linux (CentOS/Ubuntu/SUSE), covering dependencies, persistence mode, and Fabric Manager.

Read article →

CUDA MPS in Practice: Multi-Process GPU Sharing for Small-Task Concurrency

Reproducible commands dissecting CUDA MPS: positioning vs MIG, console/daemon deployment, small-task concurrency experiments, timeout and fault-isolation pitfalls, with engineering takeaways.

Read article →

Domestic GPU/NPU Porting: Migrating from CUDA to Heterogeneous Accelerators

Reproducible flow dissecting domestic accelerator porting: the CUDA migration roadmap, Ascend and other domestic accelerator adaptation points, operator mapping and performance regression, dual-stack coexistence, with engineering takeaways.

Read article →

GPU Resource Partitioning in Practice: MIG, vGPU and Multi-Tenant Sharing

In multi-tenant clusters an A100 often sits below 50% utilization: MIG (hardware partitioning), vGPU (virtualization) and soft memory/time-slice partitioning — principles, selection, Slurm integration with gres, and field pitfalls like isolation boundaries, NCCL communication and failure domains.

Read article →

Slurm Heterogeneous Scheduling: Resource Management for CPU/GPU Mixed Clusters

Slurm heterogeneous node management, gres resource declaration, partition and QOS design, job submission templates, and scheduling strategy with common pitfalls for GPU/CPU mixed clusters.

Read article →

Unified Resource Management for Heterogeneous Accelerators

GRES modeling, mixed strategies, isolation, observability.

Read article →

## Suggested Reading Path

Start with environment setup and GPU partitioning (MIG/MPS); scheduling readers go to Slurm heterogeneous scheduling; migration readers go to domestic-accelerator porting.

Key Takeaways

The core of heterogeneous scheduling is "letting each chip run what it does best": raise small-task density with MIG/MPS, isolate architectures with Slurm partitions, and lower multi-stack maintenance with unified toolchains.