Unified Resource Management for Heterogeneous Accelerators: Mixed NVIDIA/Ascend/Cambricon Scheduling

Experiment Goal

When NVIDIA, Ascend, and Cambricon accelerators coexist in one cluster, "installing each vendor's driver" is only the start. The real challenge is resource management:

• How to make the scheduler "know" every card type and treat them as uniformly schedulable resources?
• How to avoid hidden performance issues from mixed deployment (jobs landing on incompatible cards)?
• How to keep multi-vendor coexistence observable and auditable (who is using which card, how many, health status)?

This article dissects unified heterogeneous resource management from the Slurm view: heterogeneous resource modeling, GRES extension, mixed deployment strategies, performance isolation, and observability. Commands and configs are generic practice; specific fields follow the installed version and vendor docs.

Step 1: Heterogeneous Resource Modeling — Make the Scheduler Count Correctly

1. Register GRES Per Type — Never Mix All Cards Under One Name

Slurm describes accelerators via GRES (Generic RESource). The first principle for heterogeneous clusters: one GRES type per card class, rather than calling everything gpu.

# /etc/slurm/gres.conf (configure per node by actual card type)
# NVIDIA nodes
NodeName=node[01-20] Name=gpu Type=a100 File=/dev/nvidia[0-7]
# Ascend nodes
NodeName=node[21-30] Name=ascend Type=910B File=/dev/davinci[0-7]
# Cambricon nodes
NodeName=node[31-40] Name=mlu Type=mlu370 File=/dev/mlu[0-7]

# Corresponding slurm.conf declarations
GresTypes=gpu,ascend,mlu
NodeName=node[01-20] Gres=gpu:8
NodeName=node[21-30] Gres=ascend:8
NodeName=node[31-40] Gres=mlu:8
PartitionName=hetero Nodes=node[01-40] Default=YES MaxTime=INFINITE State=UP

Key points:

• Use Type to distinguish card models (a100/910B/mlu370), and refine within a type if needed (e.g. Type=a100 vs Type=h20);
• Make GRES names semantic (gpu/ascend/mlu) and fix them in docs to avoid teams inventing their own;
• sinfo -o "%n %G" quickly verifies each node's registered GRES matches physical card counts.

2. Job Request & Constraint

Users request by GRES type; the scheduler matches nodes by type:

# Request NVIDIA cards
salloc --gres=gpu:4 ...
# Request Ascend cards
salloc --gres=ascend:8 ...
# Mixed request on mixed nodes
salloc --gres=gpu:2,ascend:4 ...

# Prevent heterogeneous jobs landing on wrong cards: constraint or feature labels
sbatch --gres=gpu:4 --constraint=a100 script.sh
# Or isolate via partitions (see strategy in Step 3)

Step 2: Drivers & Runtime — The "Foundation" of Multi-Vendor Coexistence

Mixed clusters trip most on the environment layer. Three disciplines:

1. One independent driver stack per card class: NVIDIA (CUDA driver), Ascend (CANN), Cambricon (CNToolkit) coexist on one OS without interference. Isolate via modules or container images to avoid global PATH/LD_LIBRARY_PATH pollution.
2. Containerization is the best carrier for heterogeneous scheduling: wrap each vendor's runtime into images with Singularity/Apptainer or Enroot; the scheduler handles resources, the image handles runtime. NVIDIA uses --nv; Ascend/Cambricon use their own runtime flags.
3. Unified driver & firmware baseline: linked to Chapter 01 — driver versions and firmware versions consistent fleet-wide per card class, recorded in the delivery baseline. Heterogeneous troubleshooting complexity grows exponentially with vendor count; inconsistent baselines are landmines.

Engineering observation: the #1 failure source in mixed clusters is not the scheduler but environment crosstalk — CUDA and CANN libraries overwriting each other on the same node, PYTHONPATH conflicts, wrong container runtime selection. The cleaner the environment isolation, the more stable the mixed cluster.

Step 3: Mixed Deployment Strategies — Three Typical Layouts

Strategy	Layout	Fit	Trade-off
A. Partition per vendor	One Partition per card class (gpu / ascend / mlu)	Clear business boundaries, stable proportions	Simple & reliable; manual node migration when proportions change
B. Single large partition + constraints	One partition for all nodes; jobs pick cards via --constraint/features	Dynamic card-type demand	Flexible; wrong constraints fail scheduling — needs strong conventions
C. Mixed nodes (multi-vendor per node)	Cards of different vendors in one node	VRAM/compute complementarity, high density	Highest crosstalk risk; only with containerization + strong isolation

Recommendation: most teams start with A (partition isolation), evolving to B (constraint scheduling) after business stabilizes; attempt C only with clear density needs and mature containerization. Strategy choice directly affects troubleshooting cost — in heterogeneous clusters, prefer "less flexible scheduling" over "uncontrollable environment."

Step 4: Performance Isolation & Health Checks

1. Performance Isolation

• Compute/VRAM granularity: Ascend and Cambricon cards also support VRAM/compute partitioning (like MIG/vGPU). Whether to enable partitioning depends on the workload (inference multi-tenancy vs training exclusive);
• Inter-card interconnect: with mixed deployment, ensure same-vendor interconnect domains (NVLink, HCCS, MLU-Link) and separate cross-vendor traffic — mixed deployment ≠ mixed interconnect; different vendors usually lack high-speed links, so training jobs must stay within one interconnect domain;
• Network planes: compute network (RDMA) separated from management; each vendor's RDMA stack (NVIDIA InfiniBand / Ascend RoCE / Cambricon RoCE) configured independently.

2. Health Checks

# Each card class has its own health command; aggregate into one patrol script
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,clocks.gr --format=csv
npu-smi info                       # Ascend
cnmon info                         # Cambricon

• Patrol output in unified format (node, card type, index, util, VRAM, temp, error count), stored to file/DB;
• Alert thresholds set per card type: temperature/utilization semantics differ across vendors; don't use one threshold for all;
• ECC/error counts collected per type, included in the delivery baseline (Chapter 01) and runtime patrol.

Step 5: Observability & Auditability

• Scheduling audit: sacct records each job's requested GRES type/count; produce resource utilization reports by card type;
• Ledger & quotas: with multi-team mixed usage, meter by GRES type separately (NVIDIA and Ascend "quota" can't be mixed); quota and billing unified at the platform layer;
• Resource view: combine sinfo -o "%P %n %G" + patrol data into an ops view of "which card class, which node batch, health status", feeding capacity planning (which cards to expand, which are underutilized).

Mixed-Cluster Checklist

• [ ] GRES registered per card type; sinfo -G matches physical counts
• [ ] Independent driver stack per class; containerized runtimes; no crosstalk
• [ ] Mixed strategy explicit (partition/constraint/mixed nodes) and documented
• [ ] Same-vendor interconnect domains guaranteed; cross-vendor traffic separated
• [ ] Health patrol in unified format; thresholds set per type
• [ ] sacct auditable by GRES type; quotas metered per type

Engineering Takeaways

• The core of heterogeneous management is modeling: model each card in the scheduler first (type/count/features); scheduling, isolation, and audit then have a foundation.
• Environment isolation > scheduling tricks: the stability ceiling of a mixed cluster is set by environment isolation; scheduling strategy only optimizes on top.
• Partition first, then converge: heterogeneous coexistence is evolved, not one-shot; start with partition isolation, then relax constraint scheduling after proven maturity.

Next article preview: with heterogeneous compute ready, attention turns to "what exactly happens between one line of Python and machine code" — the GPU kernel compilation pipeline. The next article, CUDA Compilation Pipeline Deep Dive: The Optimization Space from nvcc to SASS, dissects kernel build artifacts and optimization judgment.