Unified Resource Management for Heterogeneous Accelerators: Mixed NVIDIA/Ascend and More Scheduling

Experiment Goal

When NVIDIA, Ascend and other domestic accelerators coexist in one cluster, "installing each vendor's driver" is only the start. The real challenge is resource management:

• How to make the scheduler "know" every card type and treat them as uniformly schedulable resources?
• How to avoid hidden performance issues from mixed deployment (jobs landing on incompatible cards)?
• How to keep multi-vendor coexistence observable and auditable (who is using which card, how many, health status)?

This article dissects unified heterogeneous resource management from the Slurm view: heterogeneous resource modeling, GRES extension, mixed deployment strategies, performance isolation, and observability. Commands and configs are generic practice; specific fields follow the installed version and vendor docs.

Step 1: Heterogeneous Resource Modeling — Make the Scheduler Count Correctly

1. Register GRES Per Type — Never Mix All Cards Under One Name

Slurm describes accelerators via GRES (Generic RESource). The first principle for heterogeneous clusters: one GRES type per card class, rather than calling everything gpu.

# /etc/slurm/gres.conf (configure per node by actual card type)
# NVIDIA nodes
NodeName=node[01-20] Name=gpu Type=a100 File=/dev/nvidia[0-7]
# Ascend nodes
NodeName=node[21-30] Name=ascend Type=910B File=/dev/davinci[0-7]

# Corresponding slurm.conf declarations
GresTypes=gpu,ascend
NodeName=node[01-20] Gres=gpu:8
NodeName=node[21-30] Gres=ascend:8
PartitionName=hetero Nodes=node[01-40] Default=YES MaxTime=INFINITE State=UP

Key points:

• Use Type to distinguish card models (a100/910B), and refine within a type if needed (e.g. Type=a100 vs Type=h20);
• Make GRES names semantic (gpu/ascend) and fix them in docs to avoid teams inventing their own;
• sinfo -o "%n %G" quickly verifies each node's registered GRES matches physical card counts.

2. Job Request & Constraint

Users request by GRES type; the scheduler matches nodes by type:

# Request NVIDIA cards
salloc --gres=gpu:4 ...
# Request Ascend cards
salloc --gres=ascend:8 ...
# Mixed request on mixed nodes
salloc --gres=gpu:2,ascend:4 ...

# Prevent heterogeneous jobs landing on wrong cards: constraint or feature labels
sbatch --gres=gpu:4 --constraint=a100 script.sh
# Or isolate via partitions (see strategy in Step 3)

Step 2: Drivers & Runtime — The "Foundation" of Multi-Vendor Coexistence

Mixed clusters trip most on the environment layer. Three disciplines:

1. One independent driver stack per card class: NVIDIA (CUDA driver), Ascend (CANN), other vendors (their own toolkits) coexist on one OS without interference. Isolate via modules or container images to avoid global PATH/LD_LIBRARY_PATH pollution.
2. Containerization is the best carrier for heterogeneous scheduling: wrap each vendor's runtime into images with Singularity/Apptainer or Enroot; the scheduler handles resources, the image handles runtime. NVIDIA uses --nv; other domestic accelerators use their own runtime flags.
3. Unified driver & firmware baseline: linked to Chapter 01 — driver versions and firmware versions consistent fleet-wide per card class, recorded in the delivery baseline. Heterogeneous troubleshooting complexity grows exponentially with vendor count; inconsistent baselines are landmines.

Engineering observation: the #1 failure source in mixed clusters is not the scheduler but environment crosstalk — CUDA and CANN libraries overwriting each other on the same node, PYTHONPATH conflicts, wrong container runtime selection. The cleaner the environment isolation, the more stable the mixed cluster.

Step 3: Mixed Deployment Strategies — Three Typical Layouts

Strategy	Layout	Fit	Trade-off
A. Partition per vendor	One Partition per card class (gpu / ascend / npu)	Clear business boundaries, stable proportions	Simple & reliable; manual node migration when proportions change
B. Single large partition + constraints	One partition for all nodes; jobs pick cards via --constraint/features	Dynamic card-type demand	Flexible; wrong constraints fail scheduling — needs strong conventions
C. Mixed nodes (multi-vendor per node)	Cards of different vendors in one node	VRAM/compute complementarity, high density	Highest crosstalk risk; only with containerization + strong isolation

Recommendation: most teams start with A (partition isolation), evolving to B (constraint scheduling) after business stabilizes; attempt C only with clear density needs and mature containerization. Strategy choice directly affects troubleshooting cost — in heterogeneous clusters, prefer "less flexible scheduling" over "uncontrollable environment."

Step 4: Performance Isolation & Health Checks

1. Performance Isolation

• Compute/VRAM granularity: Ascend and other domestic cards also support VRAM/compute partitioning (like MIG/vGPU). Whether to enable partitioning depends on the workload (inference multi-tenancy vs training exclusive);
• Inter-card interconnect: with mixed deployment, ensure same-vendor interconnect domains (NVLink, HCCS, etc.) and separate cross-vendor traffic — mixed deployment ≠ mixed interconnect; different vendors usually lack high-speed links, so training jobs must stay within one interconnect domain;
• Network planes: compute network (RDMA) separated from management; each vendor's RDMA stack (NVIDIA InfiniBand / Ascend RoCE and others) configured independently.

2. Health Checks

# Each card class has its own health command; aggregate into one patrol script
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,clocks.gr --format=csv
npu-smi info                       # Ascend

• Patrol output in unified format (node, card type, index, util, VRAM, temp, error count), stored to file/DB;
• Alert thresholds set per card type: temperature/utilization semantics differ across vendors; don't use one threshold for all;
• ECC/error counts collected per type, included in the delivery baseline (Chapter 01) and runtime patrol.

Step 5: Observability & Auditability

• Scheduling audit: sacct records each job's requested GRES type/count; produce resource utilization reports by card type;
• Ledger & quotas: with multi-team mixed usage, meter by GRES type separately (NVIDIA and Ascend "quota" can't be mixed); quota and billing unified at the platform layer;
• Resource view: combine sinfo -o "%P %n %G" + patrol data into an ops view of "which card class, which node batch, health status", feeding capacity planning (which cards to expand, which are underutilized).

Mixed-Cluster Checklist

• [ ] GRES registered per card type; sinfo -G matches physical counts
• [ ] Independent driver stack per class; containerized runtimes; no crosstalk
• [ ] Mixed strategy explicit (partition/constraint/mixed nodes) and documented
• [ ] Same-vendor interconnect domains guaranteed; cross-vendor traffic separated
• [ ] Health patrol in unified format; thresholds set per type
• [ ] sacct auditable by GRES type; quotas metered per type

Engineering Takeaways

• The core of heterogeneous management is modeling: model each card in the scheduler first (type/count/features); scheduling, isolation, and audit then have a foundation.
• Environment isolation > scheduling tricks: the stability ceiling of a mixed cluster is set by environment isolation; scheduling strategy only optimizes on top.
• Partition first, then converge: heterogeneous coexistence is evolved, not one-shot; start with partition isolation, then relax constraint scheduling after proven maturity.

Next article preview: with heterogeneous compute ready, attention turns to "what exactly happens between one line of Python and machine code" — the GPU kernel compilation pipeline. The next article, CUDA Compilation Pipeline Deep Dive: The Optimization Space from nvcc to SASS, dissects kernel build artifacts and optimization judgment.