QA Highlight: How to Configure Slurm Heterogeneous Scheduling on a CPU/GPU Hybrid Cluster?

This post is distilled from a real Q&A session with the AI-HPC Assistant (chat.ai-hpc.org) — the answer is grounded in the 14-chapter engineering knowledge base (mainly Chapters 07-heterogeneous and 06-cloud), then curated and extended by hand.

Question

How to configure Slurm heterogeneous scheduling? How to manage GPU resources on a CPU/GPU hybrid cluster?

Core Principle

Slurm is heterogeneity-friendly by design: accelerators such as GPUs are integrated into a unified resource view through GRES (Generic RESources), combined with node definitions in slurm.conf, partition layout, and --gres submission constraints — enabling fine-grained heterogeneous scheduling on CPU/GPU hybrid clusters (KB · Chapter 07-heterogeneous).

I. Declaring GPU Resources: GRES Core Configuration

Declare GPUs per node in slurm.conf:

# Node: 4× A100, GRES named gpu, type a100
NodeName=gpu01 Gres=gpu:a100:4 CPUs=64 RealMemory=500000 Sockets=2 CoresPerSocket=16 ThreadsPerCore=2

# Enable GRES plugin
GresTypes=gpu
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory

• select/cons_tres: selects by consumable trackable resources (core/memory/gres) — the key to on-demand GPU allocation.
• Submit with --gres=gpu:2, or --gres=gpu:a100:2 to pin a type.

II. MIG / vGPU and Multi-Tenant Sharing

When GPUs are scarce and tenants must share (KB · Chapter 07-heterogeneous):

• MIG (Multi-Instance GPU): A100/H100 can be partitioned into instances (e.g., 7× 1g.5gb); declare with Gres=gpu:mig:... and schedule at instance granularity — well suited to mixed inference + light-training loads.
• vGPU: virtualized environments allocate compute/memory slices via NVIDIA vGPU.
• Multi-tenancy: use Partition + QOS (e.g., QOS=interactive, QOS=debug) for quotas and preemption so a single tenant cannot saturate the cluster.

III. Hybrid-Cluster Partition Strategy (Chapter 06-cloud)

• Separate by purpose: cpu (CPU-only jobs), gpu (training/inference), gpu-highprio (SLA-sensitive).
• Control priority/preemption across partitions with PriorityTier and QOS.
• On mixed nodes, use Feature (e.g., Feature=ib,avx512) to constrain scheduling targets and avoid jobs landing on the wrong node type.

IV. Dockerized Training Submission Template (Chapter 06-cloud)

#!/bin/bash
#SBATCH -p gpu
#SBATCH --gres=gpu:8
#SBATCH -N 2
#SBATCH --ntasks-per-node=8

srun --container-image=registry.example.com/train:latest \
     --container-mounts=/data:/data \
     python train.py --config /data/llm/large-scale.yaml

• --container-image with Slurm container plugins (e.g., Pyxis/Enroot) lets heterogeneous clusters share a single image artifact.
• For large-model training, combine NCCL env vars (NCCL_SOCKET_IFNAME, NCCL_IB_HCA) with InfiniBand networking (Chapter 03).

V. Verification and Tuning

• sinfo -o "%n %G %f" shows node GRES and features; scontrol show node gpu01 verifies GPU visibility.
• Smoke-test GPU passthrough with srun --gres=gpu:1 nvidia-smi.
• Heterogeneous debugging checklist: GRES not declared / select/cons_tres not enabled / partition & QOS preemption misconfigured.

Note: The above is a general configuration framework; exact slurm.conf parameters must be benchmarked against your cluster topology, scheduling policy and workload. The knowledge base contains finer config snippets and troubleshooting cases — see Chapter 07-heterogeneous and Chapter 06-cloud.

---

Follow-ups (try the assistant): How to implement GPU preemption in Slurm? How to plan MIG instances? How to configure the Pyxis container plugin? → Chat with the AI-HPC Assistant