openEuler 24.03 Kernel & System Tuning: BIOS, Kernel Params and IRQ Affinity

"System tuning" is the easiest thing to turn into mysticism in cluster delivery: copy a sysctl block and call it done. This article breaks the low-level optimization of a 384-GPU L40S IB cluster into three layers — BIOS hardware state, kernel parameters, IRQ affinity — and explains the why, the what and the how-to-verify for each.

1. BIOS: Pin Down the Hardware State First

BIOS decides the hardware's "factory state"; the OS layer cannot fix it afterwards. Unified checklist (verify per node):

Setting	Value	Purpose
Work mode	Performance	Disable power saving, favor performance
CPU C-states	C0/C1 only (disable C6+)	Reduce RDMA latency jitter
Turbo Boost	Enabled	8468V boost 3.8GHz
Hyper-Threading	Enabled	Can disable later for A/B comparison
SR-IOV / VT-d	Enabled	Virtualization + IOMMU
PCIe link speed	Gen4 Max	Full-speed links
NUMA	Enabled	Prerequisite for GPU/NIC NUMA affinity
Boot mode	UEFI	openEuler 24.03 UEFI install

Verification (at power-on self-test, per node):

dmidecode -t memory | grep -E "Size|Speed"   # 24×32GB @ 4800
lspci -vvv | grep -A8 -i l40s                # LnkSta: 16GT/s, x16
lspci | grep -i mellanox                     # expect 7 NIC functions

Judgment: disabling C6 is mandatory for an RDMA cluster — when the CPU enters deep sleep, network interrupt latency jumps from microseconds to milliseconds; NCCL is latency-sensitive, showing up directly as throughput jitter.

2. Kernel Parameters: Clear the Way for RDMA and Collectives

Parameter	Value	Effect
transparent_hugepage	madvise	Huge pages on demand, avoid IO-path fragmentation
vm.swappiness	10	Less swapping
kernel.numa_balancing	0 (train nodes)	Disable auto migration, avoid NUMA thrash
IOMMU	On demand (if SR-IOV used)	Note DMA mapping overhead in GDR scenarios

GDR tradeoff: GPUDirect RDMA maps GPU memory directly into NIC DMA; enabling IOMMU adds mapping overhead. Measure first, then decide — run ib_write_bw with IOMMU on and off, let the data decide, not intuition.

3. IRQ Affinity: Pin NIC Interrupts to the Right CPUs

• NIC MSI-X interrupts can scatter across any CPU by default; handling interrupts cross-NUMA adds latency;
• Generate the affinity script from the measured nvidia-smi topo -m topology: wherever the GPU lives on NUMA, bind that NIC's interrupts to the same NUMA's CPUs;
• Key point: generate from measured topology, do not copy a template — vendor layout variance means "identical-looking" machines can have different NUMA ownership.

# Approach (generated from measured topology, not a template)
# Read /proc/irq/*/smp_affinity_list, bind mlx5 interrupts to GPU-side CPU cores
# Verify: interrupts concentrated on expected CPUs (watch with mpstat -I CPU)

4. Tuning Order: Hardware First, Kernel Second, Application Last

Wrong example: tune NCCL parameters chasing performance, discover jitter, then go back to disable C-states — variables are mixed, conclusions invalid.

Correct order:

1. BIOS pins the hardware state (verify first, then tune);
2. Kernel parameters and IRQ affinity (rolled out once, consistent across nodes);
3. Run baselines (STREAM/NCCL/IB bandwidth), archive them;
4. Only then touch application-layer params (NCCL, MPI), one variable at a time.

5. Delivery Checklist

1. BIOS: 8 settings verified per node, dmidecode/lspci spot checks;
2. Kernel params: sysctl/kernel cmdline consistent across nodes (Ansible rollout);
3. IRQ affinity: script generated from measured topology, interrupts on GPU-side CPUs;
4. GDR decision: IOMMU on/off measured and archived;
5. Baselines archived: STREAM/NCCL/IB, as reference for future changes.

---

Based on field measurements from a 384-GPU L40S IB cluster delivery; exact versions and commands follow on-site validation.