PXE Troubleshooting Handbook for Large-Scale Provisioning

Experiment Goal

PXE provisioning (PXE boot → kernel boot → install source → disk install → first-boot config) almost never fails on a single machine, but breaks in weird ways at scale: TFTP timeouts when 50 nodes PXE-boot simultaneously, some nodes stuck before the installer, network not coming up after install… These failures share a trait: "intermittent on each node, hard to reproduce". Without a systematic method, they burn huge delivery time.

This article walks through six classic PXE-at-scale failures in "symptom → diagnosis → root cause → fix" order. Commands are reproducible operations; symptoms and fixes come from cluster delivery experience without fabricating environment numbers.

Failure 1: DHCP/TFTP Timeout or Flapping During PXE Boot

Symptom: A few nodes stuck at the DHCP stage after PXE boot, or TFTP retries timing out while downloading boot files.

Diagnosis:

# Server side: see if DHCP and TFTP requests arrive
tail -f /var/log/messages | grep -E "dhcpd|tftp"
# Client side: confirm which DHCP server responded and the next-server option

Root causes & fixes (by probability):

1. DHCP request storm: simultaneous boots overwhelm the DHCP server's lease pool or capacity. Fix: confirm pool size ≥ batch node count; provision in batches (e.g. 20–30 per batch) instead of all nodes at once.
2. TFTP throughput bottleneck: TFTP default blksize is 512B; concurrent pulls from dozens of nodes queue badly. Fix: enable blksize/tsize extension on the server (e.g. 1456 or larger), preload boot files into RAM disk, or switch to HTTP boot (iPXE + HTTP) to reduce concurrency pressure.
3. Multiple DHCP conflicts: another DHCP in the management network responds faster and gives wrong config. Fix: confirm DHCP uniqueness in the management network; verify the boot server via DHCP options and IP checks.
4. Switch DHCP Snooping / broadcast limits: some access switches restrict DHCP broadcasts by default. Fix: check DHCP Snooping config and port policies.

Engineering observation: first law of large-scale PXE — prove the full PXE chain on one node and archive it, then open batches. Batch size is not a guess; it's decided by measured DHCP/TFTP server concurrency capacity.

Failure 2: Boot Kernel Starts, but Install Source Not Found

Symptom: PXE boots into the install kernel but cannot find the install medium/source (NFS/HTTP mount fails).

Diagnosis:

# Check inst.repo / ks parameters in boot args
showmount -e <nfs-server>
curl -I http://<http-server>/<repo-path>/

Root causes & fixes:

1. Install source vs boot kernel version mismatch: kernel newer/older than the repo breaks metadata parsing. Fix: use the same minor release for source and boot media.
2. NFS mount parameter issues: the boot kernel's network stack has limited capabilities; confirm ip=dhcp is set, NFS export is correct (fsid=0 or correct path), and firewall allows it.
3. ks file unreachable: wrong path or export for the kickstart file. Fix: place ks on the same reachable path as boot files; verify with curl first.

Failure 3: Disks Not Fully Recognized or Wrong Disk Installed

Symptom: Only some disks visible in the installer, or the OS installs to the wrong disk.

Diagnosis:

lsblk -o NAME,SIZE,MODEL,SERIAL
lspci | grep -iE "nvme|sata|raid"

Root causes & fixes:

1. HBA/RAID mode not aligned: the delivery baseline (Chapter 01 BIOS config) did not set the HBA/RAID card mode (IT passthrough vs IR RAID) per plan, so the controller changes disk count/capacity. Fix: align with the Chapter 01 baseline — all IT passthrough or all RAID.
2. NVMe not recognized under UEFI: old UEFI firmware without NVMe driver. Fix: upgrade board firmware (Chapter 01 flow) or inject the NVMe driver during install.
3. Wrong target disk in multi-disk install: ks ignoredisk/clearpart device names differ between UEFI and BIOS naming rules. Fix: use unique identifiers (disk/by-id or WWN) instead of unstable names like /dev/sda.

Engineering observation: "installed to the wrong disk" is the most expensive rework (data risk + reinstall). The delivery plan must define per-model target disk rules (matched by serial/model) and hard-code them in ks — never rely on device-name ordering.

Failure 4: UEFI / Secure Boot Related Install Failures

Symptom: UEFI install succeeds, but reboot fails to boot; or after enabling Secure Boot, the system/driver fails.

Diagnosis:

efibootmgr -v          # UEFI boot entries
mokutil --sb-state     # Secure Boot state
dmesg | grep -i "secure boot"

Root causes & fixes:

1. Boot entry write failure/order: after install, BIOS boot order still prefers PXE. Fix: verify efibootmgr after install, adjust BootOrder if needed; the plan should make PXE first only during install, then disable or demote it.
2. Secure Boot vs driver signatures: third-party drivers (GPU, NIC, RAID) unsigned or MOK not imported → "Invalid signature" at boot. Fix: align with Chapter 01 baseline — either disable Secure Boot uniformly (and record the risk), or import MOK after install. Never mix per-node.
3. UEFI/CSM mixed modes: some nodes Legacy, some UEFI, causing inconsistent ks/boot-arg behavior. Fix: unify boot mode fleet-wide and confirm per node in the checklist.

Failure 5: Driver Injection Failure (GPU/NIC Not Recognized After Install)

Symptom: system installed; lspci shows the device but no driver, or driver fails to install.

Diagnosis:

lspci | grep -i nvidia
lsmod | grep nvidia
modinfo nvidia | head -5
dmesg | grep -iE "nvidia|firmware" | tail -20

Root causes & fixes:

1. Driver/kernel mismatch: install-source kernel newer, driver RPM built for older kernel. Fix: install drivers from the matching repo after install or use dkms (auto-rebuild on kernel upgrade).
2. Firmware/BIOS layer not ready: linked to Chapter 01 — Above 4G off, Secure Boot misaligned, PCIe link degraded all manifest as "driver won't install". Fix: pass Chapter 01 baseline before installing drivers.
3. Injection flow missing: inst.dd or driver RPM injection position wrong in the PXE plan. Fix: verify boot-arg and injection source path; prove the injection chain on one node first.

Failure 6: Network Not Up After Install

Symptom: system installed; IP not up, or up but management/business network unreachable.

Diagnosis:

ip addr show
ip route
# Check whether NIC names changed vs pre-install (Predictable Names policy)
# Check ks network config (device match rule)

Root causes & fixes:

1. NIC name drift: under UEFI, names (eno/ens/enp) change with slot/firmware; ks hard-coded device=eth0 fails. Fix: bind by MAC or ifname= in ks; or unify with net.ifnames=0 (fleet-wide consistency required).
2. Management/business network conflict: dual-NIC config order in ks wrong; default route points to the wrong NIC. Fix: declare BOOTPROTO and default gateway ownership explicitly in ks; verify ip route after install.
3. Switch port/VLAN not released: new node port lacks the correct VLAN. Fix: check switch port config before install; check management connectivity from BMC side to rule out the link layer first.

Decision Tree (Quick Reference)

PXE install failure
├─ stuck at DHCP/TFTP → Failure 1 (storm/throughput/multi-DHCP/switch)
├─ kernel up, source missing → Failure 2 (version/mount/ks path)
├─ disk anomalies → Failure 3 (RAID mode/driver/naming rules)
├─ reboot won't boot → Failure 4 (boot entries/Secure Boot/mixed modes)
├─ system up, no driver → Failure 5 (kernel match/firmware baseline/injection)
└─ system up, network down → Failure 6 (name drift/routing/VLAN)

Engineering Takeaways

• Troubleshoot by layer: PXE failures break down into "DHCP/TFTP → boot kernel → install source → disk → reboot → driver/network"; each layer has independent verification commands. Don't randomly try across layers.
• Archive one node, then batch: prove and archive the PXE chain, ks file, and driver injection on one node; then provision in batches sized by measured concurrency.
• Align with the baseline: a large share of provisioning failures root in the firmware/BIOS baseline (Chapter 01) being misaligned — provisioning is one link of the delivery pipeline, not an island.

Next article preview: with provisioning and baseline ready, the cluster enters the runtime phase. Beyond Slurm, more and more GPU workloads run on Kubernetes — the next article, Kubernetes Inference Service in Production: vLLM from Deployment to High Availability, focuses on the K8s orchestration side.