Cluster Delivery Automation: From PXE Provisioning + Ansible Config to Slurm-Ready

Summary: PXE gives you "an operating system", not "a usable cluster": NIC IPs, hostnames, shared storage, drivers, Slurm—configuring 100 nodes one by one is a nightmare. This post standardizes delivery: PXE install → node acceptance → Ansible batch config → Slurm readiness checks, all reproducible with checklists.

0. Experiment Goals

• Fill the gap after PXE install (Diskless Boot): from "OS installed" to "cluster usable"
• Use Ansible for batch config: hostname/network, shared storage mount, GPU drivers and firmware, Slurm config distribution
• Build a node acceptance checklist: hardware health, topology, drivers, performance baseline
• Deliver a "new node ready in 30 minutes" standard operating procedure (SOP)

1. Delivery Flow Overview

Bare metal → PXE install (OS ready)
           → node acceptance (hardware/network/driver health)
           → Ansible batch config (hostname, storage, drivers, Slurm, monitoring)
           → Slurm readiness checks (slurmctld/slurmd health, node registration, job acceptance)
           → join the production cluster

2. Step 1: Node Acceptance Script (Run Immediately After Install)

# Run on every node after install; output an acceptance report
cat > /root/node_acceptance.sh <<'EOF'
#!/bin/bash
echo "== CPU =="; lscpu | grep -E "Model name|^CPU\(s\)"
echo "== MEMORY =="; free -g | head -2
echo "== GPU =="; nvidia-smi -L 2>/dev/null || echo "NO GPU DRIVER"
echo "== GPU TOPO =="; nvidia-smi topo -m 2>/dev/null | head -9
echo "== PCIe LINK =="; for b in $(nvidia-smi --query-gpu=pci.bus_id --format=csv,noheader); do
  lspci -vvv -s $b | grep -i LnkSta | head -1; done
echo "== IB NIC =="; ibstat 2>/dev/null | grep -E "State|Rate|Port" || echo "NO IB"
echo "== DISKS =="; lsblk -d -o NAME,SIZE,MODEL | head -8
EOF
bash /root/node_acceptance.sh | tee /var/log/node-acceptance-$(hostname).log

Step-by-step:

• First thing after install is acceptance, not config—surface hardware faults now, don't spend hours configuring a broken node.
• Five mandatory checks: CPU/memory, GPU driver & count, GPU topology (NVLink full-mesh?) , PCIe link (x16 vs x8) , IB NIC state. Each directly determines later performance (see GPU Server Topology).
• Archive acceptance reports (/var/log/) for later comparison with performance baselines.

3. Step 2: Ansible Batch Config

# ansible/hosts.ini — group by role
[login]
login01 ansible_host=10.0.0.11
[compute]
node01 ansible_host=10.0.0.21
node02 ansible_host=10.0.0.22
[slurm_controller]
login01

# ansible/playbooks/cluster-baseline.yml
- name: Cluster baseline config
  hosts: all
  become: true
  tasks:
    - name: Hostnames and /etc/hosts
      copy:
        content: |
          10.0.0.11 login01
          10.0.0.21 node01
          10.0.0.22 node02
        dest: /etc/hosts
    - name: Mount shared storage (/share)
      mount:
        path: /share
        src: 10.0.0.10:/share
        fstype: nfs
        opts: defaults,hard,intr
        state: mounted

- name: GPU drivers and firmware
  hosts: compute
  become: true
  tasks:
    - name: Install NVIDIA driver (unified version matrix)
      apt:
        name: "nvidia-driver-{{ nvidia_driver_version }}"
        state: present
    - name: Check IB firmware version
      shell: ibstat | grep Firmware || true

- name: Slurm config distribution
  hosts: compute
  become: true
  tasks:
    - name: Deploy slurmd.conf
      copy:
        src: "files/slurm/slurm.conf"
        dest: /etc/slurm/slurm.conf
        mode: "0644"
    - name: Start slurmd
      systemd:
        name: slurmd
        state: started
        enabled: true

Engineering notes:

• Unified version matrix: driver/firmware/Slurm versions defined in variables (group_vars)—no "this node 535, that node 545" drift.
• Ansible idempotency: rerunning the same playbook yields the same result; adding a node = edit the hosts inventory and rerun.
• Templated configs: generate slurm.conf with Jinja2 templates per role (controller/compute differ), avoiding hand-copied errors.

4. Step 3: Slurm Readiness Checks

# Controller side: all nodes registered and healthy
sinfo -N -o "%N %t %C %G" 

# Node health: DRAIN count should be 0
sinfo -t drained

# Acceptance job: run a GPU job on each compute node
srun --nodelist=node01 -N1 --gres=gpu:1 \
  nvidia-smi -L && echo "node01 GPU OK"
srun --nodelist=node02 -N1 --gres=gpu:1 \
  nvidia-smi -L && echo "node02 GPU OK"

# Batch acceptance: submit to all nodes (use -b to avoid a storm)
sbatch --wrap "nvidia-smi -L > /tmp/gpu_check_$SLURM_JOB_ID.log"

Readiness checklist:

Check	Command	Pass criteria
Node registration	sinfo -N	all idle, no drain
GPU visibility	srun --gres=gpu:1 nvidia-smi -L	GPU count per node matches config
Shared storage	srun df -h /share	all nodes see the same mount
Job scheduling	sbatch sleep 60	queues and runs normally
Network bandwidth	srun mpirun -np 2 pingpong	cross-node IB bandwidth meets spec

5. New-Node 30-Minute-Ready SOP

1. Rack & cable → 2. PXE install (DHCP assignment) → 3. run acceptance script (10 min, hardware issues surface immediately) → 4. Ansible full config (10 min) → 5. Slurm registration + acceptance jobs (5 min) → 6. join production.

Engineering takeaways:

1. Acceptance before config: never proceed with unresolved hardware issues.
2. Ansible idempotent + version matrix: adding a node = edit hosts and rerun; no drift.
3. Slurm acceptance trio: sinfo no drain → GPU job runs → shared storage consistent.
4. Keep baselines: acceptance reports + first benchmark in CI; performance regressions have a reference.
5. Auto-isolate faulty nodes: Slurm HealthCheckProgram probes periodically and auto-DRAINs—don't wait for user tickets.
6. Docs as code: delivery SOP lives in the repo (playbooks + acceptance scripts + checklist); newcomers just follow it.

6. Further Reading

• Ansible docs
• Slurm quickstart
• KB: 05. Automated Provisioning · Diskless Boot · Ubuntu Autoinstall · GPU Server Topology