Kickstart + Internal Mirrors + Ansible: Version-Consistent Delivery Across 48 Nodes

Installing 48 machines one by one, by hand, ends with "every node is slightly different." Version drift is the quietest debt in mid-size cluster delivery — it explodes in month three (performance comparisons) and month six (troubleshooting). This article records how a 384-GPU L40S IB cluster pinned 48 nodes to one software version with Kickstart + internal mirrors + Ansible.

1. Why "Version Consistency" Is the Core Delivery Goal

Approach	Node-to-node variance	Reproducibility	Troubleshooting
Hand-install per node	High (paths, versions, params changed ad hoc)	None	Reconcile every machine
Clone a shared image	Medium (old/new mix after image updates)	Weak	Unknown which node runs which version
Kickstart + unified mirror + Ansible	Low (same answer file + same mirror)	Strong	Reconcile one config across all

Judgment: version consistency buys more than convenience — it buys reproducibility. When something breaks you can state exactly what the machine runs, and when performance is off you can rule out "environment variance" as a variable.

2. Kickstart: Make OS Installation Declarative

• One ks.cfg covers partitioning, timezone, package groups, root policy, SSH config;
• Internal PXE/DHCP boot: 48 nodes provisioned with zero manual intervention;
• The system RAID1 (2×480G SATA) is pre-configured in the %pre section (storcli to create the volume, write-cache Enabled) — otherwise you discover RAID was never built after install.

Pitfalls:

1. The %pre RAID script must be idempotent — on reinstall the volume already exists; delete before create;
2. Firmware/BIOS variance can change disk order on identical models — use disk identifiers, not indexes, as RAID members;
3. First action after Kickstart: refresh from the internal mirror — never let nodes pull packages from the public internet.

3. Internal Mirror: The Foundation of Version Consistency

• Unified YUM mirror: openEuler 24.03 base + EPOL mirrored internally; all nodes point baseurl at the internal mirror;
• Unified pip mirror: training dependencies (PyTorch and friends) install from the internal PyPI mirror with pinned versions;
• Key packages pinned: kernel, driver, CUDA stack, NCCL versions are written dead in a build manifest — not "install latest".

Judgment: the internal mirror is not for speed — it is to remove public-internet drift from delivery. What the public mirror gives today and tomorrow can differ; the internal mirror is frozen.

4. Ansible: Configuration Rollout, Forbidden to Drift

• All configuration (kernel params, IRQ affinity, sysctl, NCCL env, MPI env) rolls out through Ansible roles;
• No ad-hoc per-node edits: anything tried "on this node first" must flow back into a role;
• Idempotency: re-running the role N times yields the same result; at delivery, running the playbook across all nodes is the reconciliation.

Verification:

# Reconcile the stack: run the same playbook + compare key package versions
ansible-playbook -i hosts.ini site.yml --check
ansible all -m shell -a "rpm -q kernel mlx5_objs cuda-12-x" | sort | uniq -c | wc -l
# Output of 1 line = all 48 nodes identical

5. Acceptance: How to Prove Consistency

Check	Criterion
Key package versions	rpm -q output identical across 48 nodes (1 line after dedup)
Kernel params	/proc/cmdline and key sysctls consistent
Config file hash	sha256 of key /etc/ files identical (Ansible --check: 0 changes)
Random sampling	Any 2 nodes run the same benchmark (STREAM/NCCL), results within noise

6. Delivery Checklist

1. Kickstart: idempotent RAID pre-config + internal boot, 48 nodes bulk-provisioned;
2. Internal mirror: YUM + pip point internally, key versions pinned in the build manifest;
3. Ansible: all config as roles, --check zero changes;
4. Consistency acceptance: package versions, kernel params, config hashes all pass;
5. Record the baseline image tag as the diff reference for future changes.

---

Based on field measurements from a 384-GPU L40S IB cluster delivery; exact versions and commands follow on-site validation.