Large-Scale Provisioning Pipeline: Golden Image, Version Management & Canary Rollout

Experiment Goal

PXE provisioning (Chapter 05) answers "how to install"; the Golden Image pipeline answers "install fast, install consistently, and be able to roll back" — especially at hundreds of nodes when drivers/systems need upgrades:

• How to build a reproducible Golden Image, not a "snapshot of a hand-installed system"?
• How to roll out image upgrades via canary, avoiding one upgrade killing a whole batch?
• How to manage image versions and roll back when something breaks?

This article dissects the Golden Image pipeline with commands and flows: creation, version management, canary rollout, and verification. Commands are generic practice examples; specific tools (Packer/Ansible/system tools) follow the installed versions.

Section 1: Golden Image Creation Principles

A Golden Image is not a "snapshot of an installed system" — it's a reproducible build artifact:

Build (build machine / CI)
  ├─ base system install (minimal install, avoid manual drift)
  ├─ config hardening (Ansible/cloud-init: timezone, kernel params, SSH, NTP, sysctl)
  ├─ baseline hardening (Chapter 01: BIOS/firmware baseline before install; image pins driver versions & config)
  └─ output Golden Image (versioned: 20261016-01-golden.tar.gz)

Core disciplines:

• Build with code, not clicks: all config changes go through Ansible/scripts; the image is rebuildable and auditable;
• Minimal + runtime layers: keep the base image minimal; layer business software (drivers, runtimes, monitoring) on top for granular upgrades;
• Versioning & checksums: each image carries a version + sha256; build records (content, time, changes) traceable — rollback requires knowing "what's inside each version".

Section 2: Image Version Management & Upgrade Flow

Image directory (versioned, Golden is read-only)
├─ 20261001-01-golden  (base + driver 550)
├─ 20261008-02-golden  (base + driver 570 + kernel update)
└─ 20261016-03-golden  (base + driver 570 + cuda 12.4 + security patches)

• Version rules: Golden images are never modified in place — new requirements produce new versions; old versions stay for rollback;
• Upgrade paths: node reinstall uses new Golden (provisioning pipeline); existing-node online upgrades follow canary (below);
• Rollback: Golden rollback = reinstall with the old version image (or online driver rollback); versioned images give rollback a target — unversioned images mean no rollback capability.

Section 3: Canary Rollout — Don't Let One Upgrade Kill a Batch

Image/system upgrades adopt canary strategy, same principle as release:

# Canary batch example (Ansible batch limit & failure threshold)
- hosts: gpu_nodes
  serial: 5            # 5 nodes per batch
  max_fail_percentage: 20
  tasks:
    - name: upgrade driver & kernel
      ...
    - name: verify (health patrol: GPU recognition, bandwidth, services)
      ...

Canary points:

1. Progressive batches: 1-2 nodes "smoke" (upgrade + full verification) → 5% batch → 20% batch → full; each batch passes verification before the next;
2. Verification as the gate: after each batch, run Chapter 01 baseline checks + Chapter 02 smoke benchmarks (GPU recognition, nvidia-smi health, training startup); on failure, stop and roll back that batch;
3. Batches, not time-slices: during canary, new and old versions coexist; the scheduler (Slurm) confirms new nodes join the pool and old nodes can roll back; business impact stays within a batch;
4. Window selection: run canary in business low peaks, aligned with Chapter 02 delivery cadence.

Engineering observation: the biggest cause of canary failure isn't the upgrade itself but insufficient verification — "starts after install" ≠ "training works". Verification must include real-load smoke (short training jobs / benchmarks); otherwise problems slip through canary and explode after full rollout.

Section 4: Image Verification — Read Back Before Delivery

• Build artifact read-back: unpack and verify sha256; check key files (driver modules, services, configs) exist with correct versions;
• Real-machine smoke: reinstall 1 node with the new image, run the full verification chain (baseline checks + benchmark smoke + service readiness);
• Record the baseline: save the new image's verification results as the reference for subsequent canary batches and full rollout — linked to the Chapter 02 benchmark toolkit.

Pipeline Checklist

• [ ] Golden Image built from code (Ansible/scripts), rebuildable & auditable
• [ ] Image versioned + sha256 + build records; old versions kept for rollback
• [ ] Canary batch strategy (smoke → 5% → 20% → full), verification as gate per batch
• [ ] Canary verification includes real-load smoke (short training/benchmarks), not just "boots"
• [ ] Rollback path explicit (reinstall old Golden / online driver rollback)
• [ ] Build artifact read-back + real-machine smoke archived

Engineering Takeaways

• The essence of image engineering is reproducibility: code-built + versioned + checksummed, Golden Image is no longer a "mystical snapshot";
• Canary is standard for upgrades: killing a whole batch of hundreds of nodes in one upgrade is unacceptable; progressive batches with verification gates are the floor;
• Rollback capability = version management capability: without versions there's no rollback target — version management comes first.

Next article preview: with the provisioning pipeline ready, pretraining clusters enter the "accounting" phase — the next article, Pretraining Compute & Cost Planning: GPU Estimation & Card-Hour Budget, breaks down capacity and cost models for training workloads.