PXE Batch Provisioning Server Architecture: High-Availability DHCP/TFTP/HTTP Design

Note: This is a review article (no first-hand experimental data), based on public documentation and common production practices.

Experiment Objective

The provisioning articles added earlier (PXE diskless boot, Golden Image pipeline) answered the client side: "how does a node install." This article answers the server half: when 512 machines PXE-install at once, which of DHCP/TFTP/HTTP collapses first? How do you keep the provisioning server from becoming the delivery bottleneck? The goal is a production-ready PXE server architecture: the responsibilities and parameters of the three services, a concurrent-throughput model, HA and failure retry, and an on-site troubleshooting path.

Code Walkthrough

Part 1: DHCP configuration (the key to PXE bootstrapping)

# /etc/dhcp/dhcpd.conf snippet: PXE key options
subnet 10.0.0.0 netmask 255.255.255.0 {
  range 10.0.0.100 10.0.0.250;          # provisioning address pool
  option routers 10.0.0.1;
  option domain-name-servers 10.0.0.1;
  next-server 10.0.0.2;                 # TFTP server address
  filename "pxelinux.0";                # boot filename
  option arch code 93 = unsigned integer 16;  # arch-based boot selection
  class "pxeclients" {
    match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";
    option vendor-encapsulated-options 06:01:01:0a:00:00:02;  # PXE server IP
  }
}

Key points:

1. next-server and filename are the PXE bootstrap "entrance": after DHCP, the client pulls pxelinux.0 from TFTP based on these two. Point them wrong and machines hang at "no boot filename received."
2. option arch distinguishes UEFI/BIOS: UEFI clients need grubx64.efi; BIOS clients need pxelinux.0. A mixed machine room that does not branch by architecture fails half its installs.
3. class "pxeclients" gives boot parameters only to PXE requests: ordinary DHCP clients (office network) do not download boot files, preventing accidental provisioning.
4. The pool must match provisioning concurrency: range size determines "how many IPs can go out at once" — 512 concurrent installs need 512+ available addresses.

Part 2: TFTP and HTTP layering

# Topology: small files via TFTP, large files via HTTP
# tftp-hpa: /srv/tftp/{pxelinux.0, grubx64.efi, kernel initrd}
# nginx:   /srv/pxe/{vmlinuz, initrd.img, kickstart/autoinstall config, OS images}

# nginx snippet: images served only to the provisioning subnet
server {
    listen 80;
    root /srv/pxe;
    allow 10.0.0.0/8;
    deny all;                     # provisioning assets are intranet-only
    client_max_body_size 4g;      # large images
    location /images/ {
        alias /srv/pxe/images/;
        sendfile on;              # zero-copy, high-concurrency throughput
        aio on;
    }
}

Key points:

• TFTP holds only small boot files; large files go over HTTP: TFTP is "one packet per request + ACK," so single-connection throughput is tiny (tens of KB/s) and hundreds of concurrent clients collapse it. After the boot stage, the kernel and OS images are pulled via HTTP (PXELINUX supports http:// paths), boosting throughput by orders of magnitude.
• sendfile on + aio on: image files are sequential large reads; zero-copy pushes files straight from disk into the NIC, keeping nginx CPU negligible and supporting hundreds of concurrent downloads.
• allow 10.0.0.0/8; deny all;: provisioning images contain kickstart configs (possibly root passwords and partitioning scripts); limiting access to the provisioning subnet is a security baseline.

Principles

Load model of the three services

Service	Protocol traits	Concurrency bottleneck	Scaling
DHCP	UDP broadcast/unicast, instantaneous bursts	high concurrency, small per-request cost; pool and response speed	pool expansion, DHCP failover
TFTP	UDP one-packet-one-ACK, tiny throughput	easiest to collapse: one session per client, congestion at hundreds	keep small files only; move large to HTTP
HTTP	TCP long connections, high throughput	disk IO and NIC	nginx + local SSD + multiple NICs

Design rule: moving traffic from TFTP to HTTP is lesson one of PXE server HA. In real provisioning, "stuck at boot" is usually TFTP congestion — clients retry TFTP downloads repeatedly, showing as endless "PXE spinning."

Concurrent provisioning throughput model

A 1Gbps NIC on a single server is ~110MB/s theoretical; a minimal OS image is 1-2GB (with drivers/tools) — so one hour can feed 200-400 machines in theory; in practice disk and protocol overhead cuts it to 50%-70%. For 512 machines, the bottleneck is not the server but each node's disk-write speed (5-10 min per OS install on SSD). The server just must not be the slower link: HTTP bandwidth ≥ Σ node disk write speeds.

HA and failure retry

1. DHCP failover: two DHCP servers as a failover peer; if the primary dies the secondary takes over, avoiding "address service vanished mid-install."
2. TFTP/HTTP redundancy: put LVS/keepalived virtual IP in front of nginx; TFTP and HTTP point at the same virtual IP for seamless failover.
3. Centralize provisioning logs: PXE boot failures must show "which step" — DHCP logs (did not receive boot filename), TFTP logs (file missing/timeout), HTTP logs (404/broken stream). Aligning the three log timelines is the troubleshooting entry.
4. Failure retry must be idempotent: PXE clients retry on their own; the server must guarantee idempotency — repeated installs must not corrupt existing configs (guard partitioning scripts, e.g. clearpart --all behind a %include check), or retries cause cascading failures.

Results

(Estimated, for capacity design reference)

Metric	Single-server estimate	Note
DHCP throughput	thousands of requests/s	not a bottleneck; watch the pool
TFTP concurrency ceiling	50-100 clients	beyond that: congestion, stuck boot
HTTP concurrent downloads	200-400 clients (1Gbps)	near-linear with sendfile
512-node full install	1-2 hours	node disk-write bound
Dual-node HA switchover	seconds	DHCP failover + virtual IP

Engineering Takeaways

1. Move traffic before talking HA: push TFTP down to "boot small files only" and serve images over HTTP — half the reliability problems disappear.
2. Isolate the provisioning VLAN from production: PXE serves only the provisioning subnet, preventing accidental installs and image leakage, and keeping provisioning traffic off the production network.
3. Read the three logs together: align DHCP/TFTP/HTTP timestamps; troubleshooting goes from guessing to protocol-level localization.
4. Pair with the Golden Image pipeline: the server architecture is set; image quality is chapter-05's golden-image-pipeline job — the server only guarantees "the image reaches every machine."

Next article preview: with provisioning covered, we return to another way to obtain compute — "Cloud GPU Compute: Cost Models and Elastic Architecture."