Containerizing HPC Clusters: Apptainer SIF, GPU Injection and Slurm Integration

In one sentence: on a cloud host a container means "package the app"; on an HPC cluster it means "isolate the environment without root". The two are not the same stack: you cannot give every user a Docker daemon on a shared machine, which is why Apptainer exists — user-space execution, single-file images, callable straight from the scheduler. The problem it solves is never "how do I package software", it is "how do I run a fixed software environment on a machine shared by five hundred people when I have no root".

1. Why a Shared Cluster Will Not Hand Users Docker

Docker's model is daemon plus root: dockerd runs as root, owns the image layers and the overlay mounts, and users issue instructions over a socket. On a shared cluster that has three direct consequences:

Problem	What it looks like in practice
Concentrated privilege	Anyone who can reach the docker socket effectively has root (they can mount the host root with -v /:/host)
Shared state	Every user's images, containers and networks live in one daemon's state, visible to and affecting each other
Scheduler mismatch	On job exit, who cleans the container up; how GPUs and IB devices are handed to a specific job — the daemon model has no natural answer

So the accepted cluster practice is a different runtime: no resident daemon, the container is started by the user's own process and dies with it, and the image is a read-only file you can drop on shared storage. That is the design premise of Apptainer, the successor to Singularity. It does not replace Docker; the two divide the work — Docker builds, Apptainer runs.

Dimension	Docker	Apptainer
Runtime shape	Resident daemon, clients issue commands	No daemon; apptainer exec is just a process launch
Identity	root in the container is root on the host (barring rootless setups)	You are yourself inside; "root-like" behaviour needs --fakeroot
Image	Layered, stored in a local graph driver	Single-file SIF (compressed read-only squashfs) by default, shareable as-is
GPU	Needs the NVIDIA Container Toolkit	--nv injects driver and basic CUDA libraries from the host
Scheduler	Needs extra wrapping	Invoked directly inside srun / mpirun

2. SIF: One File Is One Environment

apptainer build produces a SIF (Singularity Image Format) by default: compressed, read-only, single file. That is precisely what makes it cluster-friendly — it sits on NFS or a parallel file system and is usable everywhere after one distribution, with no docker pull per machine.

# Convert straight from Docker Hub / an OCI registry into a SIF
apptainer build pytorch-24.05.sif docker://nvcr.io#nvidia/pytorch:24.05-py3

# Build from a definition file (the reproducible, by-the-book path)
apptainer build myapp.sif myapp.def

# Need to poke around interactively? Use a sandbox (a writable directory), then convert back
apptainer build --sandbox myapp/ myapp.def
apptainer build myapp.sif myapp/

Build flags that are often overlooked:

Flag	Purpose	When to use it
--arch	Pull an image for another architecture (arm64 on an amd64 host)	Cross-architecture delivery, edge nodes
--notest	Skip the %test section	The build host has no GPU but %test runs a GPU self-check
--reproducible	Use the source image's creation date as SOURCE_DATE_EPOCH, skip the UUID	You need byte-identical output from one definition
--mksquashfs-args	Change compressor and level	Trading image size against startup time

The compressor deserves its own paragraph, because it directly sets the decompression cost on every start:

Algorithm	Default level	Optimised for
gzip	9	Compatibility (the default)
lz4	1	Fastest
zstd	15	Smallest

For an 8 GB framework image, zstd buys a smaller download and charges more CPU at startup. On nodes that are already CPU-tight, lz4 is often the better trade than zstd. That is an engineering call, not "smaller is always better".

Three Paths to Building Without Root

Users have no root on the cluster, so how does an image get built? Three routes, in order of practicality:

1. A privileged build host: run apptainer build where you do have root or user namespaces, then copy the SIF over. Most common and cleanest;
2. --fakeroot (unprivileged build): user namespaces manufacture a root-like identity. It is implied automatically when an unprivileged user builds from a definition file;
3. Remote build / conversion from an existing artefact: docker save on a Docker host then apptainer build x.sif docker-archive:x.tar, or use a remote build service.

There are three implementations of --fakeroot with very different capabilities, and you must know which one you are in:

Mode	Prerequisites	Limits
rootless (/etc/subuid and /etc/subgid mappings)	Admin pre-configures UID/GID ranges and enables user namespaces	The most complete emulation: can change file ownership inside, can su/sudo
Root-mapped user namespace	Only needs user namespaces	Maps root to you alone; weaker than rootless
fakeroot command only	The host has fakeroot	Only fools some install scripts; many packages still fail

One hard boundary: fake root can never touch a host file it has no rights to. You are root inside and still an ordinary user outside; files created by container-root belong to you on the host. Expecting --fakeroot to edit /etc/shadow, to mount, or to load kernel modules will not work — a container is not a virtual machine; it shares the host kernel.

3. The Mount Model: More Is Bound In Than You Think

On startup Apptainer binds a set of host paths into the container. The default list is:

• $HOME
• the current working directory (unless it contains symlinks that resolve differently)
• /dev, /proc, /sys
• /tmp, /var/tmp
• /etc/hosts, /etc/localtime
• plus /etc/passwd, /etc/group and /etc/resolv.conf, adapted to the container's configuration

Friendly for interactive use, but on a cluster job it has two side effects.

First, binding $HOME hides the image's own $HOME. If the image ships data or config under $HOME, the host home covers it. Keep the image's content with --no-home (equivalent to --no-mount home):

apptainer exec --no-home myapp.sif bash -lc 'ls -la $HOME'

Second, --containall replaces $HOME, /tmp and /var/tmp with in-memory temporary directories. That is isolation, and also a trap: a job writing several GB of scratch into /tmp is really consuming RAM, and those files vanish when the job ends, leaving nothing to debug. To use --containall and still see the image's home content, add --no-home.

The usual cluster pattern is bind only the data volumes and nothing else:

apptainer exec \
  --bind /lustre/project/teamA:/data:ro \
  --bind /lustre/scratch/$USER:/scratch \
  --env PYTHONPATH=/data/lib \
  myapp.sif python /data/train.py

Points worth remembering:

• the spec is src:dest:opts, with opts being ro or rw (default rw);
• paths containing : or , break --bind; switch to --mount type=bind,src=...,dst=...;
• the destination need not exist beforehand — Apptainer fills it in via overlay (not allowed under --writable);
• in batch jobs APPTAINER_BINDPATH is steadier than the command line, especially when the container is started via its runscript;
• to switch system mounts off, use --no-mount by name (home, tmp, sys, dev) or by path (/data2).

4. GPU: What --nv Actually Does

People treat --nv as a "GPU switch". It does three concrete things:

1. It makes the host's /dev/nvidiaX device nodes available inside the container;
2. It binds the basic CUDA libraries from the host into the container (from the configured library list, resolved through the ldconfig cache);
3. It sets LD_LIBRARY_PATH inside the container so applications use those bound-in libraries.

Which imposes two constraints:

• The container's CUDA version must be supported by the host driver. However new the image, an old driver will not run it — the same rule as installing CUDA on bare metal;
• The container's libc must not be too far from the host's. The libraries come from the host, so a container libc markedly older (or newer) than the host's produces mismatch errors.

Limiting Which Cards Are Visible

By default the container sees all host GPUs. To allocate per job, use CUDA_VISIBLE_DEVICES — and note how it is injected:

# Inject through the APPTAINERENV_ prefix
APPTAINERENV_CUDA_VISIBLE_DEVICES=0,1 apptainer exec --nv myapp.sif nvidia-smi

CUDA_VISIBLE_DEVICES only makes CUDA programs "blind" to the other cards; the device nodes are still there. Real device-level isolation needs the experimental --nvccli, which drives nvidia-container-cli: it can pass only specified GPUs or MIG instances into the container (NVIDIA_VISIBLE_DEVICES, by UUID or MIG identifier) and can select which capabilities are injected via NVIDIA_DRIVER_CAPABILITIES (compute, utility, graphics — Apptainer's default is compute,utility). The price: it is experimental, needs a writable layer (--writable-tmpfs is implied), and cannot be combined with a privileged setuid installation.

One counter-intuitive combination: with --contain plus --nvccli, if NVIDIA_VISIBLE_DEVICES is not set there are no GPUs in the container at all — only a warning. Restore the default behaviour by setting NVIDIA_VISIBLE_DEVICES=all or 0 explicitly.

That Classic Error: CUDA_ERROR_UNKNOWN

The host runs nvidia-smi fine, the image ships CUDA, and the container dies with CUDA_ERROR_UNKNOWN. The cause is not the container but an incompletely initialised driver stack:

CUDA depends on several kernel modules, and some are not loaded at boot — they are pulled in on first use by a setuid-root binary. Privilege escalation is blocked inside a container, so that binary cannot run and half the modules are missing.

The fix is ordered: initialise the driver stack on the host first — run a CUDA program, or modprobe nvidia_uvm as root, then keep nvidia-persistenced running so the driver is not unloaded when idle. This belongs in the delivery baseline: run a GPU self-check right after compute nodes boot, rather than leaving first-time initialisation to a user job.

5. Environment Variables: Who Overrides Whom

The container environment is five layers stacked, lowest precedence first:

1. Variables defined in the base image (a Dockerfile ENV);
2. The %environment section of the definition file;
3. Variables written to $APPTAINER_ENVIRONMENT in %post (these beat %environment);
4. Variables injected via the APPTAINERENV_ prefix, --env, --env-file;
5. The host environment (fills gaps only, never overrides what exists, and is not imported at all under --cleanenv).

Two frequent traps:

• Module spill-over: a cluster module load sets a pile of variables (PYTHONPATH, LD_LIBRARY_PATH, assorted *_ROOT) that leak into the container, turning a "reproducible image" into "whatever you happened to load". --cleanenv (-e) in the job script is the steady choice; pass what you need explicitly with --env;
• The reversed LD_LIBRARY_PATH rule: the host's is not imported, but CUDA libraries bound in by --nv are appended to LD_LIBRARY_PATH, so the container's own copies win. To prefer the host driver libraries, set PREPEND_LD_LIBRARY_PATH=/.singularity.d/libs inside the container.

For Slurm jobs the cheapest habit is put variables in an env file and ship it with the job — no giant command line, and something to point at when debugging:

# job.env
NCCL_DEBUG=INFO
OMP_NUM_THREADS=8
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

# At submission time
apptainer exec --env-file job.env --nv myapp.sif python train.py

6. MPI and Slurm: Two Models, One Alignment Rule

There are only two ways to put an MPI program in a container, and picking wrong yields the bizarre "every rank is 0" output:

Model	How	Upside	Cost
Hybrid	The host's mpirun / srun starts the container, the container also has MPI, and connects back to the host over PMI	Natural scheduler integration, near-bare-metal performance	The container's MPI must be version-compatible and support the same PMI2/PMIx
Bind	The container has no MPI; the host's MPI directory is --bind-mounted in	Small images, version consistency for free	Requires user-defined binds; the compile-time MPI must still match the host

# Hybrid: the host's mpirun launches the container
mpirun -n 8 apptainer exec --nv myapp.sif /opt/mpitest

# Equivalent in Slurm (srun launches one container per task)
srun --ntasks=8 --gres=gpu:1 apptainer exec --nv myapp.sif /opt/app

Check alignment on three points: MPI versions compatible, process management interface identical (a container MPI that only speaks PMI2 will misbehave under a PMIx-only srun), interconnect libraries available inside (IB needs ucx/verbs; with --contain the /dev tree is minimal, so bind /dev/infiniband if necessary).

Two documented traps:

• UCX misbehaves under unprivileged user namespaces: the symptom is broken communication; the workaround is UCX_TLS=sysv,ib;
• Each apptainer command gets its own user namespace, which breaks MPI. --sharens puts the batch into one namespace, provided they were started by the same parent process (such as mpirun) on the same node.

The diagnostic signature is easy: if every rank reports itself as rank 0, the MPI stacks are mismatched (host and container are not the same MPI), not the scheduler.

7. Image Distribution: Do Not Let 200 Nodes Pull at Once

This is where clusters differ most from a laptop, and where day one usually goes wrong.

Wrong: put apptainer pull docker://... in the job script. Two hundred nodes decompress the same image simultaneously, the egress link saturates, jobs time out together, and 200 copies of the cache land on shared storage.

Right, in three steps:

1. Build once, centrally, and put the artefact on a shared path (say /lustre/apps/images/) with version and checksum alongside;
2. Distribute by verification, not by re-pulling: nodes only run sha256sum and fetch when it mismatches; move big files with scp/rsync or the site's distribution tree;
3. Point the cache at local disk: APPTAINER_CACHEDIR at node-local NVMe, never NFS — otherwise decompression's flood of small files buries the metadata service.

# Node side: verify first, then use — avoids half-written files
sha256sum -c /lustre/apps/images/pytorch-24.05.sif.sha256 && \
  apptainer exec --nv /lustre/apps/images/pytorch-24.05.sif python -c "import torch;print(torch.cuda.device_count())"

Two performance judgments that are engineering experience rather than vendor statements:

• A SIF on a parallel file system pays network latency on first-start random reads, so startup is visibly slower than from local disk; for large images, short jobs and many nodes, local cache plus periodic sync usually wins;
• squashfs decompression costs CPU. The higher the compression and the tighter the node's CPU, the slower the start. For jobs that only run tens of seconds, that overhead is ugly in the profile.

8. Deployment Checklist and Limits

Stage	Point
Naming	app-frameworkversion-cudaversion-date.sif, with a matching .sha256 beside it
Build	Definition file in Git; --reproducible; CPU self-check in %test, GPU self-check run separately outside --notest
Distribute	Build once, land on shared storage, verify per node; cache directory on local disk
Run	--cleanenv plus --env-file; --bind only the data volumes needed; --nv with the GPUs this job owns
Audit	%labels inside the image (build time, source, commit), readable with apptainer inspect

Limits, stated plainly:

• A container is not a virtual machine: it shares the host kernel, cannot load kernel modules, cannot change host sysctls, and --fakeroot is not root;
• Containers are not automatically lossless: mmap over NFS, decompression of a compressed image and the in-memory tmp of --containall all change the performance picture; re-measure the baseline inside the container before quoting a benchmark;
• Do not use --writable casually: a writable SIF on shared storage means one mutable state shared by the whole cluster; make temporary changes with --writable-tmpfs;
• Signing and encryption are optional but worthwhile: apptainer sign / verify when tamper resistance matters, encrypted containers for sensitive images (decrypted in memory at runtime, never written to disk).

Three Closing Takeaways

• Docker builds, Apptainer runs: what the cluster needs is "no daemon, single file, callable from the scheduler", not another image registry;
• --nv injects, it does not switch on: the driver and basic CUDA libraries come from the host, so "the CUDA version in the image" must obey "the driver version on the host";
• Image distribution is cluster engineering, not a container problem: build once, distribute by checksum, keep the cache on local disk — skip those three and containerisation will collect the debt at scale.

Further Reading

• Slurm Job Submission and Management Guide
• Ubuntu 22.04 + Slurm 24.05 Cluster Deployment
• Deploying an NVIDIA PyTorch Environment with Docker
• AI4S Reproducible Workflows: Data Governance and Containerisation
• GPUDirect Storage in Practice: NVMe to GPU Memory, Bypassing the CPU