BCM Cluster Management in Practice: The Bare-Metal Lifecycle from Software Image to Category

In one sentence: PXE plus Ansible handles the one-time act of installing a machine; BCM handles the persistent state of what that machine is supposed to look like — images, categories, kernel parameters and disk layouts become inheritable, overridable, bulk-deployable objects instead of variables scattered across dozens of playbooks.

1. Why Provisioning Scripts Break Down at Thousand-GPU Scale

At a few dozen machines, a PXE + Kickstart/Autoinstall + Ansible stack is sufficient: provisioning is a one-off event and a bad config is fixed by reinstalling. At hundreds or thousands of DGX/HGX-class nodes the problem changes shape:

1. State has no single source of truth. One node has a hand-edited /etc/sysctl.conf, another never blacklisted nouveau, a third partitioned its disks differently from its neighbours. None of these differences surface on their own — they show up as an unexplainable outlier during NCCL runs or baseline reproductions;
2. Baselines cannot be changed as a batch. Adding one kernel parameter to every compute node means a full Ansible sweep, plus your own bookkeeping of which nodes have actually rebooted into it;
3. Provisioning and scheduling are disconnected. Slurm/Kubernetes is deployed inside the provisioning script, so nobody guarantees that stack still exists after the image is swapped;
4. It is the vendor's delivery form. The standard practice inside NVIDIA DGX SuperPOD reference architectures is BCM: bcm-pod-setup, BaseOS images, category-level kernel args and disk layouts are part of the official delivery path.

BCM (NVIDIA Base Command Manager, formerly Bright Cluster Manager) folds all of this into a database-backed management layer: a node's filesystem is abstracted into an image, nodes are grouped into categories, and properties are inherited. The official definition is blunt: BCM manages a compute node's local filesystem as a software image, and an image is a directory on the head node containing a full Linux filesystem. When a compute node boots, the provisioning system sets the node up with a copy of the image.

2. The Object Model: Know Who Inherits From Whom

Six core objects decide where a given setting belongs:

Object	Definition	Typical use
device	A component in the cluster: physical node, virtual node, GPU, switch	Final per-machine override
category	Group of homogeneous nodes that inherit properties and an image	One each for compute, login, storage
nodegroup	Arbitrary operational grouping, homogeneity not required	"The batch we reboot tonight"
role	Functional role, e.g. provisioning node, storage node, Slurm client	Configuration implied by the role
rack	Physical rack	Rack-wide power operations
softwareimage	A complete root filesystem at /cm/images/<name>	The image baseline

NVIDIA publishes a very practical capability matrix showing what each object can do:

Object	Reboot	Power op	Assign image	Kernel parameters	Kernel modules
softwareimage	N/A	N/A	N/A	Yes	Yes
device	Yes	Yes	Yes	Yes	Yes
category	Yes	Yes	Yes	Yes	Yes
nodegroup	Yes	Yes	No	No	No
role	Yes	Yes	No	No	No
rack	Yes	Yes	No	No	No

Two semantics you must memorise:

• Kernel parameters are replaced, not merged. Parameters on a category replace those defined on an image wholesale, and parameters on a device replace those on a category or image. So when adding one parameter to a single node, be careful not to collapse the entire kernelparameters line down to just that one value — the documented way is append, not set;
• A category is not a nodegroup. Category members share the exact same configuration, disk layout and image; a nodegroup is only "the machines I want to operate on together", and its members may differ in hardware and in image. Treating a nodegroup as a category is the most common beginner's conceptual error.

3. cmsh: Session Semantics Matter More Than the Commands

cmsh is the BCM management shell. You can enter it interactively or hand it a command chain in one shot:

# Interactive
$ cmsh
[headnode]% softwareimage
[headnode->softwareimage]% list

# Batch mode: semicolons instead of nested prompts
$ cmsh -c "softwareimage;list"

Use exit to go back one level, Ctrl+D to leave the shell. The single most expensive gotcha: an asterisk in the prompt (e.g. [headnode->softwareimage*]%) means the change exists only in the current cmsh session and has not been written to the database. You must commit. Forget it and you get the worst kind of symptom — the command clearly succeeded, the nodes behave exactly as before.

[headnode->softwareimage]% set kernelversion 5.15.0-1078-nvidia
[headnode->softwareimage*]% commit

4. An Image Is a Root Filesystem

Images live under /cm/images/<name> on the head node. default-image ships with BCM as the base (same distribution as the head node, plus the node management software) and is what nodes without an explicitly assigned image boot. In practice you do not edit it directly — you clone it and customise per role:

[headnode->softwareimage]% clone default-image compute-image
[headnode->softwareimage]% use compute-image

Three lifecycle operations worth knowing:

• imageupdate: the node is already running; this synchronises changes made in the image down to the node's local filesystem, without a reboot. The right tool for "I edited a config file or installed a small package";
• grabimage: the reverse — capture the live state of a node you have tuned into the image. The right tool for "validate on a canary, then freeze it into the baseline"; check help grabimage and tab completion for the exact flags on your release;
• lock: lock the image so no node can pick it up until you unlock it. The fuse you install before publishing a new image — it prevents a node rebooting at 3 a.m. and pulling a half-finished image.

5. Four Ways to Change an Image

BCM does not require you to chroot in and hand-edit the image (though you can). It exposes the boot-critical settings as image properties.

5.1 Kernel version: a new kernel must be registered

Compute nodes load their kernel from the head node over PXE, not from local disk. So when the kernel inside the image changes, BCM has to know about it and regenerate the initramfs:

$ cmsh -c "softwareimage;list"
# Name                          Path                        Kernel version        Nodes
# default-image                 /cm/images/default-image    6.8.0-51-generic-64k  0

[headnode->softwareimage]% use baseos-image
[headnode->softwareimage]% set kernelversion 5.15.0-1078-nvidia
[headnode->softwareimage*]% commit

After commit, wait for a log line like Initial ramdisk for image dgx-os-6.3.2-h100-image was generated successfully. It can take minutes — do not exit the shell and do not reboot nodes before it lands.

5.2 Kernel parameters: set replaces, append adds

# Inspect the current value
$ cmsh -c "softwareimage; use baseos-image; get kernelparameters"

# Replace the whole line
$ cmsh -c "softwareimage; use baseos-image; set kernelparameters \
  \"nouveau.modeset=0 iommu.passthrough=1 systemd.unified_cgroup_hierarchy=0 \
  init_on_alloc=0 numa_balancing=disable\"; commit"

# Append only (note the leading space)
$ cmsh -c "softwareimage; use baseos-image; append kernelparameters \" nvme_core.multipath=n\"; commit"

The value passed to append must carry a leading space (unless the existing value is empty or ends with a space), otherwise the two parameters fuse into one token. Quote the whole string when setting multiple parameters, and escape carefully inside shell scripts.

5.3 Kernel modules: they must be in the initramfs

Modules needed before the root filesystem mounts or before networking comes up — bonding, multipath — must be explicitly added to the image's initial ramdisk:

$ cmsh -c "softwareimage; use baseos-image; kernelmodules; add bonding; commit"

5.4 Disk layout: RAID and mount points defined in XML

BCM describes partitions and arrays in an XML file, then attaches it to a category:

<diskSetup>
  <device><blockdev>/dev/nvme0n1</blockdev>
    <partition id="slash1"><size>max</size><type>linux raid</type></partition>
  </device>
  <device><blockdev>/dev/nvme1n1</blockdev>
    <partition id="slash2"><size>max</size><type>linux raid</type></partition>
  </device>
  ...
  <raid id="slash">
    <member>slash1</member><member>slash2</member>
    <level>1</level><filesystem>ext4</filesystem><mountPoint>/</mountPoint>
    <mountOptions>defaults,noatime,nodiratime</mountOptions>
  </raid>
  <raid id="raid">
    <!-- nvme2n1 ~ nvme9n1, level 0, mounted at /raid -->
  </raid>
</diskSetup>

The point is not the template itself but that the layout becomes a category property, so every newly provisioned node partitions itself exactly this way:

[headnode->category]% use compute
[headnode->category*[compute*]]% set disksetup /cm/local/apps/cmd/etc/htdocs/disk-setup/compute-disksetup.xml
[headnode->category*[compute*]]% commit

The official example uses eight drives in RAID 0 for /raid and two drives in RAID 1 for / — and the intent is clear: the root system needs redundancy (it has to boot when a drive dies), while the data volume needs capacity and bandwidth (it can be rebuilt).

6. Onboarding a New Node

The standard sequence is "clone a template → point it at a temporary category → let it boot once → identify the NICs → build the bond → switch to the real category". That is not bureaucratic detours: whether the physical NICs enumerate as eth2 or eth7 in the OS cannot be known in advance, so the node must boot once and be inspected.

$ cmsh
[headnode]% device
[headnode->device]% clone template-default-node oem-node001 --next-ip
[headnode->device*[oem-node001*]]% set category default
[headnode->device*[oem-node001*]]% commit

--next-ip makes the clone take the next free IP automatically, which removes a whole class of typo. Then boot the node over PXE: if its MAC is unknown, the Cluster Manager Node Installer drops into an interactive flow — choose Manually select node, pick the device you just created, accept, and let it finish.

Once the node is up, use BCM's remote shell to check the NICs:

[headnode->device]% use oem-node001
[headnode->device[oem-node001]]% rshell
root@oem-node001:~# ip a          # which port holds an address on the internal subnet?
root@oem-node001:~# dhclient      # bring up ports that are still down
root@oem-node001:~# ip a
root@oem-node001:~# exit

With the port names known, go back to cmsh and rewrite the interfaces — this is the step that turns "two physical ports" into "one LACP bond that also carries provisioning traffic":

[headnode->device[oem-node001]]% set category compute
[headnode->device*[oem-node001*]]% interfaces
[headnode->device*[oem-node001*]->interfaces*]% remove bootif
[headnode->device*[oem-node001*]->interfaces*]% add physical eth2
[headnode->device*[oem-node001*]->interfaces*[eth2*]]% exit
[headnode->device*[oem-node001*]->interfaces*]% add physical eth3
[headnode->device*[oem-node001*]->interfaces*[eth3*]]% exit
[headnode->device*[oem-node001*]->interfaces*]% add bond bond0
[headnode->device*[oem-node001*]->interfaces*[bond0*]]% set dhcp yes
[headnode->device*[oem-node001*]->interfaces*[bond0*]]% clear ip
[headnode->device*[oem-node001*]->interfaces*[bond0*]]% set network internalnet
[headnode->device*[oem-node001*]->interfaces*[bond0*]]% set interfaces eth2 eth3
[headnode->device*[oem-node001*]->interfaces*[bond0*]]% set mode 4
[headnode->device*[oem-node001*]->interfaces*[bond0*]]% set options "miimon=100 xmit_hash_policy=layer3+4"
[headnode->device*[oem-node001*]->interfaces*[bond0*]]% exit
[headnode->device*[oem-node001*]->interfaces*]% exit
[headnode->device*[oem-node001*]]% set provisioninginterface bond0
[headnode->device*[oem-node001*]]% commit
[headnode->device[oem-node001]]% reboot

Four parameters deserve a pause:

• set mode 4 is LACP (802.3ad) and requires the switch side to be configured for link aggregation;
• miimon=100 polls the link every 100 ms and sets how fast failover is detected;
• xmit_hash_policy=layer3+4 hashes on IP plus port, which distributes multi-flow traffic more evenly;
• provisioninginterface bond0 puts provisioning traffic on the bond; without it PXE and image transfer still ride a single port.

After the reboot, rshell in again and run ip a — the step is only done when bond0 is UP with an internal address. Then clone the configuration to the rest of the fleet:

[headnode->device]% clone oem-node001 oem-node002 --next-ip
[headnode->device]% clone oem-node001 oem-node003 --next-ip
[headnode->device]% commit

7. Bulk Operations: Three Selectors Are Enough

# Reboot by node-name range
$ cmsh -c "device; reboot -n node-[002-031]"

# Reboot by category
$ cmsh -c "device; reboot -c compute"

# Reboot by nodegroup
$ cmsh -c "device; reboot -g tonight-batch"

# Power operations use the same selectors
$ cmsh -c "device; power -c compute off"

Rebooting by name range is the core move of a canary rollout: give the new image to exactly one node, accept it, then widen to a range, and only then to the whole category. NVIDIA's documentation recommends that order as well.

Two delivery details:

• On machines such as DGX B200, PXE can power the node off after the NBP file is downloaded; set the category bootloader explicitly to GRUB with set bootloader grub;
• BCM controls power through IPMI/BMC, which requires it to know the BMC credential: use the bmcsettings sub-mode under category (or device) and set password, otherwise power/reboot fail without out-of-band access.

8. In-Place Update vs Image Replacement: Two Very Different Roads

This is the fork with the heaviest consequences, and the documentation is explicit about it.

Road one: in-place update. You install packages, edit configs and tune kernel parameters inside the existing image, then push the diff to running nodes with imageupdate, or let it take effect at the next reboot. On this road, your Slurm/Kubernetes deployment, your extra software and your config changes all survive.

Road two: replace the image. You create (or import) a new image and set softwareimage it onto a category or device. Then:

• every additional piece of software and configuration has to be reapplied to the new image;
• Slurm and Kubernetes must be redeployed:

$ cm-wlm-setup -c cm-wlm-setup.conf
$ cm-kubernetes-setup -c cm-kubernetes-setup.conf

If you created a new category along the way, add it to the categories list in cm-wlm-setup.conf / cm-kubernetes-setup.conf, otherwise the new nodes never enter the scheduling domain.

The one-shot post-install tool bcm-pod-setup is for first-time deployment only. It does heavy lifting: it writes DGX-model-specific kernel parameters and modules, applies model-specific sysctl (for example disabling ARP forwarding on A100/H100/B100 and configuring net.ipv4.* flow-based routing rules for GB200), and installs the disk layout and udev rules for the model (plain layout for GB200, persistent layout for everything else by default).

$ apt-get install bcm-post-install && module load bcm-post-install
$ bcm-pod-setup --dgx-type H100 --dgx-image baseos-image

Running it on an already-deployed cluster means plastering "initial-install assumptions" back onto a production image — the documentation says not to, and points you at image properties (kernel args, and so on) for later adjustments.

9. Acceptance Checklist

1. cmsh -c "softwareimage;list" shows the expected node count, and the Kernel version matches reality;
2. after commit, the initramfs success line was observed — the shell was not exited prematurely;
3. kernelparameters was read back with get to confirm whether it was replaced or appended, especially for that one node you patched individually;
4. the category's disksetup points at the right XML, and a freshly provisioned node confirms RAID levels for / and /raid via lsblk;
5. ip a shows bond0 UP in LACP mode, and provisioning rides the bond rather than a single physical port;
6. the canary order is on record: single node → name range → whole category;
7. for any image-replacement release, the Slurm/K8s redeploy command and the config-file edits are documented, and partitions/queues accept jobs again.

Three Closing Takeaways

• BCM delivers state, not scripts: image, category and kernel parameters together define what a machine should look like, and no one-shot script replaces that inheritable declaration;
• Two semantics must be memorised: kernel parameters are replaced rather than merged, and nodegroups carry bulk operations but no configuration;
• A new image is not a new kernel: in-place updates can ship diffs only, while replacing the image implies redeploying Slurm/K8s — skip that and the cluster sits for hours in a "provisioned but unschedulable" state.

Further Reading

• Cluster Delivery Automation: PXE Install, Ansible Configuration and Slurm Readiness
• PXE Troubleshooting Handbook for Large-Scale Provisioning
• Large-Scale Provisioning Pipeline: Golden Image, Version Management & Canary Rollout
• NVSwitch Fabric Manager Operations: Configuration, Restart Order and SXid Triage