NVSwitch Fabric Manager Operations: Configuration, Restart Order and SXid Triage

In one sentence: on a machine without NVSwitch, NVLink between GPUs is a point-to-point link; once NVSwitch enters the picture, whether GPUs can see each other and whether bandwidth is usable is no longer decided by the driver alone — a resident process called Fabric Manager configures the whole switch set into one shared fabric at boot. If it is not running, eight GPUs are eight islands.

1. What Fabric Manager Actually Does

NVSwitch turns every GPU in a server into a fully connected network: any two GPUs can talk through the switch instead of relying on direct links. That network does not work just by plugging it in — it has to be initialised, routed and monitored. The core software stack has two parts:

• the NVSwitch kernel driver, which performs low-level hardware management in response to FM requests;
• Fabric Manager (FM), a privileged user-space process that configures NVSwitch into a single memory fabric shared by all GPUs, programs routing, coordinates initialisation with the GPU driver, and monitors NVLink/NVSwitch errors.

NVIDIA describes it as the core software stack for NVSwitch management, alongside in-band and out-of-band monitoring paths that report NVSwitch and GPU errors and status. That positioning implies three operational facts:

1. FM is a service, not a tool. It must be resident, enabled at boot, and owns logs and a state file;
2. It has strict startup preconditions. GPUs and NVSwitches must be bound to nvidia.ko before FM can take over;
3. When it exits, CUDA job semantics change immediately. With the default ABORT_CUDA_JOBS_ON_FM_EXIT=1, FM exiting aborts running CUDA jobs.

2. Four Platform Generations: Know Which One You Are On

NVSwitch platforms differ substantially in components and in what operators actually do. This is not background colour — it decides which config item you edit and which log you read:

Platform	NVSwitch count and generation	Key difference
HGX-2 / V100	6 first-generation NVSwitches, 16 GPUs across two baseboards	Baseboards interconnected via trunk ports
HGX A100	6 second-generation NVSwitches	Trunk failures have dedicated HA degraded modes
HGX H100	4 third-generation NVSwitches	Dual-baseboard interconnect not supported; NVLink trained in hardware by ALI
HGX B200 / B300 / B100	2 fourth-generation NVSwitches	NVSwitches no longer appear as PCIe devices; presented through the ConnectX-7 bridge

The fourth generation is an architectural break that triggers three consequences:

• NVLSM appears. Generation four introduces the NVLink Subnet Manager (derived from the InfiniBand subnet manager) to own NVSwitch routing tables and PKEYs. FM handles GPU-side routing and the partitioning API, and the two cooperate over IPC — by default on /var/run/nvidia-fabricmanager/fm_sm_ipc.socket;
• Logs split. FM logs to /var/log/fabricmanager.log, while NVLSM defaults to /var/log/opensm.log (log_flags 0x3, i.e. ERROR plus INFO);
• Packages and kernel floor change. Generation four uses nvidia-open-<branch> plus nvlink5-<branch> (on RHEL, nvidia-driver-<branch>-open and nvlink-<branch>), requires kernel 5.17 or newer, and additionally needs libibumad3 and infiniband-diags.

3. The Service Surface: Commands, Unit, Packages

FM runs as a standard systemd service, and these commands are the daily routine:

systemctl status nvidia-fabricmanager
systemctl start nvidia-fabricmanager
systemctl stop nvidia-fabricmanager
systemctl enable nvidia-fabricmanager
journalctl -u nvidia-fabricmanager --since "30 min ago"

The unit lives at /lib/systemd/system/nvidia-fabricmanager.service. It does not launch nv-fabricmanager directly — it calls /usr/bin/nv-fabricmanager-start.sh, which decides per platform whether to start FM, NVLSM, or both. So when you need to know which processes are actually up, do not read the systemctl colour alone.

The binary is nv-fabricmanager, and its handful of arguments are all worth knowing:

Argument	Purpose
-c / --config	Use a non-default configuration file
-r / --restart	Resiliency restart flow for multitenancy (equivalent to FABRIC_MODE_RESTART=1)
-d / --database	Do not use on single-node HGX/DGX; explicitly not applicable

Installation splits by distribution:

# Ubuntu / Debian
apt-get install cuda-drivers-fabricmanager-<branch>
# RHEL family
dnf module install nvidia-driver:<branch>/fm
# SUSE
zypper install cuda-drivers-fabricmanager-<branch>

Minimum driver versions rise with each generation: 450.xx for HGX-2/A100, 525.xx for H100, 570.xx for B200/B300/B100. DGX systems ship with it preinstalled and enabled, while HGX systems require you to install, enable and start it yourself — which is the most common reason a freshly populated server shows no P2P.

Package structure matters too: nvidia-fabricmanager-<version> is the core package containing the FM service process, the service unit and topology files — bare metal needs only this one. nvidia-fabricmanager-devel-<version> carries the SDK and libraries, needed only for Shared NVSwitch or vGPU multitenancy.

4. fabricmanager.cfg: Only a Handful of Items Matter

The default configuration is at /usr/share/nvidia/nvswitch/fabricmanager.cfg. Changes require a service restart — the file is read at service startup.

Config item	Default	Meaning
LOG_FILE_NAME	/var/log/fabricmanager.log	Log path, up to 255 characters
LOG_LEVEL	4	0 disabled, 1 CRITICAL, 2 ERROR, 3 WARNING, 4 INFO and above
LOG_APPEND_TO_LOG	1	Append rather than overwrite on each start
LOG_FILE_MAX_SIZE	1024	In MB; once reached, logging stops (it is not rotation)
LOG_MAX_ROTATE_COUNT	3	Rotation only when non-zero; total log volume is roughly MAX_SIZE × (COUNT + 1)
LOG_USE_SYSLOG	0	Set to 1 to redirect into syslog
FABRIC_MODE	0	0 bare metal / full passthrough, 1 Shared NVSwitch, 2 vGPU multitenancy
FABRIC_MODE_RESTART	0	Set to 1 for the multitenancy resiliency restart sequence
FM_CMD_BIND_INTERFACE	127.0.0.1	FM SDK/API listener interface (multitenancy)
FM_CMD_PORT_NUMBER	6666	FM SDK/API TCP port (multitenancy)
STARTING_TCP_PORT	16000	Start of FM internal communication/IPC ports
STATE_FILE_NAME	/tmp/fabricmanager.state	Multitenancy resiliency state file
DAEMONIZE	1	Run as a background daemon
ABORT_CUDA_JOBS_ON_FM_EXIT	1	Abort CUDA jobs when FM exits

Three practical conclusions:

1. Read LOG_FILE_MAX_SIZE carefully: once the limit is hit, FM skips further logging rather than rotating. With LOG_MAX_ROTATE_COUNT set to 0, a 1 GB file fills up and you lose logs exactly when you need evidence. NVIDIA's recommendation is to stay at INFO with rotation enabled;
2. STATE_FILE_NAME lives in /tmp. A reboot wipes /tmp and breaks multitenancy resiliency state. Production deployments should move it to persistent storage;
3. ABORT_CUDA_JOBS_ON_FM_EXIT only matters before H100. The documentation notes the behaviour no longer applies on H100 and later, and is only meaningful in bare metal / full passthrough mode.

Generation-four systems have one paired setting: FM_SM_IPC_INTERFACE (the default UNIX socket path) and the NVLSM-side plugin_options grpc_mgr --grpc_server_address must point at the same path. Change one and not the other and the two processes never shake hands, so the fabric never comes up.

5. Initialization Semantics: Three Shapes of "It Will Not Start"

FM startup is not successful just because a process exists. The verdict belongs to the fabric state.

5.1 Version mismatch aborts outright

FM validates the kernel driver stack version at start. On a mismatch it does not run degraded — it aborts. So the first place to look when "the service will not start" is:

journalctl -u nvidia-fabricmanager -n 100 --no-pager
grep -i "version\|abort" /var/log/fabricmanager.log | tail

The driver branch (-<branch>) and the FM package branch must match; that is something to get right at install time.

5.2 CUDA starting too early: cudaErrorSystemNotReady

If CUDA work launches before FM finishes initialisation, applications do not get "device busy" — they get cudaErrorSystemNotReady. The semantics are precise: the fabric is not ready. It should not be filed as a hardware ticket; go back to the startup order instead.

5.3 H100 and later: GPUs must register with the fabric

From H100 onward NVLink is trained in hardware by ALI, and each GPU must actively register with the fabric. A GPU that fails to register loses P2P. The acceptance command is:

nvidia-smi -q -i 0 | grep -i -A2 Fabric
# Expect State: Completed
#        Status: Success

Run it per GPU. If any card is not Completed / Success, stop discussing performance baselines — the P2P path on that card was never established.

6. The Correct Restart Order (H100 and Later)

This is the one procedure in FM operations where the wrong order kills jobs. NVIDIA's sequence:

# 1. Stop all CUDA applications and GPU-related services
#    (nvidia-persistenced may stay; DCGM must be stopped)
systemctl stop nvidia-dcgm

# 2. Stop FM
systemctl stop nvidia-fabricmanager

# 3. Reset the GPUs
nvidia-smi -r

# 4. Start FM and wait for the fabric
systemctl start nvidia-fabricmanager
nvidia-smi -q -i 0 | grep -i -A2 Fabric

# 5. Restore the services you stopped

# 6. Run CUDA work again

What happens if you reverse it: run nvidia-smi -r before stopping FM and FM still holds fabric state while registrations are torn down; or start CUDA before the fabric is ready and collect a pile of cudaErrorSystemNotReady.

One more piece of guidance from the documentation: FM does not need frequent restarts. It is a steady-state service; legitimate triggers are driver upgrades, configuration changes and recovery from the fatal errors below — not "it feels off, let me bounce it".

7. Failure Semantics: GPUs Report Xid, NVSwitches Report SXid

The two codes must be read separately:

• GPU side is Xid, the standard error code space from the NVIDIA driver;
• NVSwitch side is SXid, formatted like this:

nvidia-nvswitch0: SXid (PCI:0000:0a:00.0): 12008, Fatal, Link 12

A line carries four facts: which switch, the PCI address, the value, fatal or non-fatal, and which link.

7.1 Non-fatal SXid: bookkeeping only, jobs survive

A non-fatal SXid only raises a notice and does not abort running work. It belongs in monitoring and trend comparison — a link that emits recurring non-fatal errors is a candidate for replacement in the next maintenance window, before it turns fatal.

7.2 Fatal SXid on a GPU ↔ NVSwitch port: that card's jobs die

When a fatal SXid lands on a GPU-to-NVSwitch port, FM kills the jobs on that GPU, and may drag along Xid 74 / Xid 45. Recovery:

nvidia-smi -r          # reset first
# If the reset does not fix it -> reboot the node

7.3 Fatal SXid on an inter-baseboard trunk port: every CUDA job dies

This is the heaviest tier. When a fatal SXid appears on a trunk port between baseboards, FM kills all CUDA jobs on the machine — part of the fabric path is no longer trustworthy, and continuing to run only buys silent data corruption. The documented recovery sequence is:

systemctl stop nvidia-fabricmanager        # stop FM
# stop every application and service using the GPUs
nvidia-smi -r                              # note: no -i / -id
systemctl start nvidia-fabricmanager       # start FM
# Still broken -> reboot / power cycle

nvidia-smi -r without -i/-id is deliberate: a trunk-level fault is not a single-card problem, and resetting one card will not rebuild the fabric.

7.4 Fourth-generation platforms: SXid no longer applies

On B200/B300 this SXid reporting mechanism no longer applies. Error observation moves to DCGM reading port and ASIC counters through NVSDM. The operations stack has to follow: the old dmesg-scraping habit finds nothing on the new platform, so NVSwitch- and port-level telemetry must be wired into DCGM instead.

8. High-Availability Degraded Modes: An A100-Era Legacy

FM exposes three degradation switches that decide whether a broken link means "exit" or "keep going degraded":

Config item	Scope	Value semantics
ACCESS_LINK_FAILURE_MODE	GPU ↔ NVSwitch access link failure	0 = remove the failed GPU from the fabric (keeping it for non-P2P use) or exit FM; 1 = degrade and continue
TRUNK_LINK_FAILURE_MODE	Inter-baseboard trunk failure	Same
NVSWITCH_FAILURE_MODE	Whole NVSwitch failure	1 = disable that switch and its peer, bandwidth drops to 5/6

These three only work on the A100 generation and are ignored on H100 and later. So when an A100 cluster migrates to H100, every degraded parameter tuned in the name of "avoid downtime" silently stops mattering — not because the config was lost, but because the new platform no longer takes that path. Put this in the migration review.

9. MIG Interaction and Permission Boundaries

9.1 MIG versus NVLink/P2P

Enabling MIG disables (or bypasses) GPU NVLink and thereby loses P2P. Generations differ visibly:

• H100 and later: NVLink stays active for the duration of MIG;
• A100: after MIG is turned off, P2P only returns if FM is running.

So on A100, "MIG is off but P2P still fails" — check whether FM is alive before suspecting the card.

9.2 Running FM as non-root

FM supports non-root operation, but these paths and devices must be granted explicitly:

directories: /var/run/nvidia-fabricmanager, /var/log/, /usr/share/nvidia/nvswitch
proc:        /proc/driver/nvidia-nvlink/capabilities/fabric-mgmt
devices:     /dev/nvidia-nvlink, /dev/nvidia-nvswitchctl,
             /dev/nvidia-nvswitchX, /dev/nvidiactl, /dev/nvidiaX

Fourth-generation systems (B200 and so on) additionally need /dev/infiniband/umadX, /dev/infiniband/issmX and read access to /sys/class/infiniband/mlx5_X/device/vpd. The annoyance is that /proc entries are recreated after a driver reload or reboot, and the permissions disappear with them. Those grants therefore have to be automated with udev rules, otherwise every driver upgrade requires manual repair.

One more boundary: using infiniband tools on ConnectX ports is not supported — do not point ibstat and friends at NVSwitch-side problems.

Acceptance Checklist

1. systemctl is-enabled nvidia-fabricmanager and is-active both green, and on HGX machines confirm you were the one who enabled it;
2. nvidia-smi -q -i N | grep -A2 Fabric reports Completed / Success on every GPU;
3. FABRIC_MODE matches the deployment shape (bare metal/passthrough = 0, multitenancy = 1/2), and any change was followed by a restart;
4. on generation four, FM_SM_IPC_INTERFACE and the NVLSM grpc_server_address point at the same path;
5. LOG_LEVEL=4 and a non-zero LOG_MAX_ROTATE_COUNT, so logs rotate before the disk fills;
6. STATE_FILE_NAME moved off /tmp to persistent storage (multitenancy);
7. after every driver upgrade, run the restart-order script and confirm udev rules restored the non-root permissions;
8. monitoring is wired to SXid (generation three and earlier) or to DCGM's NVSwitch port counters (generation four).

Three Closing Takeaways

• FM is not an optional monitoring add-on, it is the fabric's initializer: without it, more GPUs only means more point-to-point links; treat cudaErrorSystemNotReady as a startup-order problem, not a hardware failure;
• There is exactly one correct restart order: stop CUDA and GPU services → stop FM → nvidia-smi -r → start FM → restore services. Wrong order means dead jobs;
• SXid fatality sets the blast radius: port-level affects one GPU's jobs, trunk-level makes FM kill every CUDA job, and generation four replaces the whole observation channel with DCGM plus NVSDM.

Further Reading

• 10K-GPU IB Compute Fabric Design: Six Steps to a Topology
• IB Admin Guide
• UFM Install
• DCGM Metrics and GPU Health Checks: From Four Diagnostic Levels to Prometheus Telemetry