DCGM Metrics and GPU Health Checks: From Four Diagnostic Levels to Prometheus Telemetry

In one sentence: you cannot accept a GPU node just because one training run did not crash — DCGM offers three different things: an active diagnostic that answers "can this machine take work right now" within 2.5 seconds, a passive health watch that does not disturb jobs, and an exporter that turns GPU metrics into Prometheus time series. Blur those roles and you get the familiar outcome: beautiful dashboards, and failures that still blow up.

1. What DCGM Is, and What It Explicitly Is Not

DCGM (Data Center GPU Manager) is a lightweight user-space library and agent that gives a GPU host a shared management plane: GPU behaviour monitoring, configuration management, policy enforcement, health and diagnostics, accounting and process statistics, plus NVSwitch configuration and monitoring. It runs in two modes:

• Standalone: an nv-hostengine daemon is resident and dcgmi connects to it as a client;
• Embedded: a third-party agent embeds it in its own process — dcgm-exporter's container mode is exactly this.

The host engine is not a command dispatcher. It owns the entity namespace, field watches, retained samples, reusable groups, module state and job records, which is why it is long-lived: several clients can consume the same collection stream.

One version fact that matters: since v2.x, DCGM no longer bundles Fabric Manager. NVSwitch systems need FM installed separately; DCGM only observes the fabric — it does not configure it and does not manage the FM lifecycle.

The documentation also draws a hard boundary — DCGM Diagnostics are not designed to:

1. provide comprehensive hardware diagnostics;
2. actively fix problems;
3. replace the vendor's field diagnostics tools;
4. participate in any RMA process.

Those four points fix its place in the workflow: it is a filter, not a judge. "This node has a problem" means isolate and escalate; "passed" does not mean the card is healthy forever.

2. Entities and Fields: Read IDs from the Running Engine

DCGM's world is made of entities and fields. An entity is not only a GPU — it may be a MIG instance, an NVSwitch, an NVLink port, an NVIDIA CPU and its cores, or a ConnectX adapter.

# Inventory what this host engine currently manages
$ dcgmi discovery --list

# Align with asset and scheduler systems via GPU UUID / PCI bus ID
$ dcgmi discovery --info a --gpuid 0

# Read the parent-child hierarchy on MIG systems
$ dcgmi discovery --compute-hierarchy

One rule to memorise: entity IDs are local to this running host engine and are not stable identifiers you can copy across machines. Tickets and CMDB entries belong to the GPU UUID or PCI bus ID.

A field is one observable of the hardware, in one of four flavours: inventory value, gauge, counter, event. A watch tells the host engine which entity-field pairs to collect, how often, and how much history to retain:

$ dcgmi dmon --list                 # the field catalogue of this DCGM release
$ dcgmi dmon --field-id 150,155 --count 5   # temperature (150) and power (155), five samples

dmon creates temporary watches and prints retained samples for all GPUs by default. Reading a sample usually does not re-query the hardware; the timestamp is when DCGM collected it, which matters when correlating across metrics.

Also: a field being in the catalogue does not mean every GPU supports it. N/A in the output can mean unsupported, unavailable, not permissioned, or not yet sampled — do not read it as a hardware failure.

3. Four Diagnostic Levels: Time Budget Decides the Choice

dcgmi diag has four run levels intended for different situations, and the documentation states the runtimes plainly (measured on Hopper systems):

Level	Name	Time budget
-r 1	Quick / System Validation	< 2.5 seconds
-r 2	Medium / Extended System Validation	4 GPUs < 2.5 min; 8 GPUs < 10.5 min
-r 3	Long / System HW Diagnostics	4 GPUs < 10 min; 8 GPUs < 35 min
-r 4	Extended	4 GPUs < 45 min; 8 GPUs < 2.25 hours

Higher levels include everything beneath them. The plugin-to-level mapping (same official table):

Plugin	r1	r2	r3	r4
software	✓	✓	✓	✓
pcie (includes NVLink)		✓	✓	✓
memory		✓	✓	✓
memory_bandwidth		✓	✓	✓
diagnostic			✓	✓
targeted_stress			✓	✓
targeted_power			✓	✓
nvbandwidth			✓	✓
nccl_tests			✓	✓
memtest				✓
pulse (Input EDPp)				✓

The table dictates the selection strategy:

• r1 is the gate. It finishes in 2.5 seconds, so it belongs in the job prologue — run it before every allocation and the cost is negligible;
• r2 is the post-mortem after a failed job. The job died for no obvious reason and the GPUs are idle: sweep memory and PCIe/NVLink;
• r3 is administrator isolation. This is where real stress begins (diagnostic, targeted stress, targeted power, NVLink bandwidth, NCCL), so the node must be out of the scheduler or at least drained;
• r4 is the long run before escalation. memtest and pulse only appear here, and 45 minutes to over two hours means a maintenance window, not daily routine.

4. Parameters and Thresholds: Defaults Mean "It Runs", Not "You Should Use This"

# Run a level against a specific entity
$ dcgmi diag -r 2 --entity-id gpu:0 --json

# Run only named plugins (comma separated)
$ dcgmi diag -r "pcie,diagnostic"

# Override one plugin parameter
$ dcgmi diag -r 3 -p diagnostic.test_duration=300.0

High-value parameters and their defaults:

Parameter	Default	Meaning
diagnostic.test_duration	180.0	Seconds of sustained stress
diagnostic.matrix_dim	2048.0 (8192.0 from Blackwell)	Starting GEMM dimension
diagnostic.gflops_tolerance_pcnt	0.0	Fractional shortfall from the tested-GPU mean that fails; 0 disables the comparison
pcie.max_pcie_replays	80.0	PCIe replays allowed per GPU over the plugin duration
pcie.max_latency	100000	Microseconds; exceeding it fails the subtest
pcie.min_pci_width	1.0	Write 16.0 for x16
memory.minimum_allocation_percentage	75.0	Share of framebuffer to allocate for the memory test

Three practical conclusions:

1. gflops_tolerance_pcnt is a cross-GPU comparison, not an absolute performance promise. It compares against the mean of the GPUs being tested, so it catches one lagging card in an eight-GPU node; on a single-GPU run it has no meaningful input;
2. Not setting min_pci_width means not checking it. The default 1.0 lets an x16 card degraded to x1 pass. To catch wrong slot population or link down-training, set 16.0 explicitly;
3. temperature_max has changed behaviour. Older releases used it as the temperature ceiling during the test; newer releases (DCGM 4.6) monitor against the GPU's reported maximum operating temperature and keep the parameter for compatibility — do not use it to impose a site-specific low threshold, do that in external monitoring.

One more worth calling out: --fail-early makes targeted_power, targeted_stress and diagnostic check for failures while the test runs, once every 5 seconds by default (--check-interval, 1–300 s), instead of only at the end. On a 10-minute-plus r3 run, early failure saves a lot of wall-clock time.

5. Exit Codes and Logs: Automation Must Read Both

dcgmi exit codes are the unsigned 8-bit representation of dcgmReturn_t (negative values wrapped positive), and signals appear as 128 + the signal number. The ones you will meet:

Value	Symbol	Meaning
0	DCGM_ST_OK	Success
255	DCGM_ST_BADPARAM	Bad parameter (note -1 also maps to 255)
226	DCGM_ST_NVVS_ERROR	The diagnostic executed successfully but reported an error — the critical distinction
217	DCGM_ST_DIAG_ALREADY_RUNNING	A diagnostic is already running
237	DCGM_ST_RESET_REQUIRED	The GPU requires a reset
227	DCGM_ST_REQUIRES_ROOT	Host engine runs unprivileged and cannot perform this operation

226 is the most frequently misread: it does not mean "the command failed to run", it means "it ran, and the result was a failure". Reading only the exit code loses which card and which plugin failed, so automation should consume -j JSON alongside it and persist test names, per-entity results and error codes.

Logs default to /var/log/nvidia-dcgm/nvvs.log. For post-mortems that file plus the JSON result is worth far more than an exit code — the documentation also expects you to retain the command, target selection, full output, debug log and plugin statistics when escalating.

6. Multi-Node Diagnostics: A Much Higher Bar

dcgmi mndiag runs an MPI application called mnubergemm on all participating nodes simultaneously, exercising cross-node interconnect, memory and compute. It catches what single-node diagnostics cannot (cross-node bandwidth shortfalls, NCCL initialisation failures).

# Launch from one node; the host list includes itself
$ dcgmi mndiag --hostList "node1;node2;node3"

# Custom duration (default 3600 seconds)
$ dcgmi mndiag --hostList "node1;node2" -p "mnubergemm.time_to_run=600"

# JSON output
$ dcgmi mndiag --hostList "node1;node2" -j

It comes with a stack of hard prerequisites, and missing any one stops it:

• DCGM 4.3.0 and newer, and currently only GB200 NVL / GB300 NVL are supported;
• OpenMPI on every node, and the same NVIDIA driver version everywhere;
• Passwordless SSH between nodes with the private key stored unencrypted on disk — ssh-agent does not help, because the SSH sessions are spawned by the nv-hostengine process and never inherit your shell environment;
• an nv-hostengine running on each node;
• IMEX channels configured first on multi-node NVLink systems, otherwise NCCL initialisation fails outright with unhandled system error;
• a runtime limit of one hour plus 60 seconds.

The last two are what to remember: most mndiag failures are not hardware at all, they are unconfigured IMEX or an encrypted SSH key.

7. Passive Health Monitoring: The Layer That Leaves Jobs Alone

Active diagnostics occupy the GPU, so they cannot run continuously. Day-to-day you rely on health watches, which evaluate telemetry collected during ordinary operation without imposing any load:

$ dcgmi group --create fabric-adapters --add cx:0
$ dcgmi health --set x --group <group-id>
$ dcgmi health --check --group <group-id>

Some semantics to respect:

• dcgmi health --set a --group s — s is the built-in "all NVSwitches" group, and a is the full watch selector that includes the NVSwitch checks;
• watches need time to accumulate samples, so a check immediately after enabling may have nothing to report;
• diagnostics and health monitoring are two separate bodies of evidence and cannot substitute for each other: a node can be clean in passive monitoring yet fail an active diagnostic, and it can carry a historical health warning after later active tests all pass. Read both to get the full picture of the node's current state.

8. dcgm-exporter: Wiring Metrics into Prometheus

dcgm-exporter is a Go exporter that exposes GPU metrics at /metrics for Prometheus-style systems:

$ docker run -d --rm --gpus all --net host --cap-add SYS_ADMIN \
    nvcr.io/nvidia/k8s/dcgm-exporter:<tag>-ubuntu20.04 \
    -f /etc/dcgm-exporter/dcp-metrics-included.csv

$ curl localhost:9400/metrics

Defaults and the changes you are likely to make:

Item	Default	Note
Listen address	:9400	-a to change
Collection interval	30000 ms	-c to change; 30 s is coarse for triage
Collector list	/etc/dcgm-exporter/default-counters.csv	-f for a custom CSV
Connect to existing host engine	disabled	-r localhost:5555 (mandatory on DGX-class systems that already ship DCGM)
Device selection	f (all GPUs, or all instances under MIG)	-d g:0,1 / -d i:0,2-4

One hard compatibility rule: the exporter image ships its own libdcgm.so, which talks a private protocol to the host nv-hostengine, so the DCGM version the exporter is built on must be greater than or equal to the host DCGM version. Compare nv-hostengine --version against the image tag.

Key metrics and their operational meaning:

Metric	Meaning	When to look
DCGM_FI_DEV_GPU_TEMP / DCGM_FI_DEV_MEMORY_TEMP	Core / memory temperature	Cooling problems, clock throttling
DCGM_FI_DEV_POWER_USAGE	Board power	Compare against TDP to spot a power cap
DCGM_FI_DEV_SM_CLOCK	SM clock	Compare against spec to catch throttling
DCGM_FI_DEV_PCIE_REPLAY_COUNTER	PCIe replay count	One of the earliest link-quality signals
DCGM_FI_DEV_XID_ERRORS	Most recent Xid value	Correlate with kernel logs; the entry point to GPU faults
DCGM_FI_DEV_FB_FREE / FB_USED	Framebuffer free/used	Leaks, fragmentation, whether isolation is real
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL	Aggregate NVLink bandwidth	Whether the P2P path is working
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE	Tensor pipe activity (0–1)	Whether compute is actually being fed
DCGM_FI_PROF_DRAM_ACTIVE	DRAM activity (0–1)	Whether the bottleneck is compute or memory
DCGM_FI_PROF_PCIE_TX_BYTES / RX_BYTES	PCIe bytes out/in	Whether the data-loading path is the bottleneck

The two 0–1 PROF metrics (Tensor active, DRAM active) are what expose "fake high utilisation": 95% GPU utilisation with Tensor activity at 0.3 means the math pipelines are not saturated and the bottleneck is data or synchronisation.

Under MIG, metrics carry GPU_I_PROFILE / GPU_I_ID labels and resolution drops to the GPU-instance level; do not aggregate by GPU there, or several instances stack onto one line.

One security boundary: the host engine's TCP protocol has no authentication, authorisation or encryption, so port 5555 belongs on a trusted management network only.

9. How the Three Layers Fit: A Decision Table

Situation	Layer	Concrete action
Before an allocation, decide in 2 seconds	Active diagnostic r1	dcgmi diag -r 1 --entity-id gpu:N
A job died, GPUs are idle	Active diagnostic r2	-r 2 --json, persist the failing entity and error code
A subsystem is suspected (PCIe / memory / comms)	Named plugins	-r "pcie,diagnostic", with --iterations
Day-to-day operation	Passive health watch	health --set plus periodic --check
Cluster-level baseline and acceptance	Multi-node diagnostics	mndiag --hostList, after confirming IMEX and SSH
Long-term trends, capacity, alerting	exporter + Prometheus	Intervals below 30 s; watch PCIe replays, Xid, throttling
Evidence collection before RMA	r3/r4 with JSON and logs	Retain the command, output, nvvs.log, plugin stats

Acceptance Checklist

1. dcgmi discovery --list entity count matches the physical GPUs, and MIG systems show a correct hierarchy;
2. dcgmi diag -r 1 passes on all GPUs in under 3 seconds, and is wired into the prologue or the delivery pipeline;
3. dcgmi diag -r 2 -j has been run on at least one canary node, and the JSON parses into test/entity/error codes;
4. key plugin parameters are set explicitly per site policy (especially pcie.min_pci_width=16.0) instead of left at defaults;
5. dcgmi health --set a is enabled and --check output carries severity levels;
6. the dcgm-exporter version is ≥ the host DCGM version, -r points at the host's 5555, and :9400/metrics scrapes;
7. Prometheus has alerts and dashboards on DCGM_FI_DEV_PCIE_REPLAY_COUNTER, DCGM_FI_DEV_XID_ERRORS and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE;
8. nvvs.log has a rotation policy, so evidence still exists when you need it.

Three Closing Takeaways

• The three layers are not interchangeable: diag is active evidence gathering (drain or pull the card), health is passive observation (no load), exporter is long-term time series — substituting one for another always leaves a blind spot;
• Defaults mean "it runs", not "you should use this": min_pci_width=1.0 and gflops_tolerance_pcnt=0.0 effectively disable checks, so the site must write its own policy;
• Exit code 226 and JSON must be read together: "diagnostic ran but judged a failure" is exactly the case pipelines pass off as success.

Further Reading

• NCCL Performance Testing & Tuning Guide
• GPU HPL Benchmark
• Acceptance & Performance Baselines
• NVSwitch Fabric Manager Operations: Configuration, Restart Order and SXid Triage