GPU Server Fault Diagnosis: XID Codes, ECC and the Replace-or-Not Decision

In one sentence: when a GPU throws an error, the reflex is to reboot. The right move is to classify the XID first — application fault, driver fault, or hardware fault — and only then decide between restarting the job, draining the node, replacing the card, or filing an RMA. Getting this wrong cuts both ways: pulling a healthy card because an application bug tripped Xid 13 costs money, and rebooting a node whose HBM is quietly degrading costs weeks.

This article covers exactly one thing: getting from a symptom to a verdict. Every conclusion should land on one of three sentences — restart that application, reset that GPU, or start an RMA.

1. What an XID actually is: a driver report, not a hardware verdict

An XID is an error report the NVIDIA driver writes to the kernel log (/var/log/messages or syslog) when it detects a GPU anomaly. The format is fixed:

NVRM: GPU at 0000:03:00: GPU-b850f46d-d5ea-c752-ddf3-c4453e44d3f7
NVRM: Xid (0000:03:00): 48, pid=7062, ...

The preceding line carries the GUID and PCI address, which is what binds every later Xid line to the same card. That is the entry point for triage: aggregate by PCI address first, then read the sequence of XIDs on that one card.

NVIDIA's own wording is deliberately cautious: Xid messages indicate a general GPU error, "most often due to the driver programming the GPU incorrectly or to corruption of the commands sent to the GPU", and can point to a hardware problem, an NVIDIA software problem, or a user application problem. In other words, the number is a classification starting point, not a conclusion. In theory, corruption in the frame buffer or on PCIe can fabricate almost any Xid.

So triage must split classes first, then conclude.

2. Step one: three questions that settle most tickets

Do not reboot first. Answer three questions:

Question	What it tells you
Did only one process die, or every process on that GPU?	One process → suspect the application; all processes → driver or hardware
Is it reproducible (same job always fails) or random?	Reproducible → application or a specific kernel; random → hardware or link
Does the failure disappear on a different GPU?	Disappears → hardware located; follows the job → application or framework

Three questions usually classify 90% of tickets. The remaining 10% is what the error codes are actually for.

3. Triaging the high-frequency XIDs by root cause

Grouped by ownership rather than by number — which is how the table is actually useful during an incident.

3.1 Application class: suspect code before touching hardware

XID	Official meaning	How to read it
13	Graphics Engine Exception	Typically an out-of-bounds access, but also an illegal instruction or register. Compute Sanitizer memcheck or CUDA-GDB will point at the offending line
31	GPU memory page fault (FIFO: MMU Error)	MMU reports an illegal address access; overwhelmingly an application bug, occasionally driver or hardware
43	Reset Channel Verif Error	The application hit a software-induced fault and had to terminate. The GPU is still healthy; NVIDIA states this is usually not a driver bug but a user application error
45	Preemptive Channel Removal	Informational companion to a job being torn down by Ctrl-C, kill or GPU reset — usually no action. If it appears alone on A100, check whether Fabric Manager runs on the same host and exited abnormally

The key judgement for this group is "is the GPU still healthy". Xids 43 and 45 are explicitly not hardware problems; Xids 13 and 31 must be disproved with a tool before the hardware path is opened.

3.2 Memory class: 48 / 63 / 64 / 92 / 94 / 95 / 140

This is the family that gets misjudged most often, and the one that must be read as numbers rather than prose. Three concepts first:

• SBE (single-bit error): correctable by ECC, not fatal, but repeated at the same location triggers page retirement or row remapping;
• DBE (double-bit error): uncorrectable — that is Xid 48;
• contained / uncontained: from A100 onward, error containment limits an uncorrectable error to the application that hit it (94) or lets it affect every application on the GPU (95).

XID	Meaning	What to do
48	Double Bit ECC Error	If followed by 63 or 64: drain the node, let jobs finish, reset the GPU. If not followed by 63/64: run field diagnostics, then evaluate against RMA thresholds
63	Row-remapping / page retirement recorded successfully	With 48 → drain and reset. Alone (from an SBE) → keep running, reset at a convenient time
64	Row-remapping / page retirement recording failed	A hard signal: on A100 and later, reboot the node immediately; on legacy cards, reset the GPU now. If it repeats, drain and escalate
92	High single-bit ECC error rate	Run field diagnostics; decide RMA from the error-rate trend against thresholds
94	Contained ECC error	Only the affected application must restart; other workloads continue. Reset the GPU when convenient so remapping takes effect
95	Uncontained ECC error	Uncorrectable and not contained: with MIG enabled, drain other instances then reset; with MIG disabled, reboot the node immediately
140	ECC Unrecovered Error	The driver cannot even mark pages unusable or remap them. Reset; if it persists, contact the vendor
171 / 172	Refinements of 48: DRAM error / SRAM error	Decides whether the case follows the DRAM RMA policy or the SRAM one (see section 5)

One line to remember the priority: 94 hurts only itself, 95 hurts the whole GPU, 64 means the record itself failed and you act now, 140 means even the remedy failed.

3.3 Link and bus class: 74 / 79 / 32 / 110

XID	Meaning	How to read it
74	NVLink Error	A link problem — but the fault may sit at the remote end of the link. A failed GPU makes its neighbours report 74 as well. DWORD decoding is mandatory (section 4)
79	GPU has fallen off the bus	The driver can no longer reach the GPU over PCIe. Often a PCIe link hardware failure bringing the link down. Check system event logs and kernel PCIe events first, then drain and escalate
32	PBDMA Error	The DMA controller managing the PCIe communication stream reported a fault; primarily a PCI link quality problem, and explicitly "generally not caused by user application actions"
110	Security fault error	Should not occur absent a hardware failure. Revert recent hardware changes and cold reset; if it persists, contact the vendor

74 and 79 demand the most on-site discipline: 79 usually puts a card on the replacement path, while 74 requires checking the far end first or you replace the wrong card.

3.4 Platform class: 61 / 62 / 119 / 120 / 121

XID	Meaning	Action
61	PMU Breakpoint	Report and reset the GPU
62	PMU Halt error	Report and reset the GPU
119 / 120	GSP RPC Timeout / GSP Error	Reset first; if it persists, power-cycle the node; if it still recurs, follow the GPU Debug Guidelines
121	C2C Link corrected error (Grace)	Already corrected by the system, no operational impact; reset the GPU in a service window to retrain the link

The common thread for 61/62/119/120 is "the GPU's internal co-processor stopped answering". Software can only reset and power-cycle; repetition is the hardware signal.

4. Decoding XID 74 at the bit level

Xid 74 carries a series of hex DWORDs at the end of the line, like (0x12345678, ...). Take the first DWORD and count bits from the least significant:

Bits set	Meaning	On-site action
bit 4 or bit 5	Likely a hardware ECC/parity issue	More than 2 occurrences on the same link → report it
bit 21 or bit 22	Marginal channel signal integrity	Check mechanical connections, re-seat cables and boards
bits 8, 9, 12, 16, 17, 24, 28	Possibly hardware	Check connections and re-seat; if it persists, run diagnostics

The value of bit-level decoding is that it separates "re-seat" from "replace". Without it, operators loop through re-seating forever because "74 showed up again".

Pair it with nvidia-smi nvlink -e to see per-link CRC and replay counters, which tells you whether this is sustained degradation or a one-off event.

5. ECC thresholds: the actual replace-or-not rules

This section is the most bookmarkable part of the article, because "should we replace it" should not be a feeling.

5.1 Row remapping: three failure conditions (A100 / H100 and later)

Row remapping replaces degraded cells with spare rows inside HBM, up to 512 remappings per GPU. The row-remapping failure flag is set by any of:

1. A remapping attempt for an uncorrectable error on a bank that already has 8 uncorrectable rows remapped;
2. A remapping attempt on a row that was already remapped — this can happen with far fewer than eight remaps in that bank;
3. 512 total remappings for uncorrectable errors.

Once the failure flag is set and validated by NVIDIA Field Diagnostic, the RMA criteria are met.

5.2 SRAM (L2) RMA criteria

SRAM is protected by ECC or parity, is dense, and a low background level of uncorrectable errors from cosmic radiation is expected and does not indicate faulty hardware. The thresholds are:

• parity-protected SRAM: more than 4 UCE unique-count events within an address bank;
• SECDED ECC-protected SRAM: more than 2 within an address bank.

5.3 Page retirement (Volta and earlier)

Older GPUs retire pages dynamically, capped at 64 pages of 64 KiB each:

• 60 pages qualifies for RMA;
• 64 pages fails Field Diagnostic outright;
• Special case: 15+ pages retired and still growing at ≥1 page per week can be evaluated before reaching 60.

5.4 The commands that produce the numbers

# Memory errors and remapping state (look at Remapped Rows / Pending / Failure)
nvidia-smi -q -d ECC,PAGE_RETIREMENT,ROW_REMAPPER

# Compact per-GPU view
nvidia-smi --query-gpu=index,pci.bus_id,ecc.errors.uncorrected.volatile.total,retired_pages.pending,remapped_rows.pending,remapped_rows.failure --format=csv

One distinction to keep straight: remapped_rows reports the number of entries written to the InfoROM, not the number of remaps completed in hardware — the remap only takes effect after a GPU reset. Seeing pending does not mean it is fixed, and a non-zero count does not mean the card is finished.

6. The replace-or-not matrix

Evidence	Verdict	Action
13/31 only, and memcheck points at a specific line	Application bug	Send it back to the user; do not touch hardware
43/45 alone, health check clean	Normal companion to job teardown	Log it and close the ticket
48 → (63 or 64)	Uncorrectable memory error, recorded	Drain, wait for jobs to finish, reset; keep watching
64 recurring (recording failure)	InfoROM write path broken	Reboot/reset immediately; escalate to RMA if it continues
94	Containment succeeded	Restart the application only; reset in a service window
95	Containment failed	Drain and reboot the node now; preserve evidence for RMA
140	Even remapping failed	Reset; recur → RMA
row-remapping failure flag = true (8 rows / repeated row / 512 total)	DRAM RMA criteria met	Run Field Diagnostic, open the RMA
SRAM UCE over threshold (parity > 4, SECDED > 2)	SRAM RMA criteria met	Same as above
74 with bit 4/5 set, ≥2 times on the same link	NVLink hardware degradation	Escalate; stop re-seating
74 with bit 21/22 set	Marginal signal integrity	Check mechanical connections, re-seat
79	GPU fallen off the bus	Drain, collect PCIe AER logs, RMA
92 with a rising single-bit rate	HBM degradation trend	Field diagnostics; evaluate against the trend
61/62/119/120 recurring after reset	Co-processor fault	One cold boot; still recurring → RMA
121 (Grace C2C)	Corrected, no impact	Reset the link in a service window

7. Evidence collection: gather once, so the vendor does not come back for more

Most rejected RMA requests fail on incomplete evidence. This script collects a complete bundle in one pass:

#!/usr/bin/env bash
OUT=/tmp/gpu-evidence-$(hostname)-$(date +%F-%H%M)
mkdir -p "$OUT"

# 1. XID timeline (aggregate first, read the sequence, not a single line)
grep -h "NVRM: Xid" /var/log/messages* 2>/dev/null | tail -n 200 > "$OUT/xid.log"
grep -h "NVRM: GPU at" /var/log/messages* 2>/dev/null | tail -n 50 > "$OUT/gpu-map.log"

# 2. Memory and remapping state
nvidia-smi -q -d ECC,PAGE_RETIREMENT,ROW_REMAPPER > "$OUT/ecc-rowremap.txt"
nvidia-smi -q > "$OUT/nvidia-smi-full.txt"
nvidia-smi -q -d TEMPERATURE,POWER,CLOCK,PERFORMANCE > "$OUT/thermal-power.txt"

# 3. PCIe link and AER (key evidence for bus drops and down-negotiation)
lspci -vvv 2>/dev/null | grep -A3 -i "nvidia\|3d controller" > "$OUT/lspci.txt"
dmesg | grep -iE "pcie|aer|link.*down|corrected|fatal" | tail -n 200 > "$OUT/pcie-aer.log"

# 4. NVLink counters
nvidia-smi nvlink -s > "$OUT/nvlink-status.txt" 2>/dev/null
nvidia-smi nvlink -e > "$OUT/nvlink-errors.txt" 2>/dev/null
nvidia-smi nvlink -c > "$OUT/nvlink-capabilities.txt" 2>/dev/null

# 5. Active short diagnostic
dcgmi diag -r 1 > "$OUT/dcgm-diag-short.txt" 2>&1

# 6. Official bundle (kernel logs plus driver internal state)
nvidia-bug-report.sh --output-file "$OUT/nvidia-bug-report.log.gz"

tar czf "$OUT.tar.gz" -C /tmp "$(basename $OUT)"
echo "collected: $OUT.tar.gz"

Three collection rules:

1. Preserve before resetting. A reset clears volatile counters — capture first;
2. Keep the timeline. A single Xid means nothing; the ordering in tail -n 200 is the evidence;
3. Short -r 1, long -r 3. dcgmi diag -r 1 takes seconds to a minute and fits a pre-job prologue; -r 3 takes roughly 30 minutes and is what you run before shipping a card back.

8. Post-replacement verification: skipping this means you did not fix it

After swapping a card, a cable or a switch port, you must leave proof behind, or the same class of failure returns wearing a different mask.

1. Node health: dcgmi diag -r 2 clean, XID count zero;
2. Memory clean: nvidia-smi -q -d ECC,ROW_REMAPPER shows zero DBE/volatile counts and pending = false;
3. Links at spec: nvidia-smi nvlink -s shows the expected per-link rate, and lspci -vvv LnkSta shows the intended width and speed (x16 / 32GT/s), not a down-negotiated x8;
4. Collectives back to baseline: compare all_reduce_perf busbw before and after — the delta should be inside noise;
5. Archive the record: card SN, slot, XID sequence and verification data, all stored. That record is what turns the next identical incident into a 20-second lookup.

9. Three misjudgements that cost real money

1. Treating 43/45 as hardware faults. They are companions to an application being torn down, and 45 on A100 may only mean Fabric Manager exited abnormally on the same host. Replacing a card for this is pure waste.
2. Confusing 94 with 95. 94 needs an application restart; 95 needs a node reboot. Get it backwards and you either reboot a whole node for nothing or let a corrupted GPU keep serving jobs.
3. Reading the XID without the far end. Most wasted NVLink replacements come from skipping the peer. When one GPU drops a link, its neighbours and NVSwitches log errors too; bit decoding plus nvlink -e counters is how you identify the real culprit.

Three closing lines

• Split the class before reading the code: three questions (scope, reproducibility, does another GPU fix it) close most tickets.
• Memory is judged by numbers, links by bits: ECC by row-remapping and page-retirement thresholds, NVLink by DWORD bits to separate re-seat from replace.
• Preserve evidence before resetting: an incomplete bundle gets bounced back and forces a second reproduction.