GPU Cluster Observability: DCGM Monitoring and Alert Design

Note: This is a review article (no first-hand experimental data), based on public documentation and common production practices.

Experiment Objective

Once a cluster grows from 8 to 512 GPUs, troubleshooting shifts from "checking nvidia-smi card by card" to "finding which card is loafing." The goal of this article is a three-layer closed loop for GPU cluster observability: metric collection (DCGM) → storage and query (Prometheus) → alerting and capacity watermarks. After reading you can answer: where do per-card bandwidth, power, SM utilization, and temperature come from uniformly? Which metrics must have alerts, and which are just noise? How do you judge whether the cluster still has room for the next training job?

Code Walkthrough

Part 1: The DCGM metric exporter

# docker-compose snippet: expose DCGM metrics to Prometheus
services:
  dcgm-exporter:
    image: nvcr.io/nvidia/k8s-dcgm-exporter:3.3.5
    command: ["-f", "/etc/dcgm-exporter/dcp-metrics-included.csv"]
    volumes:
      - ./dcp-metrics.csv:/etc/dcgm-exporter/dcp-metrics-included.csv
    environment:
      - DCGM_EXPORTER_COLLECT_INTERVAL=5000   # 5s collection period
    ports:
      - "9400:9400"

Key points:

1. -f selects a metric whitelist CSV: dcgm-exporter emits hundreds of metrics by default; a CSV whitelist keeps only what you need and reduces Prometheus storage pressure. Each line is # HELP <name> <unit>; <type> or <name>, <type>, <help>, <unit>.
2. COLLECT_INTERVAL=5000: GPU metrics default to 10s; 5s is enough to catch power/bandwidth jitter during training without blowing up the TSDB (about 20-30 series per card per 5s).
3. Port 9400: the standard exporter port that Prometheus scrapes via scrape_configs.

Part 2: Prometheus scraping and core alert rules

# prometheus.yml snippet
scrape_configs:
  - job_name: 'dcgm'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['gpu-node-01:9400', 'gpu-node-02:9400']
        labels: { cluster: 'train-prod' }

# Alert rules (the 4 that matter)
groups:
  - name: gpu
    rules:
      - alert: GPUMemoryECCErrorRate
        expr: rate(dcgm_memory_correctable_errors_total[5m]) > 0.5
        for: 5m
        labels: { severity: warning }
      - alert: GPUClockThrottle
        expr: dcgm_clocks_throttle_reasons_active{reason!="GpuIdle"} == 1
        for: 10m
        labels: { severity: warning }
      - alert: GPUHighTemperature
        expr: dcgm_temperature_gpu > 90
        for: 10m
        labels: { severity: warning }
      - alert: GPUUtilizationAnomaly
        expr: avg(dcgm_sm_utilization) by (instance) < 30
        and on(instance) (time() - process_start_time_seconds{job="dcgm"} > 300)
        for: 15m
        labels: { severity: info }

Explanations:

• dcgm_memory_correctable_errors_total: the HBM correctable-ECC counter. Sustained growth signals memory-cell degradation — a precursor to a failing card. Use rate(...[5m]), not the raw counter.
• dcgm_clocks_throttle_reasons_active: a bitmask of downclock reasons (Power/Thermal/Software/Sync/HwSlowdown). If SM utilization is high but throughput is flat, one of these bits is set. Filter out GpuIdle, or idle machines alert all day.
• dcgm_temperature_gpu > 90: H-class cards throttle near 95°C; alerting at 90 leaves headroom.
• SM utilization anomaly: during training, utilization below 30% for over 5 minutes means the script is stuck on I/O or communication — not that the GPU is slow.

Principles

What DCGM is

DCGM (Data Center GPU Manager) turns the "look at one card manually" ability of nvidia-smi into programmatic batch collection. Metrics fall into five classes — device basics (model/serial/PCIe topology), utilization (SM/memory/encoder/decoder), power and thermal (instant power, temperature, fans), memory and ECC (used/total, correctable/uncorrectable errors), and clocks and throttling (current frequencies, throttle-reason bitmap). dcgm-exporter is just its Prometheus adapter.

Collection pipeline and data model

GPU (DCGM library) → dcgm-exporter (one per node, :9400)
     → Prometheus (scrape 5s, retain 15d)
     → Grafana (dashboards) + Alertmanager (alerts)

Each metric carries labels such as instance (node), gpu (card index), and UUID, so Prometheus supports two-dimensional drill-down by node × card: dcgm_sm_utilization{instance="gpu-node-03"} for one card, avg(...) by (cluster) for the cluster watermark.

How to compute capacity watermarks

The endgame of observability is capacity management: deciding whether a 64-GPU job can be scheduled immediately. Build three watermarks — total memory, total SM, total bandwidth:

available_mem = Σ cards where dcgm_mem_used < threshold × card capacity
schedulable  = job_peak_mem ≤ idle_mem && job_bw_demand ≤ idle_bw_estimate

Bandwidth is the hardest watermark: there is no "idle bandwidth" metric, so use memory utilization (dcgm_fb_used) and SM utilization as proxies — high SM + high memory means a card is training at full load; low SM + high memory means the card is "filled but not computing" (e.g. a big model loaded but not serving). These cards are the easiest to misjudge as idle.

Results

Typical baselines and alert thresholds for a 16-GPU node (estimated, for reference):

Metric	Normal baseline	Alert threshold	Meaning
SM utilization (training)	85%-98%	< 30% for 15m	card not computing; check I/O/comm
Memory utilization	70%-90%	> 97%	near OOM; check for leaks
GPU temperature	55-75°C	> 90°C	liquid/air cooling anomaly
Correctable ECC	occasional < 0.1/s	rate > 0.5/s	memory degradation precursor
Power	near power wall during training	< 30% wall in training	card limited/downclocked

Engineering Takeaways

1. Fewer alerts is better: five core rules (ECC rate, throttle, temperature, SM anomaly, memory watermark) cover 90% of failures. Do not start with 50 rules and create alert fatigue.
2. Observability is the input to tuning: the numerator of MFU is compute; the denominator's "actual SM utilization" comes from DCGM — without monitoring, MFU optimization is empty talk.
3. Wire it to the scheduler: a "bad card" (high ECC / persistent throttle) should be auto-marked excluded so SLURM/K8s stop scheduling onto it. This is the last link of the cluster health loop.
4. Storage matters too: GPU I/O waits usually appear as low SM utilization plus busy storage/network, so storage monitoring (next article's topic) should sit on the same Grafana board for comparison.

Next article preview: this one built "compute-side observability"; the other half of training-interruption recovery lives in storage — "Checkpoint Storage and Fast Recovery: A Two-Tier Design from Local NVMe to Parallel File System."