GPU Profiling: Timeline Triage with nsys and Kernel-Level Attribution with ncu

In one sentence: a benchmark tells you how far off you are, telemetry tells you whether the node is healthy, and only a profiler tells you where the gap comes from. Profiling itself is a two-stage pipeline: nsys finds out which kernel deserves attention on the timeline, then ncu takes that kernel apart to explain why. Run it in the other order and you will pick whichever kernel looks slowest, then spend two days on an operator that was always 2% of the runtime.

1. Three Tool Classes, Three Different Questions

In the same "training got slow" situation, three classes of tool answer completely different questions:

Class	Examples	Question it answers	Typical output
Benchmarks	HPL, nccl-tests, STREAM, IOR	Does this hardware / this fabric hit its line	An efficiency percentage, a bandwidth curve
Telemetry	DCGM, dcgm-exporter	Is anything wrong at runtime (throttling, Xid, replays)	Time series, alerts
Profilers	Nsight Systems, Nsight Compute	Why is this span / this kernel slow	Timeline, metric structure, stall attribution

The boundaries are hard: a benchmark short of target does not tell you why, and a clean dashboard does not mean there is no bottleneck. A node runs HPL at 62% efficiency with DCGM all green, and the only thing that can say whether it is memory-bound or wasting SMs on a bad launch config is a profiler.

The site already has a dedicated piece on DCGM telemetry; this article covers the profiling half and how it connects to the other two.

2. nsys: First Answer "Where Did the Time Go"

Nsight Systems does whole-timeline sampling: CUDA API calls, kernel execution, memory copies, NVTX markers, OS runtime calls, plus CPU sampling and thread scheduling on the host. That is what the defaults turn on:

# Default: trace cuda / opengl / nvtx / osrt, plus CPU IP sampling and thread scheduling
nsys profile ./train_step.sh

The defaults are fine for a first look, but they cost you two things: data volume and a hard limit on capture length. The documentation states plainly that Nsight Systems does not officially support runs longer than five minutes. So real usage is the opposite of the default — narrow first, then capture.

Three Ways to Narrow

Layer one: drop trace domains you do not need.

nsys profile --trace=cuda,nvtx -d 20 \
  --sample=none --cpuctxsw=none \
  -o step20 ./train_step.sh

Useful domains include cuda, nvtx, osrt, cublas, cudnn, mpi, nccl, ucx and gds. Turning off CPU sampling (--sample=none) and context switches (--cpuctxsw=none) cuts the report size by an order of magnitude — at the cost of no longer seeing who the CPU was waiting for.

Layer two: capture only the region you care about. Two standard approaches:

# CUDA's own switch, wrapped around the hot section in code
cudaProfilerStart();
  ... hot loop ...
cudaProfilerStop();

# Or have an NVTX range trigger the capture, so only that range is recorded
nsys profile -c nvtx -w true -p train_step@training ./train_step.sh

For a training job the natural unit is one step: range_push("step") at the top, range_pop() at the bottom, and capture steps 20–40 only, so warmup, data loading and checkpoints do not ride along.

Layer three: start late. -y/--delay lets the process run for a while before collection begins, skipping initialisation.

What to Read First

# Emit summary statistics right after collection (it also builds an SQLite file, which can take minutes on large captures)
nsys profile --stats=true -o step20 ./train_step.sh

# Or generate reports afterwards, on demand
nsys stats --report cuda_gpu_kern_sum --report cuda_api_sum step20.nsys-rep

The default report set already includes cuda_gpu_kern_sum (GPU time aggregated per kernel), cuda_api_sum (time per API), and memory-operation time and size summaries. These three answer the first round: is the GPU actually busy, which kernels eat the time, and is there a blocking call in the API stream.

Three places on the timeline matter most:

1. Gaps between kernels: long blanks mean the host is stalling (synchronisation, Python overhead, data loading), not that the kernels are slow;
2. Whether compute overlaps communication: only -t nccl together with CUDA shows whether the all-reduce runs in the gaps or blocks compute outright;
3. memcpy serialised against kernels: H2D/D2H copies cutting the pipeline in half are the clearest evidence of "the GPU is waiting for data".

One multi-node trap: if every rank enables GPU metric collection they trample each other. The documented pattern is one collector per node, on local rank 0:

#!/bin/bash
# Use with srun / mpirun
if [ $SLURM_LOCALID -eq 0 ]; then
  nsys profile --gpu-metrics-devices=all "$@"
else
  nsys profile "$@"
fi

3. ncu: Taking a Single Kernel Apart

nsys produces the shortlist; ncu explains why that kernel only reaches 30%. Its unit of collection is a single kernel launch, and it replays the kernel to gather metrics in groups.

# Default basic set, writing a report file
ncu -o attn.ncu-rep python bench_attn.py

# All sections (slower, but you see everything at once)
ncu --set full -o attn.ncu-rep python bench_attn.py

With no --set, --section or --metrics, you get the basic set: a small number of high-level utilisation metrics plus launch and occupancy data, which is static information — no replay needed, so it costs almost nothing. --set full expands to everything. The documented section list and the question each one answers:

Section	Question it answers
SpeedOfLight	Compute and memory throughput as a percentage of peak — which one is closer to the limit
ComputeWorkloadAnalysis	Utilisation of each SM pipeline (Tensor included) and achieved IPC
MemoryWorkloadAnalysis / _Tables	Real throughput, hit rates and sectors-per-request at each cache level and in device memory
Occupancy	Gap between resident warps and the theoretical maximum, and which resource limits it
SchedulerStats	Eligible warps per cycle and how many issue slots go unused
WarpStateStats	Which stalls warps hit, and the cycle share of each
InstructionStats	SASS instruction mix — is everything pinned on a few pipelines
LaunchStats	Whether grid/block configuration and resource demand make sense
WorkloadDistribution	Whether SM, cache and DRAM load is balanced

These identifiers are what --section accepts, and --list-sections prints the live list — section sets change across releases, so check before hard-coding them in a script.

Replay: Where ncu's Cost Comes From

A kernel needs dozens of hardware counters, and only a limited number can be collected in one pass, so the tool replays the kernel several times. NVIDIA's own quickstart example prints 46 passes for a toy kernel. Three consequences you must know:

• Runtime is inflated: a 200 µs kernel may take seconds with all sections — never capture at this granularity across a long training run;
• Caches are flushed by default: L1/L2 are cleared before each replay pass, which makes results reproducible but also loses the "warm cache" behaviour of the real run. Use --cache-control none when you want realism;
• Clocks are locked by default: --clock-control with base, boost or none decides whether you measure at rated or boost frequency. Locking makes runs comparable, but the measured duration is not the production duration.

That is exactly why "ncu numbers as a benchmark" is wrong: it measures an execution the tool has reshaped. Use it for attribution, not for acceptance.

Capture Only What You Need

# Profile three kernels starting from the sixth launch
ncu -s 5 -c 3 --set full python bench.py

# Filter by name (regex)
ncu -k regex:attention --set full python bench.py

# Only kernels inside an NVTX range
ncu --nvtx --nvtx-include "train_step@training" python train.py

# Only kernels launched from one Python function
ncu --call-stack-type python --python-include train.py@forward python train.py

On output: the default details page groups results by section with rule hints; --page raw prints every collected metric; --page source correlates to SASS and source. For sharing, -o into a .ncu-rep is the stable path, and --import reopens it later for filtering.

4. Two Reasons ncu Refuses to Start

Permission: ERR_NVGPUCTRPERM

On Linux from driver 418.43 onward, GPU performance counters are admin-only by default. A normal user gets:

ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device.

Three legitimate routes, chosen by driver generation:

• Temporary: sudo ncu ..., or grant the user CAP_SYS_ADMIN (from R565, CAP_PERFMON also works);
• R610+ drivers: switch to the capability nodes — profiler-context (intra-context profiling), profiler-device (includes device scope) and trace-device (hardware tracing), each backed by /dev/nvidia-caps/nvidia-cap<minor>. Granting a group access is a chmod g+r on that node, which is finer-grained than opening everything;
• Older drivers / regkey method: drop a .conf under /etc/modprobe.d containing options nvidia NVreg_RestrictProfilingToAdminUsers=0, then rebuild the initrd or reboot. Check /proc/driver/nvidia/params and read RmProfilingAdminOnly (0 means opened up).

Inside containers there is a second layer: the host must enable access first, and the container must start with --cap-add=SYS_ADMIN. Neither alone is enough.

Contention: Someone Else Holds the Counters

The second failure mode is not permission but the counters already being in use. DCGM, other CUPTI clients, Nsight Graphics and a second ncu instance all hold that resource, and the symptom is "I have permission and still collect nothing".

The documented advice is specific: when DCGM is suspected, pause it.

dcgmi profile --pause     # let DCGM release the counters during profiling
# ... ncu capture ...
dcgmi profile --resume

This connects directly to the previous DCGM article: telemetry and profiling share the same hardware counters, so they must be separated in time. When you do performance attribution on a cluster, remember that the monitoring agent is the invisible competitor.

5. Reading SpeedOfLight in the Right Order

Do not start at the first table in the report. The order is fixed:

Step one: which of the two percentages is higher. SpeedOfLight reports compute and memory throughput as a share of peak, which tells you compute-bound or memory-bound first.

Step two: do not read "high" as "good". A percentage means the unit is busy, not usefully busy. 90% DRAM throughput against 15% compute throughput means a lot of data moved and very little math done — either the operator has low arithmetic intensity, or the access pattern uses only part of each cache line.

Step three: compare theoretical and achieved occupancy. sm__maximum_warps_per_active_cycle_pct is the ceiling; resident warps are the reality. A large gap is usually not the vague conclusion "occupancy is too low" but wave quantisation (grid size not a multiple of SM count) or imbalanced work.

Step four: only look at stalls when issue slots are genuinely idle. The documentation is explicit: if SchedulerStats shows eligible warps every cycle, more stalls will not slow the kernel down, and selected / not_selected are scheduler states, not real stalls. The optimisation target is issue-slot utilisation, not zeroing every stall counter.

Step five: take the stall down to source. Warp stall samples are attributed to the consumer instruction that waits, not the producer that creates the dependency. The smsp__pcsamp_warps_issue_stalled_* family tells you the category only; the line number comes from --page source against SASS.

6. Three Typical Conclusion Paths

Symptom	Most likely conclusion	What to inspect next	Usual fix
Low compute throughput, DRAM/L2 near peak	Memory-bound	Sectors per request, hit rates and bank conflicts in MemoryWorkloadAnalysis	Coalesce accesses, raise reuse, change layout, use shared memory and async copies
Neither compute nor memory high, low occupancy, long-scoreboard stalls dominant	Latency-bound (dependency chains too long)	Resource limiters in Occupancy, register/shared-memory use in LaunchStats	Raise parallelism, shorten dependency chains, prefetch, cut register pressure
Barrier/wait stalls high, or wide gaps between kernels	Synchronisation and serialisation	WarpStateStats, launch gaps on the timeline	Reduce divergence, shrink blocks, fuse kernels, move synchronisation off the hot path

All three need one caveat: a stall name is not a conclusion. High barrier in a tensor workflow may simply be the expected synchronisation phase, and high drain is often the tail effect at the end of a kernel. The deciding evidence is always whether issue utilisation actually dropped.

7. Practical Limits of Multi-GPU, Multi-Node Profiling

On one node, profiling is a precision instrument. At scale the approach has to change:

• nsys only covers the local node. Cross-node questions need the NCCL trace (-t nccl), and the hard numbers on the communication side still come from nccl-tests and IB counters;
• ncu serialises. It replays a single kernel and owns the counters, which limits its value for communication-heavy kernels and breaks overlap while it runs;
• multi-process needs a target. --target-processes all follows children, but when a training script forks a pile of workers the practical approach is to filter by rank and capture one per node;
• too many kernels means sampling. Thousands of launches per step: sort by GPU time with nsys first, run ncu on the top three, and remember the other 97% may add up to less than 10% of the time.

Set expectations on cost: one --set full on a single kernel is typically seconds to minutes depending on pass count, and a 20-second nsys capture produces reports from tens to hundreds of MB. Profiling is a sampling activity, not a resident one — the resident layer is DCGM.

8. Turning Profiling into a Fixed Workflow

You should not have to re-derive the method every time. Fix the order:

1. Benchmark short of target / training slowed down: run 20 seconds of nsys (NVTX around one step), take the cuda_gpu_kern_sum ranking;
2. Take the top three kernels: one ncu --set full each, save the .ncu-rep;
3. Read in the section-5 order: compute vs memory first, then theoretical vs achieved occupancy, and stalls only last;
4. Change one thing and re-measure the same kernel: one variable at a time, then check whether that percentage moved;
5. Regress against the benchmark: after the change, confirm with HPL, nccl-tests or end-to-end step time — ncu numbers are never the acceptance criterion;
6. Archive: .nsys-rep, .ncu-rep, driver and CUDA versions, the exact command and the clock setting used, all in one place.

Step 6 is the one people skip, and the price is that nobody can reproduce that 30% difference two weeks later.

9. Boundaries and Common Misreads

• Profiling changes behaviour: flushed caches, locked clocks and replay all make the measured code behave differently from production. Attribution is trustworthy; absolute numbers are not;
• Overhead grows with metric count: more sections and more kernels mean more passes, which can become unacceptable in long runs;
• Software-patched metrics are the most expensive: instruction-level source metrics rewrite the kernel and replay it, so use them only after you have localised the instruction;
• Sampling has platform limits: the PM sampling context-switch trace is not supported on MIG, MPS with multiple clients, or WSL, and containers need a recent driver;
• Versions move: section names, metric names and default sets change across releases, so anything hard-coded in a script needs a --list-sections / --query-metrics self-check.

Three Closing Takeaways

• The order cannot be reversed: nsys decides who to look at, ncu decides why — going straight to ncu means drawing a lottery ticket among ten thousand launches;
• A percentage means the unit is busy, not that it is useful: 90% DRAM throughput can be efficient, or evidence of low arithmetic intensity; the conclusion has to reach source and SASS;
• Profiler numbers do not replace benchmarks: they explain where the benchmark gap comes from, and they measure an execution the tool reshaped — treating them as acceptance criteria will optimise in the wrong direction.

Further Reading

• NCCL Performance Testing & Tuning Guide
• NVIDIA GPU HPL Benchmark Guide (Docker)
• DCGM Metrics and GPU Health Checks: From Four Diagnostic Levels to Prometheus Telemetry
• Operator and Runtime Accounting: Roofline, Fusion and FlashAttention IO
• GPU Server Fault Diagnosis Handbook: XID Codes, ECC and Replacement Decisions