CUDA Software Stack at Scale: Pinning Driver, CUDA, cuDNN and NCCL

Driver, CUDA, cuDNN, NCCL — each off by a minor version and training results and performance can differ. The 384-GPU L40S cluster approach: pin the version matrix first, install in bulk, then verify item by item. This article records the process.

1. Version Matrix: Lock Once, Reuse Everywhere

Component	What to pin	Why
NVIDIA driver	A version matching kernel/distro	Wrong version = module load failure or black screen
CUDA	Major version (e.g., 12.x)	Training frameworks depend on it
cuDNN	Version matching CUDA	Inference/convolution operator performance
NCCL	Version matching CUDA/driver	Collective correctness and performance

Judgment: the matrix is not "pick the latest" — it is a combination verified as a set. Validate on one node first (training passes + performance meets the bar), then freeze it as the cluster standard, so 48 nodes don't each trip over the same holes.

2. Bulk Install: Internal Mirror + Unified Script

• Driver/CUDA installers live on the internal mirror; all nodes pull from it (identical versions, no public-internet dependency);
• Fixed install order: driver → CUDA → cuDNN → NCCL → environment variables;
• Environment (PATH/LD_LIBRARY_PATH) rolls out via /etc/profile.d/ — no node-level .bashrc edits.

Pitfalls:

1. Uninstall the old driver before installing (nvidia-uninstall / rpm -e); residue causes module load conflicts;
2. Install CUDA toolkit and driver separately; business containers can depend on the toolkit alone;
3. The NCCL version may differ from the one bundled in PyTorch — frameworks use their own bundled NCCL, so standardizing the framework install cluster-wide matters more than tuning NCCL in isolation.

3. NCCL Key Parameters: Cluster-Wide, Not Per-App

Parameter	Suggested start (measure)	Effect
NCCL_P2P_LEVEL	PHB to start	PCIe-domain P2P on no-NVLink cards
NCCL_NET_GDR_LEVEL	PXB to start	GPUDirect RDMA path
NCCL_IB_DISABLE	0	Confirm IB, not Ethernet fallback
NCCL_SOCKET_IFNAME	mgmt NIC name	Control/metadata channel on the mgmt plane

These go into a cluster-wide unified environment (Ansible), not hardcoded in applications — changing environment does not require changing code.

4. Verification Chain: From Driver to Collectives

nvidia-smi                    # driver OK, 8 GPUs visible, power/temp normal
nvcc --version                # CUDA toolkit version
python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda)"
# Per-node NCCL self-check
mpirun -np 8 /opt/nccl-tests/build/all_reduce_perf -b 128M -e 128M -f 2 -g 1
# Cross-node (2 nodes): confirm GDR + IB active
# NCCL_DEBUG=INFO: look for IB/RDMA paths, not socket fallback

Acceptance: four version numbers match the matrix + single/cross-node allreduce meets the bar + NCCL_DEBUG confirms RDMA.

5. Delivery Checklist

1. Version matrix documented: driver/CUDA/cuDNN/NCCL + verified framework combination;
2. Unified internal mirror, all nodes same source;
3. Environment vars rolled out centrally, no node-level drift;
4. NCCL parameters cluster-wide (Ansible), not in-app;
5. Verification chain passes: nvidia-smi / nvcc / torch self-check / single-node / cross-node NCCL.

---

Based on field measurements from a 384-GPU L40S IB cluster delivery; exact versions and commands follow on-site validation.