GPU Training Network Troubleshooting: Packet Loss, Congestion & NCCL Timeouts

Experiment Goal

Thousand-GPU training (Chapter 11) depends on the network far more than traditional HPC — AllReduce/All-to-All run constantly over IB/RoCE, and a network tremor stalls training. But between "training is slow" and "network fault" lies fog: slowness could be network, parallelism config, or storage. This article dissects three high-frequency network faults with a symptom → diagnosis → root cause → fix structure: packet loss & PFC storms, congestion & hash polarization, NCCL timeouts & link degradation. Commands are reproducible operations; scenarios and fixes come from cluster ops experience without fabricating environment numbers.

Fault 1: Packet Loss & PFC Storms (RoCE Scenario)

Symptom: training throughput dips periodically; ethtool -S counters show growing rx_dropped/rx_crc_errors; NCCL logs show timeout; flow-control counters anomalous.

Diagnosis:

# NIC side: check drops & PFC counts (RoCE)
ethtool -S <nic> | grep -iE "drop|pfc|pause|crc" | head -20
# Switch side: buffer/drops & PFC counts
# (switch CLI, per vendor) show counters drop / pfc
# NCCL logs: locate the timed-out link and peer

Root causes & fixes (by probability):

1. PFC deadlock/storm: one congested link → PFC backpressure propagates → peers stop sending → throughput collapse. Fix: confirm PFC enabled only on lossless classes with correct queue mapping; investigate hash polarization causing single-link overload (see Fault 2);
2. Insufficient buffers: small switch buffers + large bursts drop packets. Fix: tune buffer/ECN thresholds per Chapter 03 RoCE architecture;
3. Poor link quality: optics/cables causing CRC growth. Fix: replace cable, check optical power (ethtool -m or switch optical readings), compare with the Chapter 01 delivery baseline.

Engineering observation: among RoCE loss faults, "PFC storm + hash polarization" is the most common combo — looking only at drops sends you in the wrong direction. First distinguish "drop-then-retransmit" from "PFC-backpressure-stops-sending"; the two symptoms call for completely different fixes.

Fault 2: Congestion & Hash Polarization

Symptom: aggregate bandwidth is fine, but NCCL tests (Chapter 03) show low bandwidth/high latency on some links; throughput collapses under many-to-one traffic (AllReduce convergence).

Root causes & fixes:

1. Hash polarization: flow hashing squeezes multiple large flows onto one physical link — the link overloads while neighbors idle. Fix: enable Adaptive Routing (AR) or equivalent dynamic load balancing (Chapter 03 RoCE architecture); without AR, use multi-path hashing + large-flow splitting;
2. Many-to-one congestion: AllReduce convergence traffic congests at the switch. Fix: rail topology (same-numbered GPUs nearby, Chapter 03), ECN marking + NIC-side response, traffic shaping;
3. Topology asymmetry: some links have longer paths/lower bandwidth. Fix: verify fiber connections and SU grouping against the topology map (Chapter 02 rack layout), correct miswiring.

Diagnosis commands: run Chapter 03 NCCL tests (nccl-tests or a custom script printing per-link rates for all_reduce); below baseline locates the link; cross-check switch show interface for port utilization distribution.

Fault 3: NCCL Timeouts & Link Degradation

Symptom: NCCL reports timeout / connect timed out / retries exceeded; training interrupts or hangs.

Root causes & fixes:

1. Link degradation: IB/RoCE link dropped from x4 to x2/x1 (optics/cable/port issues). Fix: check link rate & negotiation with ibstat/iblinkinfo (IB) or ethtool (RoCE); confirm no degradation vs the delivery baseline;
2. NCCL parameter mismatch: NCCL_SOCKET_IFNAME, NCCL_IB_DEVICES select wrong NICs/devices → wrong path. Fix: verify against the Chapter 03 NCCL tuning parameter baseline;
3. Topology awareness off: NCCL_TOPO_FILE unset or wrong → non-optimal communication paths. Fix: generate the correct topology file; verify path selection in NCCL_DEBUG=INFO logs;
4. Firmware version drift: some nodes on old firmware → compatibility issues. Fix: align with the Chapter 01 firmware baseline fleet-wide.

Decision Tree

Training slow/stalling
├─ drops in ethtool/NCCL logs?
│   ├─ yes → PFC counters anomalous? → Fault 1 (PFC storm/buffers)
│   │        └─ no → CRC/link quality → replace cable/check optics
├─ NCCL test link bandwidth imbalanced?
│   ├─ yes → hash polarization / topology asymmetry → Fault 2 (AR/rail/topology check)
├─ NCCL timeout/connect failures?
│   ├─ yes → link degradation / param mismatch / topology file → Fault 3
└─ all network metrics normal → investigate parallelism config (Ch 09) & storage (Ch 04)

Network Troubleshooting Checklist

• [ ] Drops classified first as "retransmit type" vs "PFC-backpressure type" before fixing
• [ ] Hash polarization confirmed (link utilization distribution vs flow distribution); AR/dynamic balancing ready
• [ ] Many-to-one congestion mitigated with rail topology + ECN + shaping
• [ ] NCCL link rate/negotiation checked against delivery baseline; no degradation
• [ ] NCCL params (IFNAME/IB_DEVICES/TOPO) verified against baseline; DEBUG logs confirm paths
• [ ] Firmware versions consistent fleet-wide (Chapter 01 baseline)

Engineering Takeaways

• Classify the symptom before fixing: loss/congestion/timeout are three fault classes with different fixes — locate the class first, then act.
• Baselines are the anchor of troubleshooting: the Chapter 03 NCCL baseline and Chapter 01 firmware baseline are the only references for "did the network get worse".
• Network troubleshooting links to training observability: read NCCL logs (timeouts) and network counters (drops/PFC) together — only then you neither wrongfully blame the network nor miss it.

Next article preview: with the network healthy, the inference side keeps evolving — the next article, Speculative Decoding: Principles, Engineering & Benefit Boundaries, dissects "using a small model to accelerate a large one".