03. High-Performance Network

Chapter focus: This chapter focuses on InfiniBand and RoCE performance for AI training and HPC communication.

Core Topics

• topology and communication patterns
• NCCL benchmarking and tuning
• network reliability and troubleshooting

🧭 Reading Map

• Prerequisites: 03. Cluster Architecture — understand rail topology and oversubscription before the network implementation
• Role: Part 1 network chapter — IB/RoCE architecture, NCCL testing and tuning; it caps collective communication
• Order: pick IB vs RoCE → ops (IB admin/UFM) → performance (NCCL tests & tuning) → troubleshooting
• Next: 05. Storage Systems — the data plane: how training data feeds the cluster

Articles

Mellanox IB Operations & Tuning Guide

A comprehensive guide covering OFED installation, switch firmware upgrades, MPI/OSU benchmarking, and BER troubleshooting.

阅读文章 →

HPC-X ClusterKit IB Network Testing Guide

A practical guide to using NVIDIA HPC-X ClusterKit for InfiniBand bandwidth and latency validation.

阅读文章 →

InfiniBand Performance Tuning for AI Workloads

Production-grade guide to optimizing InfiniBand for distributed AI training, covering NCCL tuning, GPUDirect RDMA, NUMA affinity, and real benchmarking methodology.

阅读文章 →

Linux Dual-NIC NAT Gateway Guide

How to configure a Dual-NIC Linux node as a NAT Gateway to allow internal nodes to access the Internet. Covers iptables, Firewalld, and nftables.

阅读文章 →

NCCL Performance Testing & Tuning Guide

A comprehensive guide to GPU cluster communication testing using nccl-tests, covering compilation, AllReduce benchmarks, performance baselines, and advanced tuning.

阅读文章 →

ROCE AI Fabric Architecture Guide

A deep dive into the 400G Ethernet solution for massive GPU clusters, featuring Adaptive Routing (AR), Rail-Optimized topology, and SONiC configuration.

阅读文章 →

UFM Enterprise 6.16 Bare Metal Installation

Installation guide for NVIDIA UFM Enterprise 6.16 on RHEL 7.9 bare metal, covering prerequisites, dependencies, and service startup.

阅读文章 →

GPU Training Network Troubleshooting: Packet Loss, Congestion & NCCL Timeouts

three high-frequency faults: symptom/diagnosis/root cause/fix + decision tree.

阅读文章 →

Collective Communication: From MPI to NCCL

Communicators & primitives, per-primitive traffic, MPI vs NCCL positioning, NCCL topology-aware architecture.

Read →