MLNX_OFED and VPI Adapters: A Practical Guide to the LINK_TYPE Switch

The most "taken-for-granted" trap in a 384-GPU L40S IB cluster: VPI dual-mode adapters ship in Ethernet mode; without switching LINK_TYPE, IB never comes up. This article records the full process — driver stack installation, port-mode switch, item-by-item verification — and the pitfalls.

1. Separate Three Layers First: Hardware Mode / Driver Stack / Application Verification

Layer	Tool	Role
Hardware port mode	mstconfig (MFT)	Switch ports from Ethernet to IB (LINK_TYPE=1)
Driver stack	MLNX_OFED	Kernel modules + libibverbs + tool set
Application verification	ibstat/ibv_devinfo/ibping	Confirm Active state, rate, MTU, PKey

Judgment: the three layers are independent yet interdependent — a wrong driver version and the switched mode is not recognized; an unburned mode and the right driver still cannot bring it up. The debugging order is always "hardware mode → driver → application".

2. MLNX_OFED Installation: Version Matching Is Priority One

• Pick the version: it must match the openEuler 24.03 kernel (6.6). MLNX_OFED ships openEuler-specific packages; the wrong distribution fails to compile or fails to load modules;
• Dependencies: kernel-devel must exactly match the running kernel (rpm -qa kernel-devel);
• Install: ./mlnxofedinstall --without-fw-update (firmware updates go through MFT separately — never burn firmware mid-install);
• After installation, modprobe mlx5_core and confirm ibstat sees the port (most likely in Ethernet mode at this point — next section).

3. LINK_TYPE: The "Gate" for VPI Adapters to Enter IB

ConnectX-7/6 Dx are VPI (Ethernet/InfiniBand dual-mode) and default to Ethernet. To switch to IB:

# Install MFT, then burn the port mode with mstconfig
mst start
mstconfig -d <PCI> set LINK_TYPE_P1=1 LINK_TYPE_P2=1   # 1=IB, 2=Ethernet
mstconfig -d <PCI> q                                   # confirm burned
reboot   # the mode takes effect at init — reboot is mandatory

Common traps:

1. Only P1 switched, not P2: a dual-port card switched halfway — the second port never comes up;
2. Testing without reboot: LINK_TYPE is an init-time parameter; hot changes do not apply;
3. Mode/cable mismatch: OSFP 200G DACs work in IB mode, but old firmware may identify the port as Ethernet;
4. ibstat still shows Ethernet: check mstconfig first, do not suspect the cable.

4. Verification Chain: From Port to Data Plane

# 1) Port state
ibstat                 # Port state: Active, Physical state: LinkUp, Rate: 200
# 2) Device capability
ibv_devinfo -d mlx5_0  # Port 1 state: PORT_ACTIVE, active_mtu per config
# 3) Connectivity (peer side with driver installed)
ibping -S <lid或guid>  # ping both ways, confirms the data plane
# 4) Bandwidth baseline
ib_write_bw -a -d mlx5_0   # single-port write bandwidth, near 200G line rate

Acceptance criteria: ibstat Active + ibv_devinfo PORT_ACTIVE + ibping OK + ib_write_bw meets the bar — all four passing means "this node's IB plane is usable."

5. Other Common Failure Points

Symptom	Root Cause	Fix
Port won't come up	LINK_TYPE not burned / no reboot	Reburn + reboot
Rate stuck at 100G	Cable/port negotiation	Replace cable or check breakout port ownership
RDMA apps time out	Missing GID/subnet config	Check active gids in ibv_devinfo, rescan after SM config
Module load failure	kernel-devel version mismatch	Reinstall with a matching OFED package

6. Delivery Checklist

1. kernel-devel matches the running kernel;
2. MLNX_OFED (openEuler build) installed, ibstat sees the port;
3. mstconfig burns both ports LINK_TYPE=1 and reboots;
4. The four-step chain passes: ibstat / ibv_devinfo / ibping / ib_write_bw;
5. Only after all 48 nodes pass, move to the PKey/SL configuration phase of three-plane separation.

---

Based on field measurements from a 384-GPU L40S IB cluster delivery; exact versions and commands follow on-site validation.