Skip to content

Lustre Client Tuning: Striping Layouts, Multi-NIC LNet and I/O Parameters ​

Summary: Installing Lustre and running mount -t lustre does not mean the client can consume the cluster's bandwidth. The default layout is one stripe per file, so a single stream can only touch one OST. Default LNet uses the first NIC it discovers. Client cache and RPC concurrency parameters stay at compile-time defaults. This article covers the three client-side decisions only: how to lay out files, how to wire the network, and how to tune parameters — with a verification command for each step.

1. Three symptoms, three different layers ​

SymptomMost likely layerLook at first
A single process writing one large file tops out at one disk's speed (say 1–2 GB/s), while many processes each writing their own file is fasterStriping layoutlfs getstripe <file> and its stripe_count
Node bandwidth stays low and sar -n DEV shows traffic on one NIC onlyLNet networkinglnetctl net show, ip -br a
Large directories are slow to ls or stat while bandwidth is fineMetadata path (statahead, directory layout)lfs getdirstripe, strace -c stat
Bandwidth is fine but latency jitters and small-IO throughput is poorClient I/O parametersosc.*.max_rpcs_in_flight, max_dirty_mb

Order matters: confirm the layout, then the network, then the parameters. Parameters can buy a percentage within an existing layout and network; they cannot change the order of magnitude.

2. The three striping parameters and their documented defaults ​

A Lustre file is RAID0: data is cut into stripe_size chunks and distributed round-robin across stripe_count objects on different OSTs. The defaults (manual Table 10.1):

ParameterDefaultMeaning
stripe_count1Number of OST objects a file uses
stripe_size1 MBData written to one object before moving to the next
start_ost-1Starting OST index; -1 lets the MDT pick by space and load. The manual explicitly recommends not changing it

stripe_count=1 is the most commonly missed performance fact about Lustre: the ceiling for a single stream equals the bandwidth of one OST. If a single NVMe OST can deliver 2 GB/s, one process writing one default-layout file gets 2 GB/s no matter how many OSTs the cluster has.

Change the layout with lfs setstripe:

bash
# Create a file with 8 stripes of 4 MB each
lfs setstripe -c 8 -S 4M /lustre/big.bin

# -1 means all OSTs ("unlimited", still bounded by lod.max_stripecount and the OST count)
lfs setstripe -c -1 /lustre/wide.bin

# Set the layout on a directory; new files inherit it (the easiest team-wide policy)
lfs setstripe -c 4 -S 2M /lustre/projects/llm/outputs

-S 0 falls back to the default 1 MB. lfs getstripe is the only trustworthy check — ls -l shows nothing about layout.

3. Choosing a layout: first ask how many processes write one file ​

This is the primary question, far more important than "more stripes is better".

Write patternSuggested layoutWhy
One process writing one large file sequentially (checkpoints, model weights)Large -c (4/8/all OSTs), -S 1–4 MBA single stream needs multiple stripes to aggregate bandwidth
N processes each writing their own file (MPI ranks writing per-rank output)Keep -c 1Each file is already a stream; extra stripes only fragment metadata and small IO
Multiple processes writing one shared large file (parallel I/O library)-c at least the process count, -S aligned to each process's IO sizePuts each process's writes on a different OST and avoids hotspots
Huge numbers of small files (datasets, logs)-c 1, focus on directory and metadata insteadStriping gives small files nothing and adds locks and objects

Two things people get backwards:

  • Large striping on small files is a regression. A 200 KB file with -c 8 still creates eight objects on eight OSTs, each holding tens of KB, doubling metadata work.
  • More stripes is not better. Beyond the number of concurrent streams, extra objects only add lock contention and failure surface; the manual explicitly notes that striping across every OST is not always desirable.

4. Four hard limits and one hidden switch ​

LimitValueWhat happens
Maximum objects per file2000The file cannot grow further
Maximum object size (ldiskfs)16 TiBThat object stops growing when the OST fills
Maximum file sizeldiskfs 31.25 PiB, ZFS 8 EiBBounded by object count and object size
ldiskfs without ea_inode, stripes per file160 OSTs-c -1 is effectively truncated to 160

The hidden switch is lod.*.max_stripecount. When users ask for -c -1 on a cluster with many OSTs (say 200), an administrator can cap the effective stripe count so one file does not fan out across the entire cluster:

bash
# Persist on the MGS; applies to all MDTs of the file system
lctl set_param -P lod.$FSNAME-MDT0000-mdtlov.max_stripecount=32
lctl get_param lod.$FSNAME-MDT0000-mdtlov.max_stripecount

One server behaviour that shows up as "some OSTs stopped taking new files": the MDT reserves 0.1% of space and 32 inodes per OST, stops allocating new objects to that OST below that level, and resumes when free space reaches twice the reservation and free inodes exceed 64. An OST showing "5% left" in lfs df may therefore already be out of allocation.

5. Directory defaults and PFL: users should not need to know striping ​

Telling every user to set stripes per file does not scale. Two approaches avoid the training problem:

Directory default layout: run lfs setstripe on a directory and new files inherit it. Existing files are unaffected — migrate those with lfs migrate.

Progressive layout (PFL): a single file is split by offset into components, small stripes near the front and wide stripes later. The manual's canonical example puts the first component on the MDT (DoM) and spreads the rest across all OSTs:

bash
lfs setstripe -E 1M -L mdt -E -1 -S 4M -c -1 /lustre/models/ckpt

The first 1 MB lives on the MDT (fast for small files and header reads), and everything after 1 MB is striped at 4 MB across every OST. It suits output files whose final size is unknown.

DoM has three limits worth knowing: a single DoM component is at most 1 GB (lfs setstripe's compile-time limit) and must be 64 KB aligned; the server-side lod.*MDT*.dom_stripesize defaults to 1 MB and silently truncates larger requests; setting it to 0 disables DoM on that MDT.

6. LNet: two NICs do not run in parallel just because they exist ​

Lustre network identities are NIDs of the form address@LNet-name, for example 10.13.24.90@o2ib1. The default behaviour is to use only the first NIC discovered — that is exactly why a dual-port IB node saturates only one port.

Configuration goes in /etc/modprobe.d/lustre.conf (legacy style) or through lnetctl:

bash
# Declare two NICs belonging to o2ib0 and o2ib1
options lnet networks="o2ib0(ib0),o2ib1(ib1)"

# Or add them at runtime
lnetctl net add --net o2ib1 --if ib1
lnetctl net add --net o2ib0 --if ib0 --peer_timeout 180 --peer_credits 8
lnetctl net show -v

Concepts that matter:

  • peer_credits / credits bound in-flight messages. Defaults are credits=256 and peer_credits=8; on high-bandwidth links a small peer_credits limits throughput to a single peer. Raise it with the peer's memory budget in mind, not on one side only.
  • max_intf defaults to 200, capping interfaces discoverable per node.
  • LNet routing is not IP routing. Crossing LNets needs explicit routes and a gateway; an ip route entry does not make Lustre use it.
  • Both servers and clients need MUNGE, and lnet configuration must match on clients, or pings will not get through.

Verification is plain: run a multi-stream job and watch sar -n DEV 1 to see both NICs near line rate, then confirm both nets are up in lnetctl net show -v.

7. Client I/O parameters: direction matters more than numbers ​

Client tunables fall into two groups, both about whether one stream can saturate the link:

Parameter groupEffectCost of raising it
osc.*.max_dirty_mbDirty page cache allowed per OST, i.e. write burst capacityMemory use and the data-loss window on failure
osc.*.max_rpcs_in_flightConcurrent RPCs per OSTServer pressure, client memory
osc.*.max_pages_per_rpcPages per RPC, i.e. transfer sizeLarge RPCs hurt small IO
llite.*.max_read_ahead_mb and friendsReadahead windowGreat for sequential reads, wasted on random reads
ldlm.namespaces.*.lru_sizeCached inodes and locks per clientMetadata hit rate versus memory

Compile-time defaults differ widely across versions and vendor patches, so never fill these from memory. Read the live values first:

bash
lctl get_param osc.*.max_dirty_mb
lctl get_param osc.*.max_rpcs_in_flight
lctl get_param llite.*.max_read_ahead_mb
lctl get_param ldlm.namespaces.*-MDT*.lru_size

There are two ways to set them; do not mix them up:

bash
# Temporary: this node, this mount only; lost on reboot or remount
lctl set_param osc.*.max_dirty_mb=512

# Permanent: must be run on the MGS of that file system; pushed to all nodes by upcall
lctl set_param -P osc.$FSNAME-OST*.max_dirty_mb=512
lctl set_param -P llite.$FSNAME-*.max_read_ahead_mb=32

# Inspect the persistent parameter log; delete with -P -d
lctl --device MGS llog_print params

Recommended order: try it on one client with set_param, measure the gain, then persist with -P on the MGS. A persistent parameter applies cluster-wide, and rolling it back means -P -d, which is far less convenient than a temporary value.

8. Four acceptance checks ​

CheckHowPass criteria
Layout appliedlfs getstripe <file>stripe_count / stripe_size match the intent
Single-stream bandwidth scales with stripesOne process writing two files, -c 1 versus -c 8The 8-stripe file is clearly faster (close to N× until the network saturates, not merely an even split)
Both NICs carry trafficsar -n DEV 1 during a parallel jobBoth NICs' RX/TX near line rate
Metadata no longer the bottlenecktime ls on a large directory, strace -c stat on a sampleIndividual stat calls no longer trigger extra OST RPCs (LSoM/DoM in effect)

9. Six pitfalls ​

  • Layout set on the directory only, existing files forgotten: directory layouts apply to new files; migrate the rest with lfs migrate -c N.
  • Assuming -c -1 really spans everything: it is capped by both lod.max_stripecount and the 160-OST limit without ea_inode.
  • Striping small files across many OSTs: object count is a metadata cost, not bandwidth.
  • Wide striping when each rank writes its own file: hotspots are not removed; each rank's IO gets scattered across OSTs instead.
  • set_param on one client and expecting a cluster-wide change: temporary parameters do not cross nodes or remounts; persist on the MGS.
  • Assuming LNet follows IP routes: crossing LNets needs explicit routes plus a gateway; verify with lnetctl route list.

To close: the ceiling is set by layout, whether you reach it is set by network, and parameters only fine-tune 5–20% inside that ceiling. Reverse the order and parameter tuning turns into a very long afternoon.

AI-HPC Expert · Contact: openaihpc@gmail.com