Skip to content

Capacity and Quota Operations: Diagnosing inode Exhaustion, the Two Watermarks, and Growth Forecasting ​

Summary: The most commonly misdiagnosed storage incident is "df -h shows 30% free, but users cannot write". The root cause is usually not space but inodes — and how inodes are provisioned differs fundamentally per filesystem: ext4 locks the ratio in at mkfs time, ZFS allocates dynamically with free space, and Lustre decides whether new objects can be allocated at all through a combination of MDT inodes and per-OST space/inode watermarks. This article chains diagnosis, reading the numbers, quota semantics and forecasting into one executable procedure, with every threshold and semantic taken from vendor documentation.

1. Separate the two kinds of ENOSPC first ​

A write returning ENOSPC only means "no usable resource", and the resource may be blocks or inodes. There are just two steps to tell them apart:

  1. df -h: are blocks still available (remember to subtract the filesystem's reserved blocks);
  2. df -i: are inodes still available. If df -h has headroom but df -i shows a IUse% near 100%, the problem is inode exhaustion.

A third step only matters on parallel filesystems: what df shows is often "the minimum creatable amount", not "can this particular target still allocate". The Lustre manual is explicit: df -i and lfs df -i show the minimum number of inodes that can be created at the current time; if the total objects available across all OSTs (accounting for default striping) is smaller than what is available on the MDTs, df -i reports a smaller number than could actually be created, and lfs df -i is what reports the actual free inodes per target.

2. Three inode provisioning models ​

FilesystemWhere inodes come fromChangeable afterwards?
ext4 / ldiskfsComputed at mkfs time from a bytes-per-inode ratio and written into the inode tableRatio and inode size are fixed after creation; growing the filesystem adds inodes proportionally
ZFS (including Lustre on ZFS)Allocated dynamically, count depends on free spaceNo reservation needed, and no "inode table exhausted" failure mode
Lustre (ldiskfs backend)MDT owns namespace inodes, OSTs own objects; each OST has space and inode watermarksOnly by adding OSTs/MDTs or adjusting reservation parameters

The ext4 constraint (from mke2fs(8)): -i bytes-per-inode computes one inode per that many bytes, and the man page states plainly that "it is not possible to change this ratio on a file system after it is created" — so this value must be chosen for the workload at build time. -N number-of-inodes overrides the computed default and is the only correct entry point when you know the volume will hold tens of millions of small files. Growing the filesystem does change the inode count: "resizing a file system changes the number of inodes to maintain this ratio" — so adding disks helps, but the amount of help is fixed by the ratio chosen at creation.

There is also -I inode-size: a power of two ≥128 bytes, not changeable after creation, and the man page notes a consequence many people miss — 128-byte inodes do not support timestamps beyond January 19, 2038, while 256 bytes and above support extended timestamps and project IDs.

Engineering consequence: a small-file-heavy workload (dataset samples, logs, unchunked checkpoint shards) built with a "big file" ratio cannot be rescued by adding capacity later. This is the one irreversible decision at build time.

Reading ZFS requires correcting an intuition: the Lustre manual states that on ZFS the number of creatable inodes is dynamic and depends on free space; the Free and Total inode counts are only estimates based on current usage, and only the Used count is real. So on ZFS, watching IUse% is meaningless — watch space watermarks and object-count quotas.

3. Lustre's two watermarks: why 5% free still refuses new files ​

The documentation is specific: the MDT reserves 0.1% of total OST space and 32 inodes for each OST. When an OST's available space is below that reservation, or it has fewer than 32 free inodes, the MDT stops allocating new objects to it; allocation resumes when available space reaches twice the reservation and free inodes exceed 64. One behaviour that often gets mistaken for a bug: clients can keep appending to existing files regardless of the allocation state.

That explains two common observations:

  • lfs df shows an OST with 5% free while new files avoid it or fail — it is already below the reservation;
  • within one job, writing to an old file works but creating a new file fails — appends never go through object allocation.

The reservation is tunable: the documented example is lctl set_param -P osp.*.reserved_mb_low=1024 (reserve 1 GB per OST). Do not use it to "reclaim space": underneath, ldiskfs reserves 5% for root by default (mke2fs -m defaults to 5%), and the manual warns that reducing the reservation below 5% causes severe performance degradation once an OST passes 95% full, because large contiguous free regions become hard to find — and that degradation may persist even after usage drops back below 95%. The manual therefore recommends not going below 5%.

4. The status column in lfs df: the earliest actionable signal ​

Most people read only Use% and ignore the trailing status letter. Two of the documented flags are directly about capacity operations:

  • S: OST/MDT is out of Space — available space is below the minimum requirement, so that target stops taking new object allocations until space recovers;
  • I: OST/MDT is out of Inodes — free inodes are below the minimum, so new object allocation stops as well.

In other words, the correct alerting trigger is not "90% full" but an S or I appearing in the status column. lfs df -i is the right command for inodes (-i is --inodes, per the documented option table), -v adds more target status, and -l (lazy) prevents the command from blocking when a target is offline.

5. Quotas: decide whether you are limiting space or inodes ​

The classic quota incident is "we thought it was capped, but only half was". Semantics differ across the three common filesystems:

Lustre: user and group quotas cover both blocks and inodes. lfs setquota takes block soft/hard limits (-b/-B) and inode soft/hard limits (-i/-I) as two independent parameter groups; exceeding the soft limit starts a grace period. Inspect with lfs quota -u <user> and read remaining grace with lfs quota -t. Setting block quota without inode quota lets small files fill the MDT — the most common half-configured quota on parallel filesystems.

Confirm flags and units against lfs setquota --help on your installation and the manual's "Configuring and Managing Quotas" section (25.3); help text differs slightly between releases.

ZFS: the difference between quota and refquota is not cosmetics but scope of accounting. Per the documentation:

  • quota: limits the space a dataset and all its descendants may consume, as a hard limit, and it "includes all space consumed by descendants, including file systems and snapshots"; setting a quota on a descendant of an already-quota'd dataset does not override but imposes an additional limit;
  • refquota: also a hard limit, but it "does not include space used by descendants, including file systems and snapshots".

This decides the triage conclusion: a user reports "quota is 10 TB yet writes fail" — if it is quota, check how much child datasets and snapshots consume (used is precisely "the value that is checked against this dataset's quota"); if it is refquota, snapshots do not count and the dataset's own data really is full.

Two more documented behaviours matter for operational cadence:

  • userquota@: enforcement may be delayed by several seconds, so a user can exceed the quota before the system notices and starts refusing writes with EDQUOT. Quotas are not a millisecond-level brake; orchestrators must keep headroom;
  • userobjquota@: limits the number of objects, the ZFS counterpart of an inode quota, and is equally often forgotten.

BeeGFS: beegfs-ctl --setquota has both --sizelimit and --inodelimit; this repository's BeeGFS deployment guide already covers enabling and initialising quotas, so it is not repeated here — only the same principle: set both dimensions.

6. Growth forecasting: two curves, not one ​

The usual forecasting failure is predicting only terabytes, and then inodes blow up while space is fine. The workable approach is a dual curve for capacity and inodes, taking whichever arrives first:

  1. Sample at a fixed time each day (say, off-peak): the IUsed from lfs df -ih (or df -i), the Used from lfs df -h, and per-tenant/project usage from lfs quota -u. The fixed time matters, otherwise you record checkpoint peaks instead of daily averages;
  2. Compute daily growth over both a 7-day and a 28-day window (Δ/days); the difference between the windows tells you "spike versus trend";
  3. Extrapolate days remaining: remaining days ≈ (available − safety margin) / daily growth, computed separately for capacity and inodes, and take the smaller one as the projected exhaustion date;
  4. Correct for peaks: a training job's checkpoint peak is typically 5–10x the daily average (the same burst rationale discussed in Storage advanced features), so the safety margin cannot be "one average day" — it must cover a full checkpoint cycle plus one failed retry;
  5. Decision rule: if inodes arrive first while space is plentiful, adding OSTs or MDTs will not fix inode density — the right move is changing the data shape (bundle small files, merge checkpoint shards). If space arrives first while inodes are plentiful, add OSTs. Decide which curve arrives first before deciding where to spend money — the single most important discipline in capacity operations.

7. Finding who ate the inodes ​

Never run a full-filesystem find in production: walking tens of millions of files is itself a metadata stress test. A workable order:

  1. Sample at the first level: du --inodes -s /fs/*/ 2>/dev/null | sort -n | tail -20 (GNU coreutils' --inodes counts entries rather than bytes) to find the largest subtrees;
  2. Drill one level at a time with the same command rather than diving straight to the bottom;
  3. Look at the time distribution: use mtime to find directories that have not been touched in six months yet still hold inodes — typically leftover experiment output;
  4. Look at shape: hundreds of thousands of small files in one directory (unchunked datasets, line-by-line logs, checkpoints split per step, experiment-tracker run directories) are the four classic sources.

On Lustre, confirm one more thing: inode pressure lives on the MDT, not the OSTs. lfs df -ih reports MDT and OSTs on separate lines (the documented example format is one line per target), and the MDT's IUse% is the real namespace watermark; OST inode counts track objects and are usually roomier.

8. Drills and stopping the bleeding ​

If quota and inode alerting has never been drilled, the first trigger tends to be the incident.

Drill (in an isolated directory): create a directory with a deliberately low per-user inode soft/hard limit; write small files until the soft limit is crossed and record what the client sees (EDQUOT or ENOSPC), how long the alert takes, and which metric the monitoring watches; continue to the hard limit and confirm writes are refused while existing files remain readable; clean up and confirm grace and quota state return to normal.

Order of mitigation (getting the order wrong amplifies the outage):

  1. Raise the hard limit first (or temporarily relax the watermark threshold) to bring the workload back from "write failures" to "writable" — do not start cleaning during the incident;
  2. Then clean: deletion also consumes metadata operations, and mass deletion at a full watermark competes with foreground traffic on the MDS/OSTs;
  3. Finally verify: check lfs quota -u to confirm usage really dropped (Lustre quota accounting is delayed — do not conclude "it did not work" right after deleting);
  4. Do not run layout-rewriting operations such as lfs_migrate near a full watermark.

9. One-page checklist ​

  • [ ] Before building the filesystem, classify the workload: for small-file-heavy use, set the inode count explicitly with mke2fs -N instead of relying on the default ratio
  • [ ] Monitoring samples both curves: df -i (local) and lfs df -ih (parallel)
  • [ ] Alerts trigger on the S/I status flags in lfs df, not on a single percentage
  • [ ] Lustre quotas set both blocks and inodes; ZFS sets quota/refquota plus userobjquota@
  • [ ] Quota accounting delay is known (ZFS userquota@ lags seconds) and orchestrators keep headroom
  • [ ] Dual-curve extrapolation for capacity and inodes; whichever arrives first decides the expansion direction
  • [ ] Safety margin covers one full checkpoint cycle plus one retry
  • [ ] A quota/inode drill in an isolated directory runs every quarter

10. Five pitfalls ​

  1. Judging capacity from df -h alone: the signature of inode exhaustion is "space left, writes fail"; df -i is mandatory.
  2. Assuming more disks fix inodes: ext4 growth adds inodes only in proportion to the ratio chosen at creation; a wrong ratio can never be recovered.
  3. Watching IUse% on ZFS: the total inode count there is a dynamic estimate and only Used is real; watch space watermarks and object-count quotas instead.
  4. Setting block quota only: block quota does nothing when small files fill the MDT; Lustre and BeeGFS both require the inode dimension too.
  5. Treating osp.*.reserved_mb_low as a way to free space: it is a protection threshold that keeps an OST from being driven into the ground; lowering it only makes the full-disk failure arrive sooner, and the 5% ldiskfs reservation is likewise documented as not to be reduced below.

AI-HPC Expert · Contact: openaihpc@gmail.com