Skip to content

Slurm Accounting and Fair-Share: Deploying slurmdbd, QOS Quotas and Fairshare ​

Summary: A cluster that runs Slurm only answers "can jobs run". Multi-tenant clusters must also answer three questions: who consumed how much, who may consume how much, and whether the queue is fair. All three live in the accounting subsystem. This article follows the deployment order: database and slurmdbd placement, configuration flags, the association tree, two-level quotas, fairshare weights, then reporting and retention. All parameter semantics follow the official Slurm documentation.

1. Without accounting, none of the three questions can be answered ​

Slurm does not collect accounting data by default. Without the AccountingStorage* parameters in slurm.conf, a finished job leaves only a line in the controller log, and sacct, sreport and sshare have no data source. Without AccountingStorageEnforce, quotas you define in the database are recorded but never enforced by the scheduler.

A multi-tenant deployment therefore needs three layers:

CapabilityQuestion it answersDepends on
Job ledgerWho consumed how many CPU-seconds, GPU-seconds, memoryAccountingStorageType plus a gather plugin
Quotas and admissionHow much a user or account may hold at onceAccountingStorageEnforce plus association/QOS limits
Fair queueingUnder-served users run firstPriorityType=priority/multifactor plus fairshare

The three layers can be enabled in stages, but enabling only the first is close to useless: you pay the database write cost and get no scheduling effect.

The three-tier structure and the four values that must line up

2. Three tiers: slurmctld → slurmdbd → database ​

The official architecture sends accounting records from the controller to slurmdbd (Slurm Database Daemon), which writes them into MySQL or MariaDB. Why not let slurmctld talk to the database directly? The documentation states the reason plainly: direct access requires distributing database credentials to every host that runs sacct, sreport or sacctmgr, and ordinary users run sreport, so the database security boundary disappears. slurmdbd also caches records, which reduces database pressure.

Values that must match:

ParameterFileDefault / requirement
AccountingStoragePortslurm.confMust equal the slurmdbd listening port, default 6819
StoragePortslurmdbd.confDatabase port, usually 3306
StorageLocslurmdbd.confDatabase name, default slurm_acct_db
StorageTypeslurmdbd.confOnly valid value today: accounting_storage/mysql
SlurmUserbothSame name and UID on both sides; if not root, this user needs AdminLevel=Admin

Authentication uses MUNGE. A single-cluster deployment can reuse the existing MUNGE setup. When several clusters report into one slurmdbd, every node needs MUNGE and user names must agree: accounting is keyed by user name, authentication by UID, and both must be consistent across clusters.

Two operational facts worth knowing up front:

  • A dead slurmdbd does not stop jobs. slurmctld buffers accounting records internally and flushes them when slurmdbd returns. The queue has a limit, so a long outage starts dropping records.
  • Startup order is a hard constraint. The first slurmctld start requires slurmdbd to be reachable, because accounts and limits are loaded from the database. Plan boot order and monitoring around "dbd before ctld".

3. Database: the settings that can break slurmdbd ​

The documentation recommends a small set of values, and three of them are in the "get it wrong and it breaks" category:

ParameterRecommendedConsequence if unset
innodb_buffer_pool_size5–50% of RAM, at least 4 GiBPurging and upgrades become very slow as the database grows
innodb_lock_wait_timeout900 secondsLong queries get killed
max_allowed_packetat least 16MWrites fail when AccountingStoreFlags=job_env,job_script is enabled
innodb_snapshot_isolationOFF (MariaDB)MariaDB 11.6.2+ defaults to ON; a row conflict makes slurmdbd unrecoverable and it exits

The last row is the most common cause of "slurmdbd keeps restarting after a fresh MariaDB install" and belongs in every install checklist. On MySQL 8.0.30+ use innodb_redo_log_capacity instead of innodb_log_file_size, set to 25% of the buffer pool.

4. The switches in slurm.conf and the enforce options ​

ini
# slurm.conf (excerpt)
ClusterName=ai-hpc
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=dbd-master
AccountingStoragePort=6819
AccountingStorageEnforce=associations,limits,qos
TrackWCKey=yes
PriorityType=priority/multifactor

AccountingStorageEnforce takes a comma-separated list:

ValueEffectImplies
associationsUsers without an association in the database cannot run jobs—
limitsEnforces association and QOS limitsassociations
qosEvery job must resolve to a valid QOSassociations
safeA job starts only if it can finish within remaining quota, so it is never killed mid-run by a limitassociations, limits
wckeysUsers may only use their own wckey, and TrackWCKey is setassociations
nojobs / nostepsSkip job or step detail, keeping only limit enforcementeach other

Practical advice: enable associations,limits,qos together. With only limits, QOS admission is not checked; with only qos, limits are not enforced. All three together give you "no account, no submission", "over quota, queued", and "invalid QOS, rejected".

Do not restart only slurmdbd after editing slurm.conf. Controller-side parameters need scontrol reconfigure or a slurmctld restart.

5. Building the tree: cluster → account → user, in that order ​

The documentation is explicit: define clusters first, then accounts, then users.

bash
# 1) Register the cluster (since 20.02 slurmctld adds it on start, but associations are still manual)
sacctmgr add cluster ai-hpc

# 2) Accounts, hierarchical via parent=
sacctmgr add account root  Description="root"  Organization=lab
sacctmgr add account llm,ai4s parent=root Description="two groups" Organization=lab

# 3) Users, with a default account
sacctmgr add user zhang DefaultAccount=llm
sacctmgr add user li    Account=ai4s

# 4) Inspect
sacctmgr show assoc format=cluster,account,user,partition,qos%20,grptres%30

The tree is not cosmetic. Grp* limits aggregate up the tree: a parent account's GrpTRES covers itself plus all children, and fairshare shares are also distributed over the tree. Design the tree to match the real organisation. Renaming or re-parenting later leaves historical usage attributed to the old lineage, so sreport filtered by the new account misses that history — an easy trap when reading reports.

6. Two levels of quota: association and QOS ​

Limits can live in three places: an association (the cluster/account/user/partition tuple), a QOS, and a partition. QOS limits override association limits, and partition limits act as an upper bound.

Common association and QOS limit fields:

FieldMeaning
GrpTRESTotal resources the group may hold at once, e.g. cpu=64,gres/gpu=8
GrpTRESMinsCumulative TRES-minute budget; on exhaustion running jobs are killed and new jobs are refused
GrpTRESRunMinsTotal TRES-minutes of running jobs, charged against their requested time
GrpWallCumulative wall-clock budget
GrpJobs / GrpSubmitJobsConcurrent running jobs / jobs on record
MaxTRESPerUser / MaxTRESPerJobPer-user and per-job resource caps
MaxWallDurationPerJobPer-job time limit
MaxTRESMinsPerJobPer-job TRES-minute cap

A frequent misconception is that setting GrpTRES in the database is enough to block jobs. It is not: without AccountingStorageEnforce=limits, limits are only data.

The documented precedence for limit enforcement:

  1. Partition QOS
  2. Job QOS
  3. User association
  4. Account associations, ascending the hierarchy
  5. Root/cluster association
  6. Partition limits
  7. None

Three exceptions matter. Max[Time|Wall] and [Min|Max]Nodes can never exceed the partition-level limit unless the QOS sets PartitionTimeLimit or Partition[Max|Min]Nodes. Grp* limits take the stricter of the definitions, so a tight parent account cannot be loosened at a child. And where several limits are defined at the same level, the first one encountered in the list wins rather than the minimum.

Three QOS flags are especially useful:

  • Flags=DenyOnLimit rejects a job at submission when it violates a limit on its own (by default it merely queues).
  • Flags=NoDecay stops GrpTRESMins and GrpWall from decaying, turning them into a one-shot hard quota — suitable for prepaid GPU-hour bundles.
  • UsageFactor / UsageThreshold multiply billed usage (2 means double billing), or hold jobs once fairshare drops below a threshold.

A four-tier QOS example you can adapt:

bash
sacctmgr add qos debug
sacctmgr modify qos debug set MaxWallDurationPerJob=00:30:00 MaxTRESPerUser=gres/gpu=2 Priority=100

sacctmgr add qos normal
sacctmgr modify qos normal set GrpTRES=cpu=512,gres/gpu=64 MaxWallDurationPerJob=24:00:00

sacctmgr add qos long
sacctmgr modify qos long set MaxWallDurationPerJob=7-00:00:00 GrpTRESMins=cpu=1000000 Priority=-10

sacctmgr add qos urgent
sacctmgr modify qos urgent set Priority=1000 Preempt=long
# Preemption of long by urgent also needs PreemptType=preempt/qos in slurm.conf

Note that Priority only participates when PriorityWeightQOS > 0, and Preempt only when PreemptType=preempt/qos. Both are common "configured but inactive" cases.

7. Fairshare: what the ledger actually computes ​

The fairshare factor is the difference between the promised share and the resources already consumed, on a 0.0–1.0 scale, where higher means scheduled sooner. Three decisions determine whether it is trustworthy on your cluster.

First, the unit of measure. By default usage is accounted in cpu-seconds, so GPUs and memory are invisible. A GPU cluster must set TRESBillingWeights, otherwise one A100 costs the same as one CPU core. The documented method sums each TRES quantity multiplied by its weight. With TRESBillingWeights=CPU=1.0,Mem=0.25G:

JobRequestedBilling units
Job11 CPU + 60 GiB1×1.0 + 60×0.25 = 16
Job216 CPU + 1 GiB16×1.0 + 1×0.25 = 16.25
Job316 CPU + 60 GiB16×1.0 + 60×0.25 = 31

An alternative accounting takes the max of the per-node TRES values (PriorityFlags=MAX_TRES), which suits users who reserve a whole node but use one core, so they are not billed for the entire machine. Pick deliberately: once the billing rule is live, users optimise against it.

Second, algorithm and half-life. Since 19.05 the default is the Fair Tree algorithm, which orders siblings in the account tree by share; only PriorityFlags=NO_FAIR_TREE reverts to the classic algorithm. Historical usage decays under PriorityDecayHalfLife, default 7-0 (7 days). Recalculation happens every PriorityCalcPeriod, default 5 minutes, and the age factor saturates at PriorityMaxAge, also 7 days by default.

A 7-day half-life means a heavy three-day run last week largely fades within two weeks. For quarterly quotas, lengthen the half-life (for example 14-0 or more) and set PriorityUsageResetPeriod to MONTHLY or QUARTERLY so budgets reset per period. If you want a hard "use it or lose it" budget, set PriorityDecayHalfLife=0 together with PriorityUsageResetPeriod; without reset, a zero half-life plus nothing else produces a fixed ceiling that never clears.

Third, weights. All nine factor weights default to 0, so writing PriorityType=priority/multifactor alone still behaves like FIFO. A common configuration makes fairshare dominant:

ini
PriorityWeightFairshare=10000
PriorityWeightAge=1000
PriorityWeightJobSize=1000
PriorityWeightPartition=1000
PriorityWeightQOS=0

The documentation warns against small weights: factors are decimals between 0.0 and 1.0, two jobs may differ by 0.0001 in fairshare, and a weight of 10 rounds that difference away. To resolve differences around 0.001, use weights on the order of 1000.

Three commands for inspection: sshare -l (RawUsage, EffectvUsage, FairShare, LevelFS), sprio -w (weights and per-job breakdown), and squeue --format="%.18i %.9P %.8j %.8u %.2t %.10M %.6D %Q" for priority.

8. Reporting and data retention ​

sreport reads the hourly, daily and monthly rollup tables in the database. The wider the requested range, the more likely it hits the monthly rollup, which is why "last quarter by group" returns in seconds.

bash
# Usage by user across the account tree
sreport cluster AccountUtilizationByUser start=2026-11-01 end=2026-11-30 tree

# Top users by CPU time
sreport user topusage start=2026-11-01 end=2026-11-30 TopCount=20

# Cluster utilisation: allocated, idle, down
sreport cluster utilization start=2026-11-01 end=2026-11-30

Every sreport call sends an RPC to slurmdbd, and the documentation explicitly warns against calling it in shell loops, which can degrade the daemon to denial of service. For dashboards, poll once on a schedule, store the result, and render from storage.

Decide retention at rollout, not after the database reaches tens of gigabytes. PurgeJobsAfter has a subtle unit difference:

  • PurgeJobsAfter=12months purges records older than 12 months at the start of each month;
  • PurgeJobsAfter=365days purges records older than 365 days every day.

Retention is similar but the purge cadence is very different; busy clusters should use days to spread the work. If history must be kept, use the Archive feature to write purged data to flat files and load it into a separate archive slurmdbd, copying only the association, QOS and TRES tables with mysqldump so identifiers line up.

9. Rollout order and acceptance checklist ​

  1. Install slurm and slurm-slurmdbd, create the database, and apply the settings from section 3.
  2. Write slurmdbd.conf, start slurmdbd, confirm systemctl status slurmdbd and that port 6819 is listening.
  3. Enable AccountingStorageType/Host/Port and AccountingStorageEnforce=associations,limits,qos in slurm.conf, then scontrol reconfigure.
  4. Create the cluster, account tree and users with sacctmgr.
  5. Create QOSs and attach them to associations (sacctmgr modify user xx set qos=debug,normal).
  6. Enable PriorityType=priority/multifactor, the weights, and TRESBillingWeights.
  7. Accept five checks:
CheckExpected result
A user without an association runs sbatchRejected, message says the association is missing
Submit beyond MaxTRESPerUserQueued, and squeue shows a limit reason code such as QOSMaxTRESPerUser
Exceed GrpTRESMinsRunning jobs killed at exhaustion, new jobs refused (reason like QOSGrpTRESMinutesLimit)
sshare -lLevelFS drops for persistent heavy users
sreport cluster AccountUtilizationByUserAggregates usage along the account tree

10. Known pitfalls ​

  • Limits without enforce: nothing is blocked, and the database is only a report store.
  • qos in enforce but no default QOS: every submission is rejected. Set a default QOS on the account or user first.
  • Edited enforce without scontrol reconfigure: the new flags never take effect.
  • slurmctld started before slurmdbd: the first start fails and the controller restart-loops.
  • Switching from the mysql plugin to slurmdbd: the cluster must already exist via sacctmgr add cluster, otherwise the controller cannot reach dbd.
  • Typed GRES limits silently ignored: MaxTRESPerUser=gres/gpu:tesla=1 does not stop --gres=gpu:2. Either force typed requests with a job_submit Lua plugin or record both gres/gpu and gres/gpu:tesla in AccountingStorageTRES.
  • Newer MariaDB defaults innodb_snapshot_isolation=ON: slurmdbd exits on a row conflict, so it must be turned off at install time.
  • Fairshare weights without TRESBillingWeights: on a GPU cluster, usage is billed in CPU-seconds and fairness is illusory.

The point of accounting is not the report. It turns resource allocation from first-come-first-served into something explainable, appealable and budgetable. Once quotas and fairshare go live, submission behaviour changes, so settle the rules before rollout rather than in flight — every change of accounting basis splits your history into two incompatible ledgers.

AI-HPC Expert · Contact: openaihpc@gmail.com