Aligning the Three Clocks in a Cluster: System Time, Board RTC and BMC

In one sentence: clock problems in a cluster are rarely "the clock is wrong"; they are almost always "three clocks each telling a different story and nobody defined the convention". The system clock, the board RTC and the BMC clock are three independent devices — only by writing the convention into a file, fixing the direction to one-way, and mechanising the audit do you avoid discovering an eight-hour gap inside a job timeout or an alert.

1. A Server Actually Has Three Clocks

Clock	What it is	Keeps running when powered off	Maintained by	Accuracy
System clock	The kernel's time — essentially "seconds since 1970-01-01 UTC"	No, lost on reboot	chrony / NTP	Bounded by its sources
Board RTC	An independent, battery-backed device; also called CMOS clock, BIOS clock, persistent clock in the kernel	Yes, which is why it initialises the system clock at boot	Practically nobody	Whole-second reads, drifts long-term
BMC clock	The out-of-band management chip's own clock	Yes (the BMC stays powered)	Almost nobody	Drifts faster than the RTC

Each serves something different:

• The system clock decides job accounting, log timestamps, make dependency checks and TLS certificate validity — it is the "business time";
• The board RTC has exactly one job: keep time while the machine is off, then hand its value to the system clock as a starting point;
• The BMC clock only affects the out-of-band side: SEL event timestamps, sensor alerts, SNMP traps and alert email times. There is no automatic synchronisation path between it and the OS.

One crucial realisation: the RTC stores only a date and a time — it has no concept of timezone. The clock itself carries no "I am UTC" or "I am local time" marker; that convention lives entirely in an external file (next section). The BMC does not even have that file — it just prints whatever number it holds.

2. The Direction Is One-Way: System Time → RTC / BMC

hwclock can write in both directions, but only one of them is correct:

# Right: system time → RTC
hwclock --systohc

# Wrong direction: RTC → system time
hwclock --hctosys        # manual: "should never be used on a running system"

Two solid reasons why --hctosys is wrong on a live machine:

1. A jump corrupts filesystem timestamps. Move system time backwards by a second and a process mid-write or mid-fsync can see "file modified in the future"; some filesystems and databases will error out;
2. Drift compensation gets applied twice. --hctosys "corrects" the RTC value using the drift rate recorded in the adjtime file — but that RTC value may already have been modified by something else (the kernel's 11-minute mode, another operating system), so the correction lands on the wrong base.

There is also a precondition people skip: the system time must be trustworthy before you run hwclock --systohc. If chrony is still chasing a public source that is 190 ms off and you write the RTC, you have just carved the error into the oscillator — every subsequent boot starts from the wrong place.

So the order is always: fix the time source → write the RTC → then deal with the BMC.

3. /etc/adjtime: The Convention Must Be on Disk

One paragraph of the hwclock manual deserves reading word by word:

If you specify neither --utc nor --localtime then the one last given with a set function, as recorded in /etc/adjtime, will be used. If the adjtime file doesn't exist, the default is UTC.

In other words, the reading looks correct right now possibly only because the default happens to match. The moment someone runs hwclock --localtime --systohc, or a script passes -l, the RTC becomes local time — after which every hwclock -r is off by eight hours and no file records how it was written.

/etc/adjtime is three ASCII lines:

Line 1: drift rate (seconds/day)  timestamp of last adjustment (seconds since 1969 UTC)  0.0
Line 2: timestamp of the last calibration (0 if never calibrated)
Line 3: UTC or LOCAL

Line 3 is the convention. The proper way to create it:

timedatectl set-local-rtc 0      # declare the RTC as UTC, writes /etc/adjtime
cat /etc/adjtime                 # line 3 should read UTC

If the file was lost and you actually want the RTC in local time, hwclock --localtime --adjust will create an adjtime file declaring LOCAL with a zero drift rate when the file is absent (it does not actually adjust the clock in that case).

There is also a dependency people miss: chronyd parses this same file itself (hwclockfile, defaulting to /etc/adjtime) to decide whether the RTC holds UTC or local time. With the file missing, chrony is guessing too. So adjtime is not a private matter between you and hwclock — it is the system-wide carrier of the convention.

One common misunderstanding to correct: modern util-linux does not recalculate the drift rate on hwclock --systohc (since v2.31 it requires an explicit --update-drift), and it does not read the RTC at all. It does two things: write the system time, and update the timestamps in adjtime. Expecting it to "learn how much the RTC drifts" is a habit from older versions.

4. UTC or Local: The Asymmetry in Display Is the Real Trap

The manual is blunt about LOCAL versus UTC:

• With the RTC in local time, daylight saving changes are inconsistent: if Linux is running during the switch the value written follows it; if Linux is not running, the value read does not;
• With the RTC in UTC, Linux only converts at the display layer, so DST is transparent.

Hence the standard practice: store the RTC in UTC. In a zone like ours (+08:00 all year, no DST) neither choice will actually break — but the confusion comes from somewhere else: the display asymmetry.

Clock	Stored as UTC	Stored as local
Board RTC (hwclock -r)	prints 19:11 — it converts UTC to local before printing (manual: "The time shown is always in local time, even if you keep your Hardware Clock in UTC")	prints 19:11
BMC (ipmitool sel time get)	prints 11:11 — no conversion, raw stored value	prints 19:11

That table is the whole source of the conflict: hwclock converts, the BMC does not.

• Storing the board RTC in UTC is a win-win: the convention is right and the display still equals date;
• The BMC can only choose one: UTC is correct but reads eight hours off, local reads equal to date.

If the operational habit is "what sel time get prints must equal date", then keep the BMC in local time and write that down as the convention. The actual mistake is not the choice — it is having both conventions in the same fleet with nobody recording which is which.

5. chrony and the RTC: rtcsync and rtcfile Are Mutually Exclusive

This is the most commonly misconfigured spot, because both options "keep the RTC right" while doing completely different things:

Setting	Mechanism	RTC drift observable?	Accuracy
rtcsync	On Linux this only enables the kernel's 11-minute mode: once the clock discipline is synchronised, the kernel copies system time to the RTC every 11 minutes. chrony itself does not track RTC drift	No	Second-level
rtcfile + rtcautotrim 1	chrony measures the RTC's offset and rate itself, records them, and trims when over threshold	Yes — chronyc rtcdata gives ppm directly	Trim is accurate to about 1 second; 1 second is the minimum effective threshold

The two cannot be combined (docs: rtcsync cannot be used with rtcfile).

# Just keep the RTC from wandering — least effort
rtcsync

# Want drift data to judge whether the coin cell is aging — more informative
rtcfile /var/lib/chrony/rtc
rtcautotrim 1

rtcdata produces exactly the evidence you need for hardware problems:

RTC is fast by     : -1.632736 seconds      # how far ahead the RTC is
RTC gains time at  : -107.623 ppm           # gain/loss per second
Number of samples  : 10
Sample span period : 549

107 ppm is roughly 9 seconds per day — that is not "software failed to sync", that is a crystal or a battery that needs replacing. Without rtcfile you can only estimate by repeatedly comparing hwclock -r with date, and you cannot get ppm out of that.

6. Reading chrony's Three Reports: tracking / sources / sourcestats

6.1 tracking: Look at the Error Bound, Not Just the Offset

System time     : 0.000006523 seconds slow of NTP time
Last offset     : -0.000006747 seconds
RMS offset     : 0.000035822 seconds
Frequency       : 3.225 ppm slow
Residual freq   : -0.000 ppm
Skew            : 0.129 ppm
Root delay      : 0.013639022 seconds
Root dispersion : 0.001100737 seconds
Update interval : 64.2 seconds

The accuracy bound from the manual is worth memorising:

clock_error <= |system_time_offset| + root_dispersion + 0.5 * root_delay

That formula explains why "offset is only 8 ms" is not reassuring: with a root delay of 246 ms, that single term contributes 123 ms of possible error. The offset is "the deviation measured last time"; the bound is "why you should believe that number".

6.2 sources: Three Columns Decide Whether a Source Is Usable

#* ntp1.example.net   2   6   377   23   -923us[ -924us]   +/- 43ms

Column	Meaning	How to read it
S	Selection state: * best, + combined, - selectable but unused, x falseticker, ~ too variable, ? not selectable	Only * and + actually steer the clock
Poll	Poll interval as a base-2 exponent; 6 means 64 seconds	Larger saves traffic and reacts slower
Reach	8-bit reachability register, printed octal; 377 = valid replies to all last 8	Anything other than 377 means loss or unreachable
The +/- after Last sample	That source's root distance, i.e. its measurement error bound	Hundreds of milliseconds means write it off

6.3 sourcestats: Whether the Frequency Estimate Can Be Trusted

Name/IP Address   NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev

Column	Meaning
NP	Samples currently retained (the points used for regression)
NR	Runs of residuals with the same sign — too few means "a straight line no longer fits"
Span	Time span from oldest to newest sample
Frequency	Estimated residual frequency for that source, ppm
Freq Skew	Error bound on the frequency estimate, ppm
Offset	Estimated offset for that source
Std Dev	Sample standard deviation

The criterion is in the configuration: maxupdateskew defaults to 1000 ppm, and estimates above that threshold are treated as unreliable and not used. So when a source shows a Freq Skew of tens of thousands of ppm, the conclusion is immediate — that source has not even established its own stability, and keeping it only pollutes source selection.

A real pair of readings (a public pool mixed with domestic sources):

Type	Offset	Freq Skew	Std Dev	Verdict
Public pool member A	+193 ms	55699 ppm	7316 us	Unusable
Public pool member B	−92 ms	29072 ppm	6379 us	Unusable
Public pool member C	−42 ms	63000 ppm	7213 us	Unusable
Domestic source A	−5372 us	253 ppm	1488 us	Usable
Domestic source B	+343 us	455 ppm	981 us	Usable

Note the "unusable" ones have Std Dev of only a few milliseconds — a single measurement looks accurate while the long-term frequency is completely untrustworthy. That is precisely what the Freq Skew column is for: it does not judge "was this one accurate", it judges "can this source carry you long-term".

The counter-intuitive rule: three steady sources beat seven where half are noise. More sources make majority voting easier to drag into a collective bias, and every bad source eats into the signal-to-noise ratio that minsources depends on.

6.4 Two Configuration Details

• Do not poll public servers faster than 64 seconds. The docs are explicit: intervals shorter than minpoll 6 (64 s) against public Internet servers may be considered abuse; sub-second polling is only permissible when the round-trip delay is under 10 ms, i.e. a local network;
• minsources defaults to 1. To require "at least two sources must agree before the clock is updated", raise it to 2 or 3 — the price is that the clock stops updating when too few sources are available, which is better than being dragged off by a single bad one.

7. BMC Time: Write It Right Once, Then Rely on a Fallback

ipmitool sel time get                        # read
ipmitool sel time set "11/14/2026 19:08:07"  # write

Four practical notes:

1. The format follows the locale; the manual says "using the locale's date / time format". The steadiest choice is %m/%d/%Y %H:%M:%S under LC_ALL=C; if it complains about the format, check locale first rather than guessing;
2. The manual recommends clearing the SEL before setting the time, because existing entries carry timestamps from the old clock and mixing them makes the log useless. But sel clear is irreversible — do not do it casually on a production node; confirm nobody depends on that history first;
3. Pre-Init Time-stamp appearing in the SEL is proof: the BMC clock was never set, and those event timestamps cannot be trusted. It is the fastest way to decide whether this machine's clock needs attention at all;
4. Node busy comes from serialising SEL writes. The BMC handles one write request at a time; a second request issued too soon is rejected outright. The fix is backoff and retry (ten seconds apart), not concurrent retries — and it hits most often while out-of-band monitoring is polling the SEL.

Two scenarios that inevitably roll the clock back, and belong in the process: after flashing BIOS/BMC firmware, and after the machine has been powered off for a while. So BMC time is not "set once and forget": either configure the BMC's own NTP (most management UIs have it on the Date/Time page — that is the real answer), or run a daily script as the fallback.

8. Turn the Audit into a Script, Avoiding cron's Two Traps

Three actions need mechanising: read the three clocks → judge the deviation → correct and log when over threshold. Do it in a script rather than one crontab line, because cron has two guaranteed traps:

• A % in the command part of a crontab is converted to a newline. Writing date '+%m/%d/%Y' truncates the command outright; it must be escaped as \%. The safe approach is a command containing no % at all — put everything in a script;
• Files in /etc/cron.d/ must be root:root and mode 644, otherwise cron silently ignores them and only leaves "WRONG FILE OWNER" in syslog.

cat > /usr/local/sbin/time-audit.sh <<'EOF'
#!/bin/bash
# Three-clock audit: system clock / board RTC / BMC.
# Convention: RTC stored as UTC, BMC stored as local time.
# usage: time-audit.sh [-f]   -f = auto-correct when over threshold
set -u
FIX=0; [ "${1:-}" = "-f" ] && FIX=1

sys=$(date +%s)
rtc_s=$(date -d "$(hwclock -r)" +%s 2>/dev/null || echo 0)
drift=$(( rtc_s - sys )); [ "$drift" -lt 0 ] && drift=$(( -drift ))

adj=$(sed -n '3p' /etc/adjtime 2>/dev/null || echo "MISSING")
echo "adjtime=${adj:-MISSING}  RTC_drift=${drift}s"

if [ "$adj" != "UTC" ] && [ "$adj" != "LOCAL" ]; then
    echo "WARN: /etc/adjtime missing or convention undefined" >&2
fi

if [ "$drift" -gt 5 ] && [ "$FIX" = "1" ]; then
    hwclock --systohc && logger -t time-audit "RTC corrected (drift ${drift}s)"
fi

bmc=$(ipmitool sel time get 2>/dev/null | awk '{print $1" "$2}')
echo "BMC=${bmc:-unreadable}"
if [ -n "$bmc" ] && [ "$FIX" = "1" ]; then
    for i in 1 2 3; do
        ipmitool sel time set "$(date '+%m/%d/%Y %H:%M:%S')" 2>/dev/null && {
            logger -t time-audit "BMC set ok"; break; }
        logger -t time-audit "BMC set failed try=$i"; sleep 10
    done
fi
EOF
chmod 755 /usr/local/sbin/time-audit.sh

cat > /etc/cron.d/time-audit <<'EOF'
17 3 * * * root /usr/local/sbin/time-audit.sh -f
EOF
chmod 644 /etc/cron.d/time-audit

Run it once without arguments, confirm the logic, then add -f.

Acceptance thresholds (worth putting into the delivery baseline):

Item	Threshold	If exceeded
tracking offset	±10 ms (the realistic ceiling for public sources)	Change sources; look at Freq Skew
hwclock -r vs date	≤ 1 s	hwclock --systohc, after confirming the adjtime convention
ipmitool sel time get vs date	≤ 1 s (under the BMC-stores-local convention)	Re-set by script; check the SEL for Pre-Init Time-stamp
/etc/adjtime line 3	UTC	Fix it if missing or LOCAL
RTC drift rate	≤ 1 s/week	Check battery/crystal — writing it will not help

9. Limits: Where This Method Stops

• Public NTP tops out at milliseconds. With a 246 ms round-trip, even a good offset lands in the tens of milliseconds. Microseconds require PTP with hardware timestamping; changing sources will not get you there (see PTP Time Synchronisation in Practice);
• Relative consistency matters more than absolute accuracy. Dozens of nodes each chasing their own public source can sit tens of milliseconds apart — far worse than being uniformly 8 ms slow. With no internal time server, point one node at the public source and let the rest point at it (allow in chrony for the subnet); that single move pulls inter-node spread under 1 ms;
• Leap seconds: leapsectz right/UTC lets chrony handle them via the tz database; with many long-running jobs, also consider what an inserted second does to job timeout logic — smoothtime can smear it on the serving side;
• Do not step a running node. A one-second offset should be slewed; if you must step, first confirm no running job depends on monotonic time;
• Virtualised clocks: with kvm-clock/tsc time sources, steal time inflates the offset — check whether Frequency in chronyc tracking looks abnormal. Containers share the host clock and neither need nor should run their own chronyd;
• Bake these conventions into the image. Fix chrony config, the /etc/adjtime convention and the BMC sync script at the golden-image stage rather than patching machine by machine after delivery (see Golden Image Pipeline and Cluster Delivery Automation).

Three Closing Takeaways

• Three clocks, one direction: the system clock is always the source; the RTC and BMC are only ever written, and pulling time back from them is wrong;
• The convention must be on disk: that single word on line 3 of /etc/adjtime decides whether the next person to take over this machine sees 19:11 or 11:11;
• The audit must be mechanised: sources whose Freq Skew exceeds maxupdateskew get dropped, a 1-second gap between hwclock -r and date gets written, Pre-Init Time-stamp gets fixed — none of these should depend on a human watching.

Further Reading

• PTP Time Synchronisation in Practice: from NTP Milliseconds to a Sub-Microsecond Baseline
• Golden Image Pipeline: Standardised, Reproducible Node Delivery
• Cluster Delivery Automation: Engineering Batch Deployment and Acceptance
• Slurm Job Submission and Management Guide
• PXE Troubleshooting Handbook: from DHCP to Kernel Boot

Related Articles

• BCM Cluster Management in Practice: The Bare-Metal Lifecycle from Software Image to Category
• Incident Review: How LD_LIBRARY_PATH Broke dracut and XFS Mount
• Linux Dual-NIC NAT Gateway Guide
• nvitop: An Easier GPU Monitoring & Process Management Alternative to nvidia-smi