PTP Time Sync in Practice: From Millisecond NTP to a Sub-Microsecond Cluster Time Baseline

In one sentence: NTP is considered healthy when hundreds of machines stay within a few milliseconds of each other, while PTP (the IEEE 1588 Precision Time Protocol), backed by NIC hardware timestamping, can keep clocks on the same LAN within one microsecond. When you troubleshoot NCCL timeouts, align distributed traces or correlate cross-node monitoring metrics in an AI cluster, millisecond-grade timestamps turn cause and effect into a blur — PTP is the fix.

1. Why NTP Is No Longer Enough

On a single machine, time sync only needs to keep log timestamps sane. In a 10,000-GPU cluster, time is the foundation of troubleshooting and observability:

1. Cross-node log and trace alignment: an NCCL hang spans kernel timestamps on a dozen nodes. With 50 ms of clock skew you cannot tell which node timed out first and which one got blocked;
2. Causality in monitoring: after Prometheus scrapes DCGM, NIC and filesystem metrics, it joins them by timestamp. Clock skew fabricates impossible causality such as "congestion appeared before the traffic arrived";
3. Audit and compliance: out-of-band management actions and security forensics require an arbitrable cluster-wide timeline;
4. RoCE/Ethernet telemetry: packet loss, PFC pause frames and end-to-end latency measurements rely on the NIC hardware clock themselves.

NTP applies timestamps in software: packets are stamped only after traversing the kernel stack, queues and interrupt scheduling, so queuing jitter lands directly in the sync error. Public NTP typically lands at tens of milliseconds, and even a stratum-1 server on the local network usually delivers sub-millisecond to single-digit-millisecond accuracy. PTP, in contrast, timestamps each message in hardware at the exact moment it enters or leaves the NIC port, bypassing kernel scheduling and stack queuing. With PTP-aware equipment across the path, the vendor-documented target is sub-microsecond accuracy, far better than NTP normally achieves.

2. PTP Roles and How the Protocol Works

PTPv2 (IEEE 1588-2008) organizes clocks into an automatically elected master-slave hierarchy with four roles:

Role	Meaning	Typical host
Grandmaster (GM)	The root of the timing tree, usually disciplined by a GNSS (GPS/BeiDou) receiver	Dedicated grandmaster appliance, core switch with a GNSS module
Boundary Clock (BC)	A multi-port device that is a slave on its upstream port and a master on the others, terminating and regenerating PTP messages hop by hop	1588-capable TOR/core switches, dual-NIC servers
Transparent Clock (TC)	Does not join master election; it only corrects for its own message residence time and injects that compensation into the correction field	1588-capable switches
Ordinary Clock (OC)	A single-port clock that is either master or slave; server nodes are almost always slave OCs	ptp4l on a compute node

The hierarchy is not statically configured. Every clock runs the BMCA (Best Master Clock Algorithm) and compares properties such as priority, clock class and identity carried in Announce messages to elect the Grandmaster and build the tree. To pin a specific GM manually, lower its priority1.

2.1 Core Messages and One Sync Cycle

Master                           Slave
  │ ──── Sync (sent at t1) ───────→ │   event message
  │ ──── Follow_Up (carries t1) ─→ │   general message
  │                                  │
  │ ←──── Delay_Req (sent at t3) ── │   event message
  │ ──── Delay_Resp (carries t4) ─→ │   general message

The slave derives the offset and path delay from four hardware timestamps, then drives a servo loop that gently adjusts clock frequency instead of stepping it abruptly. PTP can run over UDP/IPv4 (event messages on port 319, general messages on port 320) or over Layer 2 Ethernet (EtherType 0x88f7).

2.2 Two Delay Measurement Mechanisms

• E2E (end-to-end, default): the slave exchanges request-response messages directly with the master, while TCs on the path compensate their residence time;
• P2P (peer-to-peer): each link measures its neighbor delay independently. It reacts faster to topology changes and can be more accurate, but every device on the path, including transparent clocks, must support P2P.

ptp4l selects them with -E (default), -P, or -A (starts in E2E and switches to P2P after receiving a peer delay request). All clocks on one PTP path must use the same mechanism; mixing them produces continuous warnings.

3. Hardware Timestamping and the PHC: Where Accuracy Actually Comes From

A PTP-capable NIC carries an independent PHC (PTP Hardware Clock), exposed on Linux as a /dev/ptpN character device. In hardware mode synchronization happens in two stages: ptp4l first disciplines the NIC PHC to the GM, then phc2sys disciplines the system clock to the PHC. In software mode there is no PHC; ptp4l adjusts the system clock directly, and accuracy degrades significantly because operating-system scheduling is back in the timing path.

3.1 Check Capabilities with ethtool Before Rollout

ethtool -T eth0
# Three things to look for:
#   hardware-transmit / hardware-receive / hardware-raw-clock  → hardware timestamping
#   PTP Hardware Clock: 0                                       → maps to /dev/ptp0
ls -l /dev/ptp*

NICs or VMs that only show SOFTWARE capabilities are limited to the -S software mode — the protocol still runs, but sub-microsecond results are off the table. Mainstream datacenter NICs (Intel ice/i40e/ixgbe, NVIDIA ConnectX mlx5 and others) generally ship a PHC; the kernel must be built with CONFIG_PTP_1588_CLOCK.

3.2 Every Network Device Must Be PTP-Aware

A plain switch will happily forward PTP frames, but its queuing delay under congestion is random and asymmetric, and the error accumulates over multiple non-PTP hops. There are two correct designs:

• Deploy BCs everywhere: each switch regenerates time distribution — the standard choice for large, multi-tier clusters;
• Or deploy TCs: switches stay out of master election and only add their residence time to correction fields — cheaper and adequate for simple, flat topologies.

The official guidance is explicit: every networking component on the path should ideally be PTP hardware enabled. When large parts of the fabric do not support PTP, that network is better served by NTP rather than a forced PTP deployment.

3.3 Three Known ConnectX Side Effects

When timestamping is enabled on NVIDIA ConnectX-4 and later adapters, plan for these engineering details up front:

1. RX CQE compression is switched off automatically (the features are mutually exclusive), which changes receive memory-bandwidth behavior in RoCE workloads;
2. VLAN stripping is disabled at the same time;
3. RSS may spread port-319/320 frames across different receive rings and reorder them; steer PTP traffic to a single RX ring with ethtool:

ethtool -U eth0 flow-type udp4 dst-port 319 action 0 loc 1
ethtool -U eth0 flow-type udp4 dst-port 320 action 0 loc 0

On top of that, the real-time clock mode on ConnectX-6 Dx and later only lets the physical function (PF) adjust the hardware clock — adjustments issued by SR-IOV VFs are silently ignored by hardware. All functions of one device must stay in a single 1588 clock domain; you cannot attach them to different GMs.

4. The Software Stack: the linuxptp Family

The kernel provides PHC drivers and socket timestamping APIs; the userspace protocol lives in the linuxptp package:

Component	Responsibility
ptp4l	The protocol daemon, implementing OC/BC; prefers hardware timestamping by default
phc2sys	In hardware mode, synchronizes the system clock (CLOCK_REALTIME) to the PHC, or the reverse direction when the system feeds the PHC
pmc	PTP management client used to read local and remote clock state for acceptance and monitoring
ts2phc	Disciplines a PHC into a Grandmaster from external timestamp sources (GNSS PPS/NMEA, 1PPS signals)
timemaster	Multi-source orchestrator that lets chronyd/ntpd pick between PTP and NTP and fail over automatically

Installation (same package name on RHEL and Debian/Ubuntu families, different config paths):

dnf install -y linuxptp          # RHEL/Rocky
apt install -y linuxptp          # Ubuntu/Debian

5. Compute Node Deployment: ptp4l + phc2sys

5.1 ptp4l Configuration

On RHEL-family systems the file is /etc/ptp4l.conf; on Ubuntu it is /etc/linuxptp/ptp4l.conf (with the interface declared separately via OPTIONS in /etc/default/ptp4l). A minimal slave configuration:

[global]
# UDP/IPv4 transport; use L2 for Layer 2 deployment
network_transport UDPv4
# E2E is default; switch to P2P only when every path device supports it
delay_mechanism E2E
# Hardware timestamping; fall back to "software" where unsupported
time_stamping hardware
# PTP domain, identical across the fabric (0-127), isolates PTP networks
domainNumber 0
# Adaptive servo: no manual PI tuning needed, recommended for modern setups
clock_servo linreg
# Quiet logging: one summary line every 2^10 seconds
summary_interval 10

[eth0]
# Per-interface section may stay empty or carry overrides

Foreground debugging run (-m logs to stdout, -i picks the interface, -S forces software timestamping):

ptp4l -i eth0 -m

A healthy convergence walks the port state machine through LISTENING → UNCALIBRATED → SLAVE, followed by per-second statistics:

ptp4l: rms 226 max 525 freq -29802 ± 196 delay 9228 ± 148

Here rms is the root-mean-square offset, max the largest absolute offset in the window, and delay the estimated path delay — all in nanoseconds. Servo states s0/s1/s2 mean unlocked, stepping and locked respectively; once in s2 the clock is only frequency-adjusted and never stepped by default.

5.2 phc2sys: Handing the PHC Time to the System

On RHEL edit /etc/sysconfig/phc2sys; on Ubuntu edit /etc/default/phc2sys. The production-recommended setting:

OPTIONS="-a -r"

-a makes phc2sys follow ptp4l's port state to pick the right PHC automatically; -r tells it to discipline the system clock as well. The equivalent manual debug invocation:

phc2sys -s eth0 -w -m

-w waits for ptp4l to lock the PHC before starting and obtains the TAI-to-UTC leap-second offset from ptp4l automatically. Note that PTP runs on the TAI timescale while the system clock keeps UTC. Without -w you would have to hard-code the offset with -O (TAI is currently 37 seconds ahead of UTC, and the value changes with leap-second announcements), so always prefer -w and never hard-code it.

systemd handles ordering and boot-time enablement; with -w there is no manual startup-order constraint:

systemctl enable --now ptp4l phc2sys

5.3 Firewall Rules

UDP 319/320 must be allowed in both host firewalls and network ACLs, otherwise BMCA never receives Announce messages and the port stays in LISTENING forever:

firewall-cmd --permanent --add-service=ptp
firewall-cmd --reload

6. Living with chrony: Pick One Integration Pattern

Only one daemon may discipline the system clock on a machine — phc2sys and chrony must never write CLOCK_REALTIME at the same time. Two mature combinations exist.

Pattern A: phc2sys disciplines the system clock, chrony stays as NTP backup

While PTP is healthy phc2sys is in charge; chrony keeps internal NTP server lines as a fallback for PTP failures. This is the classic RHEL-documented chain with clean component boundaries, suited to clusters with full PTP hardware coverage.

Pattern B: chrony consumes the PHC directly as a hardware reference clock

Do not run phc2sys; let chronyd sample the NIC hardware clock through its PHC driver while retaining NTP sources for arbitration, so chrony performs source selection itself:

# /etc/chrony.conf
# The PHC runs on TAI; "tai" makes chrony apply the UTC-TAI offset automatically.
# An explicit "offset -37" also works but is discouraged because leap seconds change.
refclock PHC /dev/ptp0 poll 0 dpoll -2 tai prefer trust
# Keep NTP as failover
server ntp.internal.example.com iburst
makestep 1.0 3
rtcsync

poll 0 means one sample per second, dpoll -2 sets sub-second driver polling, and prefer plus trust prioritize the PHC in source selection and prevent ordinary NTP peers from falsely rejecting it. Cloud environments (some virtual machines expose /dev/ptp_kvm, ENA exposes a PHC) commonly use this pattern. Verify with:

chronyc sources    # A ^* marker on the PHC line means it is selected
chronyc tracking   # Leap status, system time offset and frequency

Edge case: timemaster

When you must switch between multiple PTP domains or want automatic "drop PTP, fall back to NTP" behavior without writing your own watchdog, use the bundled timemaster: it feeds ptp4l/phc2sys samples to chronyd over shared memory and lets the NTP daemon adjudicate. It is heavier to configure; an ordinary single-domain cluster is better served by pattern A or B.

7. Grandmaster and Cluster Topology

An operable PTP network typically looks like this:

GNSS antenna (GPS/BeiDou, 1PPS + NMEA)
        │
  Core Grandmaster (or GNSS-equipped core switch, BC)
        │  PTP over UDPv4 / L2
   TOR switches (BCs, regenerating PTP messages hop by hop)
   ┌────┴────┐
 Compute OC   Compute OC    … (NIC hardware timestamping + /dev/ptpN)

• GM side: purpose-built grandmasters ship with integrated GNSS receivers. When a server acts as GM, ts2phc disciplines the PHC from serial NMEA plus the 1PPS interrupt on the PHC, phc2sys runs with -r -r to push system time back into the PHC, and ptp4l advertises time with a high priority (small priority1);
• Fabric side: BCs suit large multi-hop networks, TCs suit flat topologies within two hops. Areas traversing plain switches must be explicitly marked as degraded zones with no sub-microsecond promise;
• PTP-NTP bridge: one machine acts as a PTP slave and simultaneously serves as a stratum-1 NTP server via chrony, feeding legacy devices and VMs that cannot speak PTP so you do not need a separate GM for them.

Note that native InfiniBand fabrics do not run IEEE 1588; PTP belongs on the Ethernet management network or on RoCE networks. ConnectX RoCE timestamping is exposed to applications through verbs APIs and works alongside the Ethernet PHC used by the 1588 daemons.

8. Acceptance: Prove the Lock with Numbers

8.1 Inspect ptp4l state machine and offsets

pmc -u -b 0 'GET CURRENT_DATA_SET'

Key fields: gmPresent must be true, stepsRemoved is the hop count to the GM, and offsetFromMaster plus meanPathDelay are in nanoseconds. With hardware timestamping on a lightly loaded LAN, a stable rms offset should sit within a few hundred nanoseconds, with max offset under 1–2 microseconds. Staying at s0 or seeing offsets in the tens of thousands of nanoseconds means troubleshooting per section 9.

8.2 Inspect the system-clock chain

systemctl status ptp4l phc2sys
journalctl -u phc2sys -u ptp4l --since "-10 min" | grep -E "s2|SLAVE|fault"
chronyc tracking        # In pattern B, check Last offset and RMS offset

8.3 Cluster-wide sampling

Run pmc and chronyc tracking fleet-wide through Ansible and push per-node rms/max offsets into Prometheus (node exporter textfile collector or a custom script) with threshold alerts: file a ticket when rms stays above 1 μs for 5 minutes, and page immediately when gmPresent goes false.

9. Six Most Common Pitfalls

1. Mistaking software timestamping for hardware. When ethtool -T shows no hardware-raw-clock, or a VM gets an ordinary virtual NIC, ptp4l silently falls back to software mode and barely beats NTP. The first acceptance step is always confirming that /dev/ptpN exists and the ptp4l log prints selected /dev/ptp0 as PTP clock;
2. Plain switches inserted on the path. Lab tests over a direct cable look beautiful, then jitter explodes in production behind a non-PTP aggregator. BMCA is healthy while offsets look terrible — investigate switch 1588 capabilities before touching servo parameters;
3. Two daemons fighting over the clock. phc2sys and chrony writing CLOCK_REALTIME simultaneously pull against each other, producing periodic sign-flipping offsets. Assign a single owner following section 6;
4. VM/SR-IOV misplacement. Ordinary VMs only get software timestamping; a VF cannot discipline the ConnectX real-time clock, so the PTP daemon belongs on the host or in an environment holding the PF;
5. Ignoring NIC side effects. Enabling timestamping kills CQE compression and VLAN stripping and may reorder RSS traffic. Assess RoCE throughput impact in advance and apply the ethtool steering rules;
6. Hard-coding the TAI-UTC offset. After a leap-second change the whole fabric silently drifts by one second. Stick with phc2sys -w or chrony's tai option for automatic handling.

Go-Live Acceptance Checklist

1. ethtool -T confirms the three hardware-timestamping capabilities and the /dev/ptpN index;
2. Switch side confirms BC/TC mode and a PTP domain number matching the servers; UDP 319/320 is open end to end;
3. The ptp4l port reaches SLAVE and the servo reaches s2; pmc reports gmPresent=true;
4. Exactly one of phc2sys or chrony owns the system clock; chronyc/phc2sys logs show rms in the sub-microsecond range;
5. On ConnectX, review the CQE compression, VLAN and RSS steering side effects;
6. Run a GM failure drill: pull the GM uplink and verify BMCA failover or NTP fallback matches the design, then confirm automatic re-lock on recovery;
7. Collect offset metrics fleet-wide and alert on sustained rms above 1 μs.

Three Closing Takeaways

• PTP accuracy comes from hardware, not from protocol magic: without a PHC and PTP-aware switches, perfectly formatted 1588 messages still only give you software timestamping;
• The clock chain needs a single owner: GM → BC/TC → OC handles PHC alignment, while exactly one of phc2sys or chrony may write the system clock;
• Accept on nanosecond numbers, not service state: active does not mean locked. The offsetFromMaster from pmc and the RMS offset from chronyc are the actual deliverable evidence for a time baseline.

Further Reading

• Cluster Delivery Automation: PXE Install, Ansible Configuration and Slurm Readiness
• GPU Training Network Troubleshooting: Packet Loss, Congestion and NCCL Timeouts
• L40S Cluster openEuler Kernel and System Tuning: BIOS, Kernel Parameters and IRQ Affinity
• nvitop: Real-Time GPU Monitoring and Process Management