Storage Observability & Performance Governance: A Bandwidth/IOPS/Latency Closed Loop

Note: This is a review article (no first-hand experimental data), based on public ops practice.

Experiment Objective

"Training is slow but GPU is not saturated" — besides data loading (ch09) and networking (ch03), it may be storage: throughput capped, IO queuing, or one client hogging bandwidth. Without storage monitoring these are all black boxes. Goal: a storage observability system — the storage metric model (bandwidth/IOPS/latency/queue depth) → dual client/server collection → alert design (what threshold alarms) → attribution and governance actions. After reading you can answer: is storage saturated, who is eating bandwidth, is the slowness server-side or client-side.

Code Walkthrough

Part 1: Dual-view metric collection

# View 1: server side (PFS/OSD, authoritative capacity and health)
# Lustre: lctl get_param -r llite.*.read_ahead_stats  # client read-ahead
# GPFS/BeeGFS: monitor OSS/OSD bandwidth and queues
# Object store: MinIO/S3 prometheus metrics (s3_requests_total, io_time)

# View 2: client side (training node, "application-perceived")
# Node-level disk/network throughput (within ch02's node_exporter stack)
iostat -x 1          # %util, await, svctm
# Training side: DataLoader read latency and throughput (ch09 profiler data layer)

Key points:

1. Dual view is mandatory: the server tells you "is storage busy/healthy overall"; the client tells you "what latency/throughput the app actually perceives" — a mismatch is the problem (e.g. server idle but client slow → network or client config).
2. Three core metrics: bandwidth (enough throughput?), IOPS (many small requests?), latency (how slow per request?); plus queue depth (accumulating) to judge "overloaded."
3. Metrics into unified monitoring: storage metrics into Prometheus/Grafana (ch02 stack), co-displayed with GPU/network — one glance shows whether it is storage, network, or GPU.

Part 2: Alert design

# Example alert rules (Prometheus)
# 1) High storage bandwidth utilization (PFS side)
- alert: StorageBandwidthHigh
  expr: sum(rate(pfs_bytes_total[5m])) / storage_bandwidth_capacity > 0.85
  for: 10m
# 2) Client read latency spike (training side)
- alert: ClientReadLatencyHigh
  expr: histogram_quantile(0.99, rate(data_loader_read_seconds[5m])) > 0.05
  for: 10m
# 3) Queue depth (IO overload signal)
- alert: StorageQueueDepth
  expr: pfs_queue_depth > 100
  for: 5m

Key points:

• Alerts need tiered thresholds: 85% bandwidth for 10 minutes is "warning," 95% is "critical"; the 99th-percentile latency over budget (e.g. 50ms) is the user-side signal — high bandwidth is not necessarily a problem (sequential reads should be high); high latency is UX degradation.
• for durations prevent flapping: storage metrics fluctuate; alerts must carry sustained durations (5-10 min) to avoid transient false alarms.
• Client latency alerts are closest to users: DataLoader read-latency p99 is direct evidence of "training stalled by storage" (ch09's profiler data-layer metric landing here).

Principles

Four storage performance problems and attribution

Symptom	Possible cause	Attribution
bandwidth capped	storage/network ceiling	server bandwidth utilization vs client throughput
high latency, low bandwidth	many small requests (IOPS bottleneck), queuing	IOPS + queue-depth metrics
one client slow	client network/config/hotspot	client-side latency distribution, find hotspots
occasional spikes	checkpoint/prefetch bursts (ch04)	time alignment: spike time vs checkpoint cycle

Core insight: 80% of storage performance problems are "access-pattern problems," not "storage is broken" — small-file storms (IOPS bottleneck), checkpoint bursts (bandwidth spikes), single-client hogging (hotspot). Attribution relies on "time alignment + dual-view comparison": align burst times with training events (checkpoint/data switch) and locate instantly.

Governance loop

monitoring (dual view) → alerts (tiered thresholds)
  → attribution (time alignment + dual-view comparison)
  → governance (cache tiers/prefetch tuning/quotas/capacity)
  → re-measure → iterate

Homomorphic to ch02's GPU cluster governance: observe first, attribute, govern, verify. Storage-side actions: add cache layers (the S3 article), tune prefetch (ch04), cap client quotas (anti-hogging), expand or upgrade storage.

Results

(Estimated, not measured)

Item	Without monitoring	With closed loop
storage bottleneck discovery	guess from "slow training"	bandwidth/latency metrics pinpoint
bandwidth allocation	no quotas, clients squeeze each other	quotas + monitoring, fair
burst attribution	hard (black box)	located by time alignment

Engineering Takeaways

1. Dual view first: server + client metrics both collected, co-displayed with GPU/network — storage problems go from "guessing" to "seeing."
2. Latency alerts before bandwidth alerts: high bandwidth is not necessarily a problem; over-budget latency is UX degradation — alert in "latency → IOPS → bandwidth" order.
3. Attribution by time alignment: align bursts with checkpoint/data switches; 80% of storage problems locate immediately.
4. Governance closes the loop with monitoring: cache, prefetch, quota, capacity all re-measured — verify before declaring done.

Next article preview: storage and data-path governance done; back to the kernel layer — "GPU Kernel Memory Access Optimization: Coalescing, Shared Memory, and Bank Conflicts."