05. Storage Systems

Chapter focus: ## Parallel File Systems

• GPFS (Spectrum Scale): High-performance parallel file system by IBM.
• Lustre: Open-source high-performance file system used in Supercomputers.

Storage Tiering

• Hot Data (NVMe SSD)
• Warm Data (SATA SSD)
• Cold Data (HDD/Tape)

AI Challenges

• Small file random reads (ImageNet/Audio)
• Fast Checkpoint writing (LLM Fault Tolerance)

🧭 Reading Map

• Prerequisites: 03. Cluster Architecture — storage as the cluster data plane depends on overall architecture
• Role: Part 1 storage chapter — Lustre/BeeGFS/GPFS deployment and parallel IO benchmarks; feed data fast enough
• Order: parallel-storage principles & tiering first, then deploy a parallel FS as needed, then accept with IOR/mdtest
• Next: 05. Provisioning — from "designed" to "provisioned at scale"

Articles

Deep Dive: Enterprise BeeGFS Deployment & Tuning Guide

implementation guide for BeeGFS. Covers hardware planning, OS tuning, multi-instance architecture, Buddy Mirror HA, Quota, and BeeOND.

阅读文章 →

Practical Guide: IBM Spectrum Scale (GPFS) ECE Deployment

A real-world implementation guide for GPFS ECE (Erasure Code Edition), covering IB network setup, NVMe drive mapping, recovery groups, and client mounting.

阅读文章 →

Lustre 2.15.7 Deployment Guide (EL8)

A complete guide for installing, configuring, and mounting Lustre Parallel File System on RHEL/Rocky Linux 8.

阅读文章 →

Advanced Storage Features

multi-tenant IO isolation and QoS - metadata hotspot mitigation

阅读文章 →

Parallel File System Benchmarking & Tuning: IOR/mdtest in Practice

parameter semantics, training load modeling, judgment, and tuning paths.

阅读文章 →

Training Data Storage Selection: Object Store, Parallel FS & Local NVMe

roles, data-flow analysis, tiered architecture.

阅读文章 →