GPUDirect Storage in Practice: A Direct Data Path from NVMe into GPU Memory

In one sentence: a conventional read moves data twice — disk to a CPU bounce buffer, then into GPU memory. GPUDirect Storage (GDS) lets storage DMA data straight into GPU memory; the CPU issues commands but no longer moves bytes. The network-side shortcut is GPUDirect RDMA; the storage-side shortcut is GDS — one handles how data leaves, the other how it arrives.

GPUs keep getting faster, and I/O is increasingly the bottleneck: terabytes of dataset load at job start, checkpoints are written periodically during training, and inference services constantly swap weights from disk. This article covers the GDS data path, the conditions in which it actually activates, how to benchmark it, how to integrate it, and the most expensive trap — believing direct I/O is on while the workload is silently running in compatibility mode.

1. The conventional path: two copies, with the CPU as mover

Without GDS, a read flows like this:

NVMe / network storage
   |  DMA (into the page cache / a user-space buffer)
   v
CPU system memory   <- first stop: storage to memory
   |  cudaMemcpy (CPU-driven, consuming memory bandwidth and CPU cycles)
   v
GPU memory          <- second stop: memory to GPU
   v
CUDA kernel starts

Three costs follow:

1. Data lands twice, so latency carries an extra hop by construction;
2. CPU memory bandwidth is consumed. Multi-drive NVMe sequential reads can reach tens of GB/s, all crossing the CPU memory controller and contending with dataloader preprocessing;
3. CPU utilization looks artificially high, with cycles spent on copies instead of decode and augmentation.

GDS removes the middle bounce buffer:

NVMe / RDMA-capable storage NIC
   |  peer-to-peer DMA (addresses translated by nvidia-fs / P2PDMA)
   v
GPU memory          <- one stop
   v
CUDA kernel starts

DMA engines already do address translation; what a traditional OS cannot do is target GPU virtual addresses — it only knows how to land data in kernel-managed physical pages. GDS fills that gap: when the filesystem driver builds its DMA scatter-gather list, callbacks supply the physical addresses behind GPU virtual addresses, so DMA can target video memory directly.

2. The software stack: libcufile, nvidia-fs.ko and the newer P2PDMA path

GDS is a set of components rather than a single driver:

Component	Layer	Responsibility
libcufile.so	User space	The application-facing cuFile API (cuFileRead/cuFileWrite and friends); path selection, buffer management, batched submission
nvidia-fs.ko	Kernel (the primary path before CUDA 12.8)	Serves user-space IOCTLs and provides DMA callbacks that translate GPU virtual addresses to physical addresses
Compatibility mode	User space	When nvidia-fs is unavailable, falls back to POSIX pread/pwrite into system memory plus a copy into the GPU — correct, but with no direct-path benefit

Since CUDA 12.8 there is a cleaner path: on x86_64, GDS can use the upstream Linux kernel PCI P2PDMA infrastructure without nvidia-fs.ko and without custom nvme.ko patches. It requires a recent enough kernel (6.2+ on Ubuntu-family distributions), OpenRM driver 570.x or newer, and specific registration keys enabled. Two caveats: upstream P2PDMA does not currently support RAID0 or NVMe multipathing, which still require the nvidia-fs path. Check kernel capability with:

cat /proc/kallsyms | grep -i p2pdma_pgmap_ops
# Output present => upstream P2PDMA is available

3. When a direct path actually holds: installing packages is not enough

"Installed" and "active" are different states. Whether an I/O takes the direct path depends simultaneously on the storage device, PCIe topology, filesystem and alignment.

3.1 Filesystem and storage support matrix

Storage form	GDS support	Notes
Local NVMe + ext4 / xfs	Supported, the most mature path	O_DIRECT, 4KB aligned
NVMe-oF (RDMA)	Supported	Remote NVMe reaches GPU memory over an RDMA NIC
Lustre	Supported	For small I/O, POSIX can have lower latency; posix_gds_min_kb sets the cutoff
IBM Spectrum Scale (GPFS)	dmabuf / compat path	Capabilities vary by distro and client version
BeeGFS	Supported, with dedicated client configuration	gdscheck shows compat when it is not configured
WekaFS, VAST/DDN NFS-family	Supported	Depends on vendor plugins and dynamic routing
Plain NFS / unsupported filesystems	Automatic compat fallback	Functional, no direct-path benefit

3.2 Topology and hardware conditions

• The GPU and the NVMe drive (or storage NIC) should sit under the same PCIe switch / root port. Cross-root-complex P2P has higher latency; cuFile dynamic routing then stages through the nearest GPU over NVLink instead — still better than a pure CPU path, but not optimal;
• PCIe ACS (Access Control Services) forced on at an intermediate switch blocks P2P transactions, and gdscheck reports exactly which switch is responsible;
• BAR1 mapping window must be large enough. On older cards or platforms without Resizable BAR, only part of video memory is mapped and I/O beyond the window uses bounce buffers;
• Remote storage requires an RDMA-capable NIC (IB or RoCE); without it the network path cannot do P2P;
• 4KB alignment is mandatory: file offsets and I/O sizes must be page-aligned; unaligned writes fall back to compat.

3.3 An easy-to-miss boundary: managed memory

Migratable memory from cudaMallocManaged, plus system malloc/stack memory, is not supported on the GDS P2P path — cuFile services it through internal bounce buffers. The direct-path benefit applies only to device memory from cudaMalloc / cuMemAlloc / cuMemCreate.

4. Compatibility mode: no errors, but fake performance

Compatibility mode is GDS's most misleading feature: every cuFile call returns success while the data actually travels through POSIX and the CPU. With the default allow_compat_mode: true, the library falls back silently instead of failing.

The official fallback triggers are:

1. Hardware configuration outside the supported list;
2. Capable GPU but insufficient BAR space exposed;
3. nvidia-fs.ko not loaded or failed to load;
4. Unsupported Linux distribution;
5. Filesystem not supported by GDS;
6. Network links without RDMA, or a configuration short of the direct-path optimum;
7. Writes that are not 4KB aligned.

The acceptance criterion is therefore not "the application ran", but proof that traffic took the nvfs path. Two ways to verify:

# 1. Level-3 cuFile statistics: growing nvfs counters mean direct I/O; posix means compat
gds_stats -p <pid> -l 3

# 2. The same check applies to KvikIO/Python workloads — watch the two sources separately

If every counter lands under posix, walk the seven triggers above before suspecting application code.

5. Installation and pre-production health checks

On RHEL/Rocky-family systems install three packages from the repository (same names with apt on Ubuntu):

# nvidia-gds is the meta-package pulling the kernel module and user-space libraries;
# gds-tools provides gdsio/gdscheck/gds_stats
dnf install -y nvidia-gds nvidia-fs-dkms gds-tools

# Load the kernel module (required for the classic path; optional under P2PDMA)
modprobe nvidia-fs
lsmod | grep nvidia_fs

Run two levels of gdscheck before putting any workload on top:

# Platform: GPUs, driver, nvidia-fs, PCIe ACS, BAR window, RDMA devices
gdscheck -p

# Mounts: whether each filesystem mount is recognized as supported or compat
gdscheck -m

Read the -m output line by line: two clients on the same parallel filesystem can differ between supported and compat purely because of mount options, and this is the checkpoint where such problems are cheapest to catch.

6. gdsio: benchmarking the two paths with identical parameters

gdsio, shipped in gds-tools, is the GDS counterpart of fio, focused on storage-to-GPU transfers. Key options:

Option	Meaning
-f / -D	Target file / directory (directory mode with -w for many files)
-d	Destination GPU index
-s	Dataset size
-i	Per-I/O size (the key variable for large-vs-small I/O gains)
-x 0	GDS direct path (GPUD)
-x 2	Conventional POSIX via CPU (r1.12; older releases use 1 — confirm with gdsio -h)
-I 1	Write; read is the default / 0
-w	Number of I/O threads
-T	Duration in seconds

A minimal A/B experiment:

# Direct path: 1 GiB file, 1 MiB I/Os, 8 threads, reading into GPU 0
gdsio -f /mnt/nvme/gds.test -d 0 -s 1G -i 1M -w 8 -x 0

# Same parameters through the CPU-mediated path
gdsio -f /mnt/nvme/gds.test -d 0 -s 1G -i 1M -w 8 -x 2

Read three fields in the output: Throughput, Avg_Latency and ops. NVIDIA's own documentation includes a two-drive NVMe RAID0 example: with each drive rated around 3.4 GB/s sequential read, direct gdsio sustained about 6.5 GiB/s at roughly 1.2 ms average latency — effectively saturating the hardware.

Always quantify the CPU cost in the same experiment. In a second terminal while the benchmark runs:

iostat -cxzk 1

Two things matter: under the direct path, NVMe rkB/s saturates while %system is markedly lower; under compat, user/system CPU is substantially higher. Comparing throughput without CPU comparison understates GDS in production, where the saved CPU cycles were supposed to run data decoding.

Do not assume small I/O benefits

On network filesystems such as Lustre, very small I/Os can be faster over POSIX because the fixed cost of P2P setup and address translation does not amortize. The lustre:posix_gds_min_kb setting in cufile.json expresses exactly this cutoff: below the threshold use POSIX, above it use GDS. Sweep -i from 4K to 1M during benchmarking and set the value from the measured knee, not from intuition.

7. Application integration: from cuFile to PyTorch

7.1 The native cuFile call sequence

C/C++ applications use a fixed lifecycle:

cuFileDriverOpen()                    # process-wide initialization
  cuFileBufRegister(gpuPtr, size)    # register a reusable GPU buffer
  cuFileHandleRegister(fd)           # open and register the file handle
    cuFileRead / cuFileWrite         # direct transfers; stream variants allow async ordering
  cuFileHandleDeregister()
  cuFileBufDeregister()
cuFileDriverClose()

Build with -lcufile -lcuda -lcudart; the header is /usr/local/cuda/include/cufile.h. The Stream API subset places I/O on a CUDA stream, enforcing "kernel produces data → I/O flushes" or "I/O completes → kernel starts" ordering, and multiple streams can drive multiple DMA engines concurrently.

7.2 PyTorch 2.7: the torch.cuda.gds prototype

Starting with PyTorch 2.7 (CUDA 12.6+), torch.cuda.gds offers thin wrappers over cuFile, currently marked prototype. Its most direct use is accelerating torch.save / torch.load checkpoints:

• GdsFile wraps a file descriptor and adds O_DIRECT automatically;
• save_storage / load_storage bind to the synchronous cuFileWrite / cuFileRead;
• gds_register_buffer wraps cuFileBufRegister for buffers reused across iterations;
• GDS requires 4KB alignment, configured via torch.utils.serialization.config.save.storage_alignment.

Large-model checkpoints — tens or hundreds of GB in a single file — are exactly the large sequential I/O pattern where GDS pays off most. Because the API is still a prototype, validate correctness and fallback behavior on shadow nodes before rolling it out with a PyTorch upgrade.

7.3 The Python data stack: KvikIO and DALI

• KvikIO is the Python I/O layer used across the CUDA-X data-science stack (cuDF, cuCIM), calling cuFile underneath for columnar Parquet/ORC ingestion and pathology image pipelines. Note the two-layer decision: KvikIO decides whether to call cuFile, and cuFile decides nvfs versus posix — gds_stats remains the source of truth;
• DALI exposes a GDS switch in its file-reading pipeline, useful for image training that wants even the pre-decode raw reads off the CPU;
• For custom dataloaders, evaluate KvikIO before reaching for the C cuFile API; let a mature library own path selection and compat fallback.

8. Dynamic routing: when the GPU and storage NIC live in different PCIe domains

For network filesystems, forcing cross-root-port P2P when the GPU and storage NIC do not share a root port can be slower than the alternative. For each mount, cuFile precomputes the GPU with the smallest PCI distance to the available storage NICs:

1. Target GPU and NIC under the same PCIe switch → direct P2P, no routing involved;
2. Different switches → pick the nearest GPU as a bounce target, P2P into it, then copy device-to-device over NVLink to the target GPU;
3. CPU system memory stays out of the path throughout — the CPU still only issues commands.

This explains a familiar field observation: gdsio throughput shifts noticeably after GPU slots are re-cabled. Check topology distance in gdscheck -p and pair NICs with GPUs by slot before blaming the filesystem.

9. Five common mistakes

1. Treating package installation as activation. With silent compat fallback enabled by default, only nvfs counters in gds_stats prove the direct path;
2. Ignoring 4KB alignment. Misaligned offsets or lengths quietly route writes through POSIX with no application error;
3. Forcing the new P2PDMA mode on RAID0 / NVMe multipath. Unsupported by the upstream kernel today; these forms still need the nvidia-fs path;
4. Watching throughput but not CPU usage. Half the GDS dividend is freeing CPUs for dataloaders — skipping the iostat comparison measures only half the benefit;
5. Forcing every small I/O through GDS. Fixed costs do not amortize on network filesystems; use posix_gds_min_kb at the measured knee.

10. Production acceptance checklist

1. lsmod | grep nvidia_fs shows the module (except under P2PDMA); gdscheck -p reports no ACS blockage and ready BAR/RDMA state;
2. gdscheck -m shows every target mount as supported, not compat;
3. gdsio A/B with -x 0 versus the traditional path, archiving throughput, latency and iostat CPU figures;
4. Sweep I/O sizes from 4K to 1 MiB and fix the small-I/O cutoff;
5. Under the real workload, gds_stats -p <pid> -l 3 shows nvfs growing and posix flat;
6. For PyTorch checkpoints, verify GDS-written files reload correctly with plain torch.load and that the fallback path works;
7. Inject one failure — unload nvidia-fs or point at an unsupported mount — and confirm the application degrades to compat rather than crashing.

Three closing lines

• GDS removes the CPU-memory hop: DMA reaches video memory directly, paying off in bandwidth, latency and CPU utilization at once.
• Installed is not active: any miss in the support matrix, topology or alignment silently falls back to compat; the nvfs/posix split in gds_stats is the only credible judge.
• Large sequential I/O wins most; small random I/O needs a measured knee: checkpoints and dataset preload are the main battlefield, and cutoff thresholds belong to benchmark data, not rules of thumb.

Further reading

• Training Data Storage Selection: Object Store, Parallel FS and Local NVMe
• Lustre 2.15.7 Deployment Guide (EL8)
• Enterprise BeeGFS Deployment and Tuning Guide
• InfiniBand Performance Tuning for AI Workloads
• Pretraining Engineering: Checkpoints, Fault Tolerance and Stability