CUDA MPS in Practice: Multi-Process GPU Sharing for Small-Task Concurrency

Summary: Running dozens of small inference tasks on one GPU, default process-level concurrency "tramples" each other—every process allocates its own CUDA context and takes turns on the SMs. CUDA MPS (Multi-Process Service) lets multiple processes share one CUDA context and execute concurrently—a complementary way of partitioning compute alongside MIG. This post covers when to use MPS, how to configure it, the gains, and the traps.

0. Experiment Goals

• Understand the positioning difference between MPS and MIG (software sharing vs hardware isolation)
• Master two MPS deployment modes: console mode (experiments) and daemon mode (production)
• Quantify MPS throughput gains with a concurrency experiment (small-task scenario)
• Know MPS boundaries and traps: fault isolation, memory cap, timeouts

1. Why MPS: The "Trampling" Problem of Small-Task Concurrency

How it works:

• Without MPS, each CUDA process on a GPU initializes its own full CUDA context (several GB of memory), and SM scheduling is process-level round-robin—dense small-task scenarios (dozens of inference/preprocessing workers) get poor throughput and waste memory.
• The core of MPS: multiple processes share a single CUDA context through the MPS server; GPU scheduling granularity goes from "process" to "thread/task"—small tasks truly saturate SMs in parallel, and only one context is stored.

MPS vs MIG (key comparison):

Dimension	CUDA MPS	MIG (Multi-Instance GPU)
Isolation	software-level (shared context)	hardware-level (SM/memory/bandwidth partition)
Best for	dozens of small tasks, throughput-first	multi-tenant strong isolation
Memory	one shared context, saves memory	hard partition per instance
Fault impact	one crash can affect the group	fully isolated between instances
GPUs	all NVIDIA GPUs	A100/H100 etc.

Conclusion: MIG "cuts the cake", MPS "sets the table". In a mixed compute pool, use both: MIG for hard isolation between tenants, MPS for throughput within a tenant.

2. Console Mode: Up in One Minute

# Check whether the GPU supports MPS (returns Enabled by default)
nvidia-smi -q | grep -A 2 "MPS"

# Console mode: put all CUDA processes of this shell into MPS (for experiments)
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-mps-log
nvidia-cuda-mps-control -d            # start the MPS control daemon in background

# Verify: MPS process is running
ps aux | grep "mps" | grep -v grep

# Exit after the experiment
echo quit | nvidia-cuda-mps-control

Step-by-step:

• CUDA_MPS_PIPE_DIRECTORY: the MPS control pipe directory—all processes that want to share must point to the same directory. This is the key to joining the same MPS server.
• nvidia-cuda-mps-control -d: starts the server in daemon mode; afterwards any CUDA process with CUDA_MPS_PIPE_DIRECTORY set automatically joins.
• Note: MPS only applies to processes that set the env var; processes without it run independently outside the MPS server, unaffected.

3. Daemon Mode: Production Deployment

# systemd-managed MPS daemon (production standard)
cat > /etc/systemd/system/nvidia-mps.service <<'EOF'
[Unit]
Description=NVIDIA MPS control daemon
After=network.target

[Service]
Type=forking
ExecStart=/usr/bin/nvidia-cuda-mps-control -d
ExecStop=/usr/bin/nvidia-cuda-mps-control --kill
Restart=on-failure
Environment=CUDA_MPS_PIPE_DIRECTORY=/var/run/nvidia-mps
Environment=CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now nvidia-mps

Engineering notes:

• In production, manage the MPS daemon with systemd: auto-start on boot, auto-restart on crash.
• Joining parties (inference service containers/processes) uniformly inject CUDA_MPS_PIPE_DIRECTORY=/var/run/nvidia-mps; in K8s use env or envFrom to distribute.
• MPS memory cap: by default MPS clients share all memory; set CUDA_MPS_ACTIVE_THREAD_PERCENTAGE to limit the SM percentage a single client can occupy (0-100) for coarse quota control.

4. Experiment: MPS Gains for Small-Task Concurrency

# Design: 16 concurrent small inference workers (batch=1), MPS on/off
# Before: watch GPU utilization fluctuate and tasks queue
# After: SM utilization stable at saturation, total throughput up significantly
nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1

Results (sample magnitude · 8 inference processes · single GPU):

Scenario	GPU utilization	Total throughput	CUDA contexts
No MPS (default)	~35% fluctuating	baseline 1.0×	one per process (N copies)
MPS on	~92% stable	~2.3×	one shared
MPS + 50% SM cap	~50% capped	~1.4×	one shared

Conclusion: MPS gains are significant for small-task-dense scenarios (2×+ throughput); large tasks (a single request saturating the whole card) do not need MPS—prefer MIG isolation or whole-card exclusivity. The MPS SM percentage cap suits "multiple tenants sharing one card + quota management".

5. Traps and Boundaries (production must-read)

1. Weak fault isolation: an MPS client crash can affect other processes in the group (shared context). Do not mix critical services with experimental tasks in one MPS group; fall back to MIG for hard isolation.
2. Not all APIs supported: some CUDA APIs (certain multi-process interop, graphics APIs) have compatibility limits under MPS—run an app self-test before rollout.
3. Multi-user permissions: for different users to join one MPS server, the directory permissions must be right (/var/run/nvidia-mps 777 or group-granted), otherwise "cannot find MPS pipe".
4. Mutually exclusive with MIG: with MIG enabled, MPS behaves differently (usable inside an instance but different granularity); don't mix two partitioning strategies at once.
5. Container scenario: Pods sharing a GPU in K8s must mount the MPS pipe directory (/var/run/nvidia-mps + socket) and use shared volumes/hostPID; NVIDIA's official device plugin has MPS support.

6. Engineering Takeaways

1. Choose by scenario: small-task-dense → MPS; multi-tenant strong isolation → MIG; nestable (MPS inside a MIG tenant).
2. Manage the MPS daemon with systemd in production: auto-start + crash recovery.
3. Uniformly inject CUDA_MPS_PIPE_DIRECTORY to joining parties; distribute via envFrom in K8s.
4. Use CUDA_MPS_ACTIVE_THREAD_PERCENTAGE for coarse quotas; don't run full sharing bare.
5. Separate MPS groups for critical services vs experiments; weak fault isolation is MPS's inherent weakness.
6. Validate first: MPS compatibility is per-app; self-test before rollout.
7. Monitor GPU utilization: MPS in effect shows as utilization moving from "fluctuating" to "stably saturated".

7. Further Reading

• NVIDIA CUDA MPS docs
• NVIDIA MIG docs
• KB: 07. Heterogeneous Computing · MIG Partitioning · Slurm Heterogeneous Scheduling