Domestic GPU/NPU Porting: Migrating from CUDA to Heterogeneous Accelerators

Summary: Domestic compute (Ascend, Cambricon, Moore Threads, Tianshu...) is entering real production clusters, but "moving CUDA code over" is not just swapping a compiler: ecosystem, operators, memory management, and performance characteristics all differ. This post gives a landing-able migration route: inventory CUDA dependencies → choose a path → map operators → accept performance → run dual stacks.

0. Experiment Goals

• Inventory your application's real CUDA dependencies (this decides the migration path)
• Master three migration routes: framework-level migration / operator rewriting / dual-stack coexistence
• Learn Ascend and Cambricon adaptation points
• Establish performance acceptance: correctness + precision + performance-regression tolerance

1. Inventory First: What Does Your App Actually Depend On

# Inventory toolchain deps: key CUDA symbols in source
grep -rE "cudaMalloc|cudaMemcpy|cublas|nccl|torch.cuda|numba.cuda" --include="*.py" --include="*.cu" --include="*.cpp" . | head -20

# Check runtime dynamic-library dependencies
ldd /path/to/your_binary | grep -iE "cuda|cudnn|cublas|nccl"

Step-by-step:

• Dependencies fall into four classes: framework APIs (torch.cuda, easiest), CUDA runtime APIs (cudaMalloc etc., medium), library calls (cuBLAS/cuDNN/NCCL, provided by the new stack), hand-written kernels (.cu/.pyx, hardest).
• Bottom line first: 80% of applications sit at "framework API + library calls"—framework-level migration suffices; only deep-learning kernel researchers need to touch .cu.

2. Three Migration Routes

Route	Approach	Cost	Applies to
A. Framework-level	PyTorch models + torch_npu/torch_mlu adaptation layers	low (change device & a few APIs)	the vast majority of train/infer apps
B. Operator rewriting	Triton/CANN/MLU custom ops replacing handwritten CUDA	medium-high	hotspot models with custom kernels
C. Dual-stack coexistence	CUDA and domestic cards in separate pools, routed by load	medium	migration transition, disaster recovery

How it works:

• Route A core: the adaptation layer maps torch.cuda to the domestic runtime (Ascend calls it CANN, Cambricon CNToolkit) and provides counterparts for torch.compile/AMP/NCCL (HCCL/CNCL communication libs). Most models only need changing device="cuda" → the name the adapter exposes.
• Route C (transition standard): the CUDA pool runs existing business, the domestic pool runs new business/backup; Slurm/K8s route by partition or labels (see KB Slurm Heterogeneous Scheduling and K8s GPU Scheduling).

3. Ascend Adaptation Points (Example)

# Install CANN + torch_npu
pip install torch_npu
export ASCEND_HOME=/usr/local/Ascend/ascend-toolkit

# Code change example: torch.cuda → Ascend device
import torch
import torch_npu                      # registers the npu device

device = "npu:0"                      # was "cuda:0"
model = model.to(device)
x = x.to(device)

Engineering notes (Ascend):

• torch_npu provides the npu:0 device, but not every operator has an NPU implementation—unsupported ops error out or fall back to CPU, dropping performance. Check the torch_npu operator-support list first.
• Mixed precision: Ascend supports FP16/BF16; the AMP path resembles CUDA; the communication library is HCCL (NCCL counterpart)—DDP code barely changes.
• Inference deployment: MindSpore/MindIE is the other Ascend-ecosystem path (pick one at the framework layer).

4. Cambricon Adaptation Points (Example)

# CNToolkit + torch_mlu
pip install torch_mlu
export LD_LIBRARY_PATH=$CNTOOLKIT_HOME/lib64:$LD_LIBRARY_PATH

# Code change example
import torch_mlu                      # registers the mlu device
device = "mlu:0"                      # Cambricon MLU device

Engineering notes (Cambricon):

• MLU operator coverage and performance profiles differ from NVIDIA: small-op scenarios are relatively weak; large GEMM/convolution ops are close. After porting, run an operator-level profile first—don't guess from an overall benchmark.
• The communication library is CNCL; torch.compile support maturity varies by version (Triton backend adaptation differs).

5. Performance Acceptance: What "Migrated" Means

# Acceptance in three steps: correctness → precision → performance
# 1) Correctness: same input, NVIDIA vs domestic outputs align
python compare_outputs.py --ref cuda --test npu --tolerance 1e-3

# 2) Precision: training convergence curves compared (loss shape consistent)
# 3) Performance: same batch/model, compare step time and throughput
python bench.py --device npu --batch 32

Acceptance standard suggestions:

Item	Standard
Output correctness	diff vs CUDA < 1e-3 (FP32) / same magnitude (AMP)
Training convergence	loss trends consistent, final accuracy gap < 1%
Performance	60-90% of the CUDA baseline (by scenario; operator profile decides)
Stability	72h run without OOM/hang

Key insight: acceptance is not "it runs" but "it is comparable". Quantify performance regression to the operator level—know where it dropped and why (coverage/memory bandwidth/communication library) before optimizing.

6. Engineering Takeaways

1. Inventory before acting: 80% of apps migrate at the framework layer (route A); don't start writing operators immediately.
2. The adapter is the core: torch_npu/torch_mlu map torch.cuda to domestic runtimes; most code only changes device.
3. Operator coverage is the first trap: run the official support list; unsupported ops falling back to CPU drop performance.
4. Communication-library alignment: NCCL→HCCL (Ascend)/CNCL (Cambricon); DDP code barely changes.
5. Operator profile before conclusions: an overall benchmark can't tell you where the drop is; operator-level profiles can.
6. Dual-stack coexistence transition: CUDA and domestic cards in separate pools, route by load, switch gradually.
7. Quantified acceptance: correctness/precision/performance/stability; performance regression quantified to the operator level.

7. Further Reading

• Ascend CANN docs
• Cambricon CNToolkit docs
• KB: 07. Heterogeneous Computing · Slurm Heterogeneous Scheduling · K8s GPU Scheduling · CUDA MPS