09. Deep Learning Frameworks

Chapter focus: This chapter covers runtime behavior and engineering deployment practices for mainstream frameworks.

Contents

• PyTorch environment and container setup
• training workload operations
• integration with scheduling systems

🧭 Reading Map

• Prerequisites: 09. AI Compiler — the framework operator layer builds on compiler output
• Role: closing chapter of Part 3, L3 — PyTorch runtime: Autograd/Dispatch, DDP/FSDP, container envs and Triton kernels
• Order: framework internals (Autograd/Dispatch) → distributed engines (DDP/FSDP) → environments (Docker) → hand-written kernels
• Next: 11. Pre-trained Models — with frameworks in place, build large models

Articles

PyTorch Autograd Deep Dive: From Dynamic Computation Graphs to Backpropagation

Step-by-step dissection of the PyTorch autograd engine with runnable experiment code: requires_grad, graph construction, backward execution, gradient accumulation and detach, plus memory and mixed-precision engineering lessons.

阅读文章 →

PyTorch Internals: Tensor Storage, Views, and the Dispatch Mechanism

Reproducible experiments dissecting PyTorch internals: the storage/view/stride data model, the dispatch mechanism beyond autograd, and device/dtype routing, with engineering takeaways.

阅读文章 →

PyTorch Distributed Training Engine Dissected: DDP to FSDP

gradient buckets, comm-compute overlap, sharding, selection matrix.

阅读文章 →

Training Framework Ledger: Megatron vs DeepSpeed

Four-layer abstraction, Megatron TP/PP/SP, DeepSpeed ZeRO/Offload, selection matrix & combos.

阅读文章 →