llm-algo-leetcode | 大模型算法与系统教程
Runnable notebooks for LLM algorithms and systems.
面向大模型算法与系统的可运行 Notebook 教程。
主学习路线 / Core Routes
训练微调 / Fine-tuning · 推理优化 / Inference · 显存优化 / Memory · 算子与编译优化 / Operators & Compilers（建设中 / In progress）
专题入口 / Topic Paths
量化与压缩 / Quantization · Profiling · 通信与并行 / Parallelism · 查看全部专题 / All topics
中文版 (Chinese) | English Version
中文版
🎯 项目简介
这是一个以 Notebook 为载体的大模型算法与系统教程。教程从 Part 02 的 PyTorch 算法实践出发，按需回补 Part 00 / Part 01 的基础，并进一步延伸到 Part 03 的 Triton 算子开发和 Part 04 的 CUDA 与系统优化。
内容围绕可运行实现、实验验证和专题化学习路线组织，帮助学习者逐步理解模型如何计算、如何训练、如何推理，以及性能问题如何被测量和优化。
✨ 项目特点
- Notebook-first：每节围绕可运行代码、题目区、答案区和基础验证展开，适合边学边改。
- 按目标进入：既支持从 Part 02 开始建立算法实践，也支持按训练、推理、显存、量化和并行专题跳读。
- 算法到系统贯通：沿着
PyTorch → Triton → CUDA逐步下探，连接模型实现、算子优化、显存管理、通信与系统性能。 - 用证据做项目判断：通过 benchmark、profiling、真实 GPU 和 inference backend 实验，对吞吐、延迟、显存、质量和成本进行比较。
👥 适合对象
- 大模型算法学习者：希望通过 PyTorch Notebook 理解 Transformer、训练、微调、推理和模型压缩。
- 性能与系统学习者：希望进一步学习显存、Profiling、通信、Triton、CUDA 和 GPU 优化。
- 项目实践者：希望通过 benchmark、真实 GPU 和 inference backend 实验，建立可复现、可比较的工程判断。
🌐 教程总览
教程提供两种阅读方式：按 Part 00 -> Part 04 逐层推进，或按训练、推理、显存、算子与编译优化等主路线进入，再通过量化、Profiling、并行等专题补充能力。推荐先从 Part 02 建立算法实践感，再按需要回补 Part 00 / Part 01，最后进入 Part 03 / Part 04；路线四目前处于建设阶段，Part 05 作为扩展预留。
topic_discussion 用于组织跨 Part 的专题路线，team_study 用于沉淀共学记录。页面顶部提供常用专题入口，下面的资产总览和专题总览分别说明主线结构与专题覆盖范围。
📚 查看完整资产总览
这套教程不要求从 00 开始按顺序硬读。00 主要是前置补齐区，如果你已有基础，可以直接从最相关的部分开始；下面这张表会直接告诉你：每一部分学什么、包含哪些组、适合谁、当前进度如何。
🧭 查看完整专题总览
| 层级 | 入口 | 覆盖范围 | 内容定位 | 适合对象 |
|---|---|---|---|---|
| 主学习路线 | 监督微调专题 | Part 01-02 | 导读：intro；正文：casebook。SFT、LoRA、训练控制和项目交付。 | 想从 SFT 一路走到 LoRA 项目闭环的学习者。 |
| 主学习路线 | 推理优化专题 | Part 01-02 | 导读：intro；正文：casebook。FlashAttention、解码、PagedAttention、cache 与 benchmark。 | 想系统理解推理加速路径的学习者。 |
| 主学习路线 | 显存优化专题 | Part 00-02 | 导读：intro；正文：casebook。VRAM、activation、checkpointing、offload 和 trade-off。 | 想系统优化显存和端到端性能的学习者。 |
| 主学习路线（建设中） | 算子与编译优化专题 | Part 01-04 | 导读：intro；正文：casebook。算子、Triton、CUDA、图优化、fusion、lowering、schedule 和 backend 约束。 | 想从算子实现走向编译、Kernel 和端到端性能优化的学习者。 |
| 横切支撑专题 | 量化与压缩专题 | Part 00-03 | 导读：intro；正文：casebook。PTQ、QAT、GPTQ、AWQ、FP8 与部署决策。 | 想同时考虑精度、显存、吞吐和部署取舍的学习者。 |
| 横切支撑专题 | 通信与并行专题 | Part 01-02 | 导读：intro；正文：casebook。NCCL、AllReduce、ZeRO、PP、TP 和并行验证。 | 想理解多卡训练和通信边界的学习者。 |
| 横切支撑专题 | Profiling 专题 | Part 00-02 | 导读：intro；正文：casebook。性能取证、trace 阅读、回归验证和行动决策。 | 想系统补性能意识与排障方法的学习者。 |
| 横切支撑专题 | 后训练与对齐专题 | Part 01-02 | 导读：intro；正文：casebook。RLHF、DPO、GRPO、偏好数据与项目收口。 | 想从 SFT 继续走到偏好优化与对齐的学习者。 |
| 基础支撑专题 | 反向传播与训练机制专题 | Part 01-02 | 导读：intro；正文：casebook。autograd、backward、checkpointing、offload 与训练节奏。 | 想补训练机制底座的学习者。 |
| 基础支撑专题 | 大模型架构专题 | Part 01-02 | 导读：intro；正文：casebook。结构演进、代表模型和 MoE / 稀疏化。 | 想补模型结构背景与横向对照的学习者。 |
🤝 查看共学沉淀
| 模块 | 覆盖范围 | 内容定位 | 适合对象 | 状态 |
|---|---|---|---|---|
组队学习专题 | 不固定 | part2_l1_202606 / part2_l1_202607 / part2_l2_202607 | 想通过共学沉淀知识、题目与复盘记录的学习者。 | 🛠 建设中 |
🆕 查看更新时间线
- 2026-08-22：完成 README 首页导航、项目简介与项目特点重写；收口横向专题的五层 Infra 结构与跨专题边界；完成 66、73、75、76 等真实 backend / GPU 项目的结果保存、环境说明和基础验证，并同步文档入口与链接检查。
- 2026-08-17：统一 Part 01 导读与组页口径，收紧横向专题结构和学习路线表达，补充 Part 02 项目页与图解资产审计，并明确统一验证入口。
- 2026-07-10：整理首页教程总览与状态列，校正 Part 00 / Part 01 的组名、节数和导航状态。
- 2026-06-26：重构首页教程总览、状态列和学习路径，明确 Part 00-04、横向专题与共学记录的关系。
- 2026-06-15：推进第零部分 / 第一部分的分组与导读收口，统一部分级导航，并完成网页底部评论区接入 GitHub Discussions，同时持续扩展第一部分的正文、桥接页与 Notebook 结构。
- 2026-06-13：修复 dead link，并为未完成页面补充占位页，避免学习入口出现 404。
- 2026-04-21：更新 Colab 徽章链接，统一指向官方
datawhalechina仓库。 - 2026-04-20：上线站点首页与部分导学；新增第零部分前置知识与第一部分练习内容，完善在线阅读入口与学习路径。
- 2026-04-18 ~ 2026-04-19：集中重构第二部分 / 第三部分内容，优化 Notebook、答案区与算子实现说明。
- 2026-04-02：完成教程核心 Notebook、文档与测试脚本的初始搭建。
路径兼容说明：第三部分已从
03_CUDA_and_Triton_Kernels更名为03_Triton_Kernels，CUDA / 系统优化内容拆分到第四部分。旧网页路径会保留迁移入口，建议新链接统一使用03_Triton_Kernels。
🚀 快速开始
推荐从 Part 02 开始：先通过 PyTorch Notebook 建立算法实现感，再根据遇到的知识缺口回补 Part 00 / Part 01，最后进入 Part 03 / Part 04 的 GPU 底层优化。也可以按训练、推理或显存等目标直接进入对应专题。运行 Notebook 前，请先查看 使用指南 和对应小节中的环境说明。
查看在线、本地与 CNB 的具体使用方式
方式 1：在线阅读
访问在线站点：
https://datawhalechina.github.io/llm-algo-leetcode/
学习步骤：优先选择 Part 02 或对应专题，再从 📖 完整导学 进入对应 intro.md，最后进入目标 group；遇到知识缺口时回补 Part 00 / Part 01。
适合：
- 先看目录再决定从哪一部分切入
- 先读部分导学，按目标跳转到对应 group
- Part 00 / 01 / 02 的大多数基础练习可以直接用 Colab CPU 跑
- 真实推理、训练和显存实验需要 Colab GPU 或本地 NVIDIA GPU
- Part 03 / 04 需要 Colab GPU runtime
方式 2：本地学习
git clone https://github.com/datawhalechina/llm-algo-leetcode.git
cd llm-algo-leetcode
conda env create -f environment.yml
conda activate llm_algo
jupyter lab学习步骤：在仓库中优先进入 Part 02 或对应专题目录，先阅读 intro.md，再打开目标 Notebook；遇到知识缺口时回补 Part 00 / Part 01。
适合：
- 想在本地完整跑 Part 00 / 01 / 02 的 Notebook
- 想自己控制 Python / PyTorch / CUDA 版本
- 想做更稳定的离线调试
- Part 03 / 04 需要本地 NVIDIA GPU
方式 3：CNB 统一环境
如果你希望和仓库当前推荐环境保持一致，可以使用 CNB 统一环境入口。
适合：
- 团队协作
- 统一实验镜像
- 需要减少本地环境差异
- Part 00 / 01 / 02 可以用 CNB CPU
- Part 03 / 04 需要 CNB GPU 会话
CNB 的具体使用方式和适用范围见 使用指南。
学习步骤：进入 CNB 会话后，优先从 Part 02 或对应专题的 intro.md 开始，再按导学进入目标 Notebook；基础不足时回补 Part 00 / Part 01。
📖 更多资源
- 使用指南 - 环境与学习方式
- 贡献指南 - 如何参与项目开发和测试
- 维护与发布手册 - 部分、链接、测试与发布的维护约定
- 统一验证入口 -
verify.py part0_1、verify.py part2等标准验证命令 - 自动化测试脚本索引 - 各类验证脚本入口
👨💻 贡献者名单
| 姓名 | 职责 | 简介 |
|---|---|---|
| lynn_jingjing | 项目发起人 | 一个算法工程师 |
📄 许可声明
本仓库所有 .ipynb 文件中的文字内容（Markdown 单元格、公式、图示说明）采用 CC BY 4.0 协议；代码内容（Code 单元格、可执行实现）采用 Apache-2.0 协议。使用、转载、改编时，请按单元格类型分别遵守对应协议。文字协议见 LICENSE，代码协议见 LICENSE-CODE。
English Version
📄 License Notice
All .ipynb files in this repository are mixed-content notebooks: Markdown cells (tutorial text, formulas, and figure captions) are licensed under CC BY 4.0, while Code cells (executable implementations) are licensed under Apache-2.0. Please comply with the corresponding license by cell type when using, redistributing, or adapting this repository. See LICENSE for text and LICENSE-CODE for code.
🎯 Project Introduction
This is a notebook-based tutorial on LLM algorithms and systems. It starts with PyTorch practice in Part 02, lets learners backfill the prerequisites in Part 00 / Part 01 as needed, and extends to Triton kernel development in Part 03 and CUDA and system optimization in Part 04.
The tutorial is organized around runnable implementations, experimental validation, and topic-based learning paths. It helps learners understand how models compute, train, and serve requests, and how to measure and optimize performance problems.
✨ Features
- Notebook-first: Each lesson is organized around runnable code, exercises, answer cells, and basic validation.
- Goal-oriented entry: Start from Part 02 for implementation practice, or follow topic paths for training, inference, memory, quantization, and parallelism.
- From algorithms to systems: Follow the
PyTorch → Triton → CUDApath across model implementation, kernel optimization, memory, communication, and system performance. - Evidence-based projects: Use benchmarks, profiling, real GPU runs, and inference backends to compare throughput, latency, memory, quality, and cost.
👥 Suitable For
- LLM Algorithm Learners: Use PyTorch notebooks to understand Transformers, training, fine-tuning, inference, and model compression.
- Performance and Systems Learners: Study memory, profiling, communication, Triton, CUDA, and GPU optimization.
- Project Practitioners: Build reproducible engineering judgment through benchmarks, real GPU runs, and inference backends.
🌐 Tutorial Overview
This tutorial offers two ways to read: follow the Part 00 -> Part 04 progression, or enter through the main routes for training, inference, memory, and operator/compiler optimization, then use quantization, profiling, and parallelism as supporting topics. We recommend starting with Part 02 to build implementation intuition, then backfilling Part 00 / Part 01 as needed before moving to Part 03 / Part 04; Route 4 is currently under construction, and Part 05 is reserved for future expansion.
topic_discussion organizes cross-Part topic paths, while team_study stores collaborative-learning records. The top of the page provides common topic entries; the asset and topic overviews below describe the main structure and topic coverage.
📚 View the complete asset overview
You do not need to start from 00 in strict order. 00 is the prerequisite lane; if you already have the background, jump directly to the part that matches your goal. The table below summarizes each part, its groups, its audience, and its status.
🧭 View the complete topic overview
| Layer | Entry | Coverage | Content Positioning | Suitable For |
|---|---|---|---|---|
| Main Study Path | Fine-Tuning Training Topic | Part 01-02 | Guide: intro; casebook: casebook. SFT, LoRA, training control, and project delivery. | Learners who want to go from SFT to a LoRA project closure. |
| Main Study Path | Inference Optimization Topic | Part 01-02 | Guide: intro; casebook: casebook. FlashAttention, decoding, PagedAttention, cache, and benchmark. | Learners who want practical inference acceleration. |
| Main Study Path | Memory and Performance Tuning Topic | Part 00-02 | Guide: intro; casebook: casebook. VRAM, activation, checkpointing, offload, and trade-offs. | Learners who want to optimize memory usage and end-to-end performance. |
| Main Study Path (In Progress) | Operator and Compiler Optimization Topic | Part 01-04 | Guide: intro; casebook: casebook. Operators, Triton, CUDA, graph optimization, fusion, lowering, scheduling, and backend constraints. | Learners who want to move from operator implementation to compiler, kernel, and end-to-end optimization. |
| Cross-Cutting Topic | Quantization Topic | Part 00-03 | Guide: intro; casebook: casebook. PTQ, QAT, GPTQ, AWQ, FP8, and deployment decisions. | Learners balancing accuracy, memory, throughput, and deployment cost. |
| Cross-Cutting Topic | Communication and Parallelism Topic | Part 01-02 | Guide: intro; casebook: casebook. NCCL, AllReduce, ZeRO, PP, TP, and validation. | Learners who want to understand multi-GPU scaling and communication cost. |
| Cross-Cutting Topic | Profiling Topic | Part 00-02 | Guide: intro; casebook: casebook. Evidence collection, trace reading, regression validation, and action decisions. | Learners who want systematic performance diagnosis and debugging methods. |
| Cross-Cutting Topic | Post-Training Alignment Topic | Part 01-02 | Guide: intro; casebook: casebook. RLHF, DPO, GRPO, preference data, and project closure. | Learners who want to continue from SFT into alignment and preference optimization. |
| Foundation Topic | Backpropagation and Training Mechanics Topic | Part 01-02 | Guide: intro; casebook: casebook. Autograd, backward, checkpointing, offload, and training rhythm. | Learners who want stronger training-mechanism foundations. |
| Foundation Topic | Model Architecture Topic | Part 01-02 | Guide: intro; casebook: casebook. Structure evolution, representative models, and MoE/sparsity. | Learners who want structural background and model comparison. |
🤝 View collaborative study
| Module | Coverage | Content Positioning | Suitable For | Status |
|---|---|---|---|---|
| Team Study Topic | Not fixed | part2_l1_202606 / part2_l1_202607 / part2_l2_202607 | Learners who want to accumulate knowledge and review records through collaborative study. | 🛠 In progress |
🆕 View update timeline
- 2026-08-22: Revised the README navigation, project introduction, and project features; finalized the five-layer Infra structure and cross-topic boundaries; completed result saving, environment notes, and baseline validation for real backend / GPU projects including 66, 73, 75, and 76, together with documentation and link checks.
- 2026-08-17: Unified the Part 01 guides and group-page conventions, tightened the cross-topic structure and learning paths, reviewed Part 02 project pages and visual assets, and clarified the shared validation entry points.
- 2026-07-10: Refined the homepage tutorial overview and status columns, aligning the Part 00 / Part 01 group names, lesson counts, and navigation status.
- 2026-06-26: Reworked the homepage overview, status columns, and learning paths to clarify the relationship between Parts 00-04, cross-cutting topics, and collaborative study.
- 2026-06-15: Finalized the Part 0 / 1 grouping and guide cleanup, unified the part-level navigation, connected the page comments to GitHub Discussions, and continued expanding Part 1 content, bridge pages, and notebook structure.
- 2026-06-13: Fixed dead links and added placeholder pages for unfinished content to prevent 404s in learning entry points.
- 2026-04-21: Updated Colab badges to point to the official
datawhalechinarepository. - 2026-04-20: Launched the site homepage and part guides; added Part 0 prerequisites and Part 1 practice content to unify the learning path.
- 2026-04-18 ~ 2026-04-19: Refactored Part 2 / 3 content, polishing notebooks, answer sections, and operator implementation notes.
- 2026-04-02: Completed the initial tutorial notebooks, docs, and test scripts.
Path compatibility note: Part 03 has been renamed from
03_CUDA_and_Triton_Kernelsto03_Triton_Kernels, and CUDA / system optimization content has moved to Part 04. Old web paths keep migration pages, but new links should use03_Triton_Kernels.
🚀 Quick Start
We recommend starting with Part 02 to build implementation intuition, then backfilling Part 00 / Part 01 as needed before moving to Part 03 / Part 04. You can also enter directly through a training, inference, or memory topic.
View online, local, and CNB options
Option 1: Read Online
Visit the online platform:
https://datawhalechina.github.io/llm-algo-leetcode/
Suitable for:
- Skimming the table of contents first and then jumping to the part you need
- Reading the part guides first
- Part 00 / 01 / 02 can run on Colab CPU for most basic exercises
- Real inference, training, and memory experiments, as well as Part 03 / 04, need a Colab GPU runtime
Option 2: Local Development
git clone https://github.com/datawhalechina/llm-algo-leetcode.git
cd llm-algo-leetcode
conda env create -f environment.yml
conda activate llm_algo
jupyter labSuitable for:
- Running Part 00 / 01 / 02 locally on CPU
- Controlling your own Python / PyTorch / CUDA versions
- More stable offline debugging
- Part 03 / 04 require a local NVIDIA GPU
For environment details and platform differences, see docs/guide.md.
Option 3: CNB Unified Delivery
If you want the same runtime style used by the repository, use the CNB unified environment.
Suitable for:
- Team collaboration
- Consistent experiment images
- Lower local environment drift
- Part 00 / 01 / 02 can use CNB CPU
- Part 03 / 04 need a CNB GPU session
See docs/guide.md for the exact environment rules and scope.
📖 More Resources
- docs/guide.md - environment and learning modes
- docs/contributing.md - how to contribute to development and testing
- docs/maintenance.md - maintenance rules for parts, links, tests, and releases
- Automated Test Script Index - entry points for automated verification scripts
👨💻 Contributors
| Name | Role | Description |
|---|---|---|
| lynn_jingjing | Project initiator | An algorithm engineer |
(Feel free to add your name here! )
📄 License
Tutorial text in this repository is licensed under CC BY 4.0, and code is licensed under Apache-2.0. .ipynb files are mixed-content notebooks, so please follow the corresponding license by cell type.