llm-algo-leetcode | 大模型算法与系统教程
Runnable notebooks for LLM algorithms and systems.
面向大模型算法与系统的可运行 Notebook 教程。
曾用名:大模型算法实战教程
主学习路线 / Core Routes
模型架构与训练 / Architecture & Training · 后训练优化 / Post-Training · 推理优化 / Inference · 性能优化 / Performance · 算子优化 / Operator
专题导航 / Topic Navigation
基础与诊断 / Foundations · 横向工程 / Cross-Cutting · 领域专题 / Domains · 全部专题 / All Topics
中文版 (Chinese) | English Version
中文版
🎯 项目简介
这是一个以 Notebook 为载体的大模型算法与系统教程。教程从 Part 02 的 PyTorch 算法实践出发,按需回补 Part 00 / Part 01 的基础,并进一步延伸到 Part 03 的 Triton 算子开发和 Part 04 的 CUDA 与系统优化。
内容围绕可运行实现、实验验证和专题化学习路线组织,帮助学习者逐步理解模型如何计算、如何训练、如何推理,以及性能问题如何被测量和优化。
✨ 项目特点
- Notebook-first:每节围绕可运行代码、题目区、答案区和基础验证展开,适合边学边改。
- 按目标进入:既支持从 Part 02 开始建立算法实践,也支持按五条主学习路线或基础、横向、领域专题跳读。
- 算法到系统贯通:沿着
PyTorch → Triton → CUDA逐步下探,连接模型实现、算子优化、显存管理、通信与系统性能。 - 用证据做项目判断:通过 benchmark、profiling、真实 GPU 和 inference backend 实验,对吞吐、延迟、显存、质量和成本进行比较。
👥 适合对象
- 大模型算法学习者:希望通过 PyTorch Notebook 理解 Transformer、训练、微调、推理和模型压缩。
- 性能与系统学习者:希望进一步学习显存、Profiling、通信、Triton、CUDA 和 GPU 优化。
- 项目实践者:希望通过 benchmark、真实 GPU 和 inference backend 实验,建立可复现、可比较的工程判断。
🌐 教程总览
教程提供两种阅读方式:按 Part 00 -> Part 04 逐层推进,或从模型架构与训练、后训练优化、推理优化、性能优化和算子优化五条主学习路线进入,再按问题回补基础与横向专题。五条路线的排列表示从模型结构、能力形成到服务运行、性能诊断和底层实现的对象层次,不是严格的前置依赖。推荐先从 Part 02 建立算法实践感,再按需要回补 Part 00 / Part 01,最后进入 Part 03 / Part 04;完成相关主线后,可进入 Part 05 的综合项目,把机制、实验和架构决策收敛为可交付方案。
topic_discussion 用于组织跨 Part 的专题路线。页面顶部提供常用专题入口,下面的资产总览和专题总览分别说明主线结构与专题覆盖范围。
📚 查看完整资产总览
这套教程不要求从 00 开始按顺序硬读。00 主要是前置补齐区,如果你已有基础,可以直接从最相关的部分开始;下面这张表会直接告诉你:每一部分学什么、包含哪些组、适合谁、当前进度如何。
🧭 查看完整专题总览
| 层级 | 入口 | 内容定位 | 适合对象 | 状态 |
|---|---|---|---|---|
| 主学习路线(建设中) | 模型架构与训练(Model Architecture and Training) | 从注意力、位置、专家和记忆结构,进入局部反向、有效更新与状态策略。 | 想把模型结构与训练机制联系起来的学习者。 | 🛠 建设中 |
| 主学习路线(建设中) | 后训练优化(Post-Training Optimization) | SFT / LoRA 基础、偏好数据、DPO、GRPO 与项目交付。 | 想从监督微调继续进入偏好优化与对齐的学习者。 | 🛠 建设中 |
| 主学习路线 | 推理优化(Inference Optimization) | FlashAttention、解码、PagedAttention、cache 与 benchmark。 | 想系统理解推理加速路径的学习者。 | ✅ 持续优化 |
| 主学习路线 | 性能优化(Performance Optimization) | 包含测量与诊断方法,从基线、资源约束走到单项与组合决策。 | 想系统定位并优化训练、推理和系统性能的学习者。 | 🛠 建设中 |
| 主学习路线(建设中) | 算子优化(Operator Optimization) | 从硬件直觉、图级优化、Triton、经典算子、Attention 到 CUDA 与端到端验证。 | 想从算子实现走向 kernel 和端到端性能优化的学习者。 | 🛠 建设中 |
| 基础支撑专题 | 模型架构演进(LLM Architecture Evolution) | Decoder、注意力、位置、专家与记忆结构比较。 | 想独立研究模型结构的学习者。 | 🛠 建设中 |
| 基础支撑专题 | 反向传播与训练机制(Backpropagation and Training Mechanics) | autograd、backward、checkpointing、offload 与训练节奏。 | 想补训练机制底座的学习者。 | ✅ 持续优化 |
| 横切支撑专题 | 量化与压缩(Quantization and Compression) | PTQ、QAT、GPTQ、AWQ、FP8 与部署决策。 | 想同时考虑精度、显存、吞吐和部署取舍的学习者。 | ✅ 持续优化 |
| 横切支撑专题 | 通信与并行(Communication and Parallelism) | NCCL、AllReduce、ZeRO、PP、TP 和并行验证。 | 想理解多卡训练和通信边界的学习者。 | ✅ 持续优化 |
| 横切支撑专题(建设中) | 数据工程(Data Engineering) | 数据生命周期、解析、清洗、去重、配比、合成、血缘与工程化。 | 想同时服务训练、RAG 与数据质量治理的学习者。 | 🛠 建设中 |
| 横切支撑专题(建设中) | 部署与异构系统(Deployment and Heterogeneous Systems) | 模型产物、推理 backend、CPU/GPU/NPU 协同、资源编排和部署验证。 | 想理解模型如何从文件进入可交付服务的学习者。 | 🛠 建设中 |
| 领域专题(建设中) | Agent 基础设施(Agent Infrastructure) | 工具轨迹、沙盒执行、任务调度、可观测性与 Agentic RL。 | 想让模型可靠完成多步工具任务的学习者。 | 🛠 建设中 |
| 领域专题(建设中) | 多模态与视觉推理(Multimodal and Vision Inference) | 以多模态推理为主线,覆盖视觉表示、对齐、视觉 token、推理成本和任务评测;训练作为共享前置。 | 想进入视觉语言模型、文档理解或图文推理的学习者。 | 🛠 建设中 |
🆕 查看更新时间线
- 2026-10-26:重整中英文首页导航与 Part 命名,更新教程路线图并以“能力、服务、效率”呈现三条主生命线;用实验金字塔概括从基础验证到端到端项目的证据范围,精简环境入口并同步文档站点首页。
- 2026-09-30:统一五条主学习路线及基础、横向、领域专题的中英文名称与入口;将显存优化内容正式承接到性能优化,将 SFT / LoRA 与图级编译分别并入后训练优化和算子优化;更新 Part 05 收口定位、专题资产与首页结构图。
- 2026-09-11:完成当时主路线与横向专题的阶段性梳理,补充 Task0–6 学习入口与项目分工;统一项目节的类型、环境和证据边界,并完成源文件到
docs/的镜像同步与 VitePress 构建验证。Part 02 的 60–76 项目进入持续优化和 GPU 数据采集准备阶段。 - 2026-08-22:完成 README 首页导航、项目简介与项目特点重写;收口横向专题的五层 Infra 结构与跨专题边界;完成 66、73、75、76 等真实 backend / GPU 项目的结果保存、环境说明和基础验证,并同步文档入口与链接检查。
- 2026-08-17:统一 Part 01 导读与组页口径,收紧横向专题结构和学习路线表达,补充 Part 02 项目页与图解资产审计,并明确统一验证入口。
- 2026-07-10:整理首页教程总览与状态列,校正 Part 00 / Part 01 的组名、节数和导航状态。
- 2026-06-26:重构首页教程总览、状态列和学习路径,明确 Part 00-04、横向专题与共学记录的关系。
- 2026-06-15:推进第零部分 / 第一部分的分组与导读收口,统一部分级导航,并完成网页底部评论区接入 GitHub Discussions,同时持续扩展第一部分的正文、桥接页与 Notebook 结构。
- 2026-06-13:修复 dead link,并为未完成页面补充占位页,避免学习入口出现 404。
- 2026-04-21:更新 Colab 徽章链接,统一指向官方
datawhalechina仓库。 - 2026-04-20:上线站点首页与部分导学;新增第零部分前置知识与第一部分练习内容,完善在线阅读入口与学习路径。
- 2026-04-18 ~ 2026-04-19:集中重构第二部分 / 第三部分内容,优化 Notebook、答案区与算子实现说明。
- 2026-04-02:完成教程核心 Notebook、文档与测试脚本的初始搭建。
路径说明:第三部分现用目录为
03_Triton_Kernels;CUDA / 系统优化内容已拆分到第四部分。引用时请使用当前目录路径。
🚀 快速开始
推荐从 Part 02 开始:先通过 PyTorch Notebook 建立算法实现感,再根据知识缺口回补 Part 00 / Part 01,最后进入 Part 03 / Part 04 的 GPU 底层优化,并由 Part 05 完成项目与决策收口。也可以直接选择一条主学习路线,再按问题进入基础、横向或领域专题。运行 Notebook 前,请先查看 使用指南 和对应小节中的环境说明。
| 想完成的任务 | 默认运行入口 | 开始前查看 |
|---|---|---|
| 阅读或运行 CPU 机制练习 | BASE-CPU | 使用指南 |
| 运行单卡训练、量化或 GPU 实验 | CUDA PyTorch + 对应能力 profile | 模型、执行栈与环境资产 |
| 运行 Serving 或多卡项目 | 独立 backend / distributed 环境,并先执行预检 | 使用指南与模型、执行栈与环境资产 |
在线阅读请访问 教程站点。本地运行 CPU 练习可使用:
git clone https://github.com/datawhalechina/llm-algo-leetcode.git
cd llm-algo-leetcode
python tools/install_environment_profile.py \
--profile BASE-CPU \
--env-name llm_algo_cpu
conda activate llm_algo_cpu
jupyter labCUDA、Serving、多卡、CNB 和云端平台不在 README 复制安装矩阵;请从使用指南选择路径,再进入对应专题 Quickstart。精确版本与已验证硬件见模型、执行栈与环境资产。
📖 更多资源
👨💻 贡献者名单
| 姓名 | 职责 | 简介 |
|---|---|---|
| lynn_jingjing | 项目发起人 | 一个算法工程师 |
📄 许可声明
本仓库所有 .ipynb 文件中的文字内容(Markdown 单元格、公式、图示说明)采用 CC BY 4.0 协议;代码内容(Code 单元格、可执行实现)采用 Apache-2.0 协议。使用、转载、改编时,请按单元格类型分别遵守对应协议。文字协议见 LICENSE,代码协议见 LICENSE-CODE。
English Version
📄 License Notice
All .ipynb files in this repository are mixed-content notebooks: Markdown cells (tutorial text, formulas, and figure captions) are licensed under CC BY 4.0, while Code cells (executable implementations) are licensed under Apache-2.0. Please comply with the corresponding license by cell type when using, redistributing, or adapting this repository. See LICENSE for text and LICENSE-CODE for code.
🎯 Project Introduction
This is a notebook-based tutorial on LLM algorithms and systems. It starts with PyTorch practice in Part 02, lets learners backfill the prerequisites in Part 00 / Part 01 as needed, and extends to Triton kernel development in Part 03 and CUDA and system optimization in Part 04.
The tutorial is organized around runnable implementations, experimental validation, and topic-based learning paths. It helps learners understand how models compute, train, and serve requests, and how to measure and optimize performance problems.
✨ Features
- Notebook-first: Each lesson is organized around runnable code, exercises, answer cells, and basic validation.
- Goal-oriented entry: Start from Part 02 for implementation practice, or follow the architecture, post-training, inference, performance, and operator routes.
- From algorithms to systems: Follow the
PyTorch → Triton → CUDApath across model implementation, kernel optimization, memory, communication, and system performance. - Evidence-based projects: Use benchmarks, profiling, real GPU runs, and inference backends to compare throughput, latency, memory, quality, and cost.
👥 Suitable For
- LLM Algorithm Learners: Use PyTorch notebooks to understand Transformers, training, fine-tuning, inference, and model compression.
- Performance and Systems Learners: Study memory, profiling, communication, Triton, CUDA, and GPU optimization.
- Project Practitioners: Build reproducible engineering judgment through benchmarks, real GPU runs, and inference backends.
🌐 Tutorial Overview
This tutorial offers two ways to read: follow the Part 00 -> Part 04 progression, or enter through one of five main routes—model architecture and training, post-training optimization, inference optimization, performance optimization, and operator optimization—then backfill foundation and cross-cutting topics as needed. Their left-to-right order describes a progression of objects from model structure and capability formation to serving, diagnosis, and low-level implementation; it is not a strict prerequisite chain. We recommend starting with Part 02 to build implementation intuition, then backfilling Part 00 / Part 01 before moving to Part 03 / Part 04; after completing the relevant routes, use Part 05 integrated projects to turn mechanisms, experiments, and architecture choices into a deliverable plan.
topic_discussion organizes cross-Part topic paths. The top of the page provides common topic entries; the asset and topic overviews below describe the main structure and topic coverage.
📚 View the complete asset overview
You do not need to start from 00 in strict order. 00 is the prerequisite lane; if you already have the background, jump directly to the part that matches your goal. The table below summarizes each part, its groups, its audience, and its status.
🧭 View the complete topic overview
| Layer | Entry | Content Positioning | Suitable For | Status |
|---|---|---|---|---|
| Main Study Path (In Progress) | Model Architecture and Training Topic | Attention, position, experts, memory, local backward, effective updates, and training-state strategies. | Learners who want to connect model structure with training mechanics. | 🛠 In progress |
| Main Study Path (In Progress) | Post-Training Optimization Topic | SFT / LoRA foundations, preference data, DPO, GRPO, and project delivery. | Learners who want to continue from supervised fine-tuning into alignment. | 🛠 In progress |
| Main Study Path | Inference Optimization Topic | FlashAttention, decoding, PagedAttention, cache, and benchmark. | Learners who want practical inference acceleration. | ✅ Ongoing |
| Main Study Path | Performance Optimization Topic | Measurement and diagnosis methods, memory management, communication costs, and end-to-end decisions. | Learners who want to diagnose and optimize training, inference, and system performance. | 🛠 In progress |
| Main Study Path (In Progress) | Operator Optimization Topic | Graph rewrites, fusion, Triton, CUDA, memory access, autotuning, and kernel-to-end-to-end validation. | Learners who want to move from operator implementation to kernel and end-to-end optimization. | 🛠 In progress |
| Foundation Topic | LLM Architecture Evolution Topic | Decoder, attention, position, experts, and memory structures. | Learners who want to study model architecture independently. | 🛠 In progress |
| Foundation Topic | Backpropagation and Training Mechanics Topic | Autograd, backward, checkpointing, offload, and training rhythm. | Learners who want stronger training-mechanism foundations. | ✅ Ongoing |
| Cross-Cutting Topic | Quantization and Compression Topic | PTQ, QAT, GPTQ, AWQ, FP8, and deployment decisions. | Learners balancing accuracy, memory, throughput, and deployment cost. | ✅ Ongoing |
| Cross-Cutting Topic | Communication and Parallelism Topic | NCCL, AllReduce, ZeRO, PP, TP, and validation. | Learners who want to understand multi-GPU scaling and communication cost. | ✅ Ongoing |
| Cross-Cutting Topic (In Progress) | Data Engineering Topic | Data lifecycle, parsing, cleaning, deduplication, mixtures, synthesis, lineage, and production pipelines. | Learners building reliable data paths for training, RAG, and evaluation. | 🛠 In progress |
| Cross-Cutting Topic (In Progress) | Deployment and Heterogeneous Systems Topic | Model artifacts, inference backends, CPU/GPU/NPU coordination, resource orchestration, and deployment validation. | Learners who want to understand how models become deployable services. | 🛠 In progress |
| Domain Topic (In Progress) | Agent Infrastructure Topic | Tool trajectories, sandboxed execution, task scheduling, observability, and agentic RL. | Learners building reliable multi-step tool-using systems. | 🛠 In progress |
| Domain Topic (In Progress) | Multimodal and Vision Inference Topic | Multimodal inference as the main line: visual representations, alignment, visual tokens, inference cost, and task evaluation; training is shared prerequisite. | Learners entering vision-language models, document understanding, or multimodal reasoning. | 🛠 In progress |
🆕 View update timeline
- 2026-10-26: Refined the bilingual homepage navigation and Part names; updated the tutorial map with the three cross-cutting lifelines—capability, service, and efficiency; replaced the asset map with an experiment pyramid spanning foundational checks to end-to-end projects; and simplified environment entry points while syncing the docs homepage.
- 2026-09-30: Unified the Chinese and English names and entry points for the five main routes plus foundation, cross-cutting, and domain topics; moved the former memory-optimization route into Performance Optimization, merged SFT / LoRA and graph compilation into Post-Training Optimization and Operator Optimization, and refreshed Part 05 positioning, topic assets, and the homepage structure map.
- 2026-09-11: Consolidated the main routes and cross-cutting topics available at that stage; added Task 0–6 entry points and project roles; unified project types, environments, and evidence boundaries; and completed source-to-
docs/mirroring plus VitePress build validation. Part 02 projects 60–76 then entered continued refinement and GPU-data preparation. - 2026-08-22: Revised the README navigation, project introduction, and project features; finalized the five-layer Infra structure and cross-topic boundaries; completed result saving, environment notes, and baseline validation for real backend / GPU projects including 66, 73, 75, and 76, together with documentation and link checks.
- 2026-08-17: Unified the Part 01 guides and group-page conventions, tightened the cross-topic structure and learning paths, reviewed Part 02 project pages and visual assets, and clarified the shared validation entry points.
- 2026-07-10: Refined the homepage tutorial overview and status columns, aligning the Part 00 / Part 01 group names, lesson counts, and navigation status.
- 2026-06-26: Reworked the homepage overview, status columns, and learning paths to clarify the relationship between Parts 00-04, cross-cutting topics, and collaborative study.
- 2026-06-15: Finalized the Part 0 / 1 grouping and guide cleanup, unified the part-level navigation, connected the page comments to GitHub Discussions, and continued expanding Part 1 content, bridge pages, and notebook structure.
- 2026-06-13: Fixed dead links and added placeholder pages for unfinished content to prevent 404s in learning entry points.
- 2026-04-21: Updated Colab badges to point to the official
datawhalechinarepository. - 2026-04-20: Launched the site homepage and part guides; added Part 0 prerequisites and Part 1 practice content to unify the learning path.
- 2026-04-18 ~ 2026-04-19: Refactored Part 2 / 3 content, polishing notebooks, answer sections, and operator implementation notes.
- 2026-04-02: Completed the initial tutorial notebooks, docs, and test scripts.
Path note: Part 03 now uses
03_Triton_Kernels; CUDA / system optimization content has moved to Part 04. Use the current directory paths when linking to these pages.
🚀 Quick Start
We recommend starting with Part 02 to build implementation intuition, then backfilling Part 00 / Part 01 as needed before moving to Part 03 / Part 04. You can also enter through one of the five main routes or a foundation, cross-cutting, or domain topic.
| Goal | Default runtime entry | Read first |
|---|---|---|
| Read or run CPU mechanism exercises | BASE-CPU | Usage guide |
| Run single-GPU training, quantization, or GPU experiments | CUDA PyTorch + the required capability profile | Model, execution stack, and environment assets |
| Run Serving or multi-GPU projects | An isolated backend / distributed environment with preflight checks | Usage guide and runtime assets |
Read online at the tutorial site. To run CPU exercises locally:
git clone https://github.com/datawhalechina/llm-algo-leetcode.git
cd llm-algo-leetcode
python tools/install_environment_profile.py \
--profile BASE-CPU \
--env-name llm_algo_cpu
conda activate llm_algo_cpu
jupyter labCUDA, Serving, multi-GPU, CNB, and cloud setup are kept out of the README. Choose the path in the usage guide, then follow the relevant topic Quickstart. Exact versions and verified hardware snapshots live in the environment assets.
📖 More Resources
- Usage Guide — learning entry points, runtime environments, and experiment paths
- Contribution Guide — contributing content, code, and tests
- Maintenance and Release Manual — source synchronization, verification, migration, and release
👨💻 Contributors
| Name | Role | Description |
|---|---|---|
| lynn_jingjing | Project initiator | An algorithm engineer |
(Feel free to add your name here! )
📄 License
Tutorial text in this repository is licensed under CC BY 4.0, and code is licensed under Apache-2.0. .ipynb files are mixed-content notebooks, so please follow the corresponding license by cell type.
