Diy-LLM — A Systematic Guide to Building Large Language Models

An "LLM Alchemy Workshop" tailor-made for learners worldwide

We hope this CS336 course will be more than just a translation of the Stanford original — it should be a hands-on "LLM Alchemy Workshop" where you forge understanding, polish code, control the heat, and ultimately refine your own model.

Prerequisites

• Python: Proficient in Python and software engineering skills
• Deep Learning: Familiar with PyTorch and neural network fundamentals
• Mathematics: Linear algebra, probability & statistics, calculus
• Machine Learning: Solid grasp of ML and DL basics
• GPU Programming (optional): CUDA basics help but aren't required

Course Vision

• Rigorous theory meets hands-on practice: Full technical depth with a structured learning path
• Progressive knowledge system: Breaking down the massive LLM project into digestible, hands-on modules
• Code-driven, theory-integrated: Every assignment includes implementation code and the thinking behind each line
• Localized for real-world use: Practical solutions for diverse compute environments, with examples from Qwen, DeepSeek, and other major open-source models

What You'll Gain

• Solid technical foundation: Build your own LLM from scratch, understanding every core component
• Real engineering experience: Full-pipeline skills from data processing to deployment optimization
• Industry competitiveness: Core capabilities for LLM R&D roles
• Research vision: Systematic knowledge for future deep research

Table of Contents

Chapter	Key Content	Assignment	Status
Foreword	Project background, learning roadmap, prerequisites	-	✅
Ch1 Tooling	W&B experiment tracking, hyperparameter search	-	✅
Ch2 Tokenizer	BPE algorithm, Unicode normalization, tokenizer training	HW1	✅
Ch3 PyTorch & Resource Accounting	Mixed precision, gradient accumulation, FLOPs/memory estimation	-	✅
Ch4 Architecture & Training Details	RoPE, RMSNorm, SwiGLU, AdamW, learning rate schedules	HW1	✅
Ch5 Mixture of Experts	Top-K routing, load balancing, expert parallelism, DeepSeekMoE	-	✅
Ch6 GPU & Optimization	Memory bandwidth, Flash Attention, kernel fusion, BF16	HW2	✅
Ch7 GPU High-Perf Programming	CUDA, Tensor Cores, shared memory, Triton	HW2	✅
Ch8 Distributed Training	Data/Model/Pipeline parallelism, ZeRO, FSDP, All-Reduce	HW2	✅
Ch9 Scaling Laws	Chinchilla optimal, compute-efficient training	HW3	✅
Ch10 Inference	KV cache, speculative decoding, quantization, PagedAttention	HW6	✅
Ch11 Data Engineering	Data filtering, MinHash dedup, PII removal, data recipes	HW4	✅
Ch12 Evaluation & Benchmarks	MMLU, HumanEval, HELM, Chatbot Arena, safety benchmarks	HW6	✅
Ch13 Training Pipeline	Pre-training, SFT, DPO, RLHF PPO pipeline, alignment	HW5	✅
Ch14 RLVR	GRPO, rule-based verifiers, process reward, DeepSeek-R1	HW5	✅
Ch15 Multimodal Models	CLIP, SigLIP, LLaVA, Qwen-VL, Chameleon, Omni Model	-	✅
Ch16 Extended Content	1. What is LLM Reasoning? 2. LLM Future — LeCun's Roadmap	-	🔄

Status: ✅ Complete  🔄 In Progress 📝 Draft 🚧 Planned ⏸️ Paused

Assignments

HW	Core Tasks	Status
HW1: Build an LLM	Implement tokenizer, model architecture, optimizer; train a minimal LM	✅
HW2: Systems	Benchmarking, FlashAttention-2 in Triton, distributed training code	✅
HW3: Scaling Laws	Fit scaling laws, predict model scaling behavior	✅
HW4: Data Processing	Convert Common Crawl raw data to pre-training dataset	✅
HW5: Alignment	Apply SFT and RL (e.g., GRPO) to train on math problems	✅
HW6: Evaluation	Multi-dimension evaluation using lm-evaluation-harness and evalscope	✅

Quick Start

git clone https://github.com/datawhalechina/diy-llm.git
cd diy-llm

Learning Path

1️⃣ Theory → Read docs/en/ chapters in order
2️⃣ Practice → Complete the 6 assignments in coursework/
3️⃣ Deep Dive → Read code implementations, understand design decisions

Project Structure

diy-llm/
├── docs/
│   ├── zh/                 # Chinese docs (default)
│   ├── en/                 # English docs
│   └── .vitepress/         # VitePress config
├── coursework/             # Assignments
│   ├── assignment1-basics/
│   └── ...
└── README.md

Links

• Repository: https://github.com/datawhalechina/diy-llm
• Online Reading: https://datawhalechina.github.io/diy-llm/
• Original Course: Stanford CS336 (Spring 2026)

FAQ

Q: Can I learn without a GPU?

Theory can be studied normally. Some assignments can be debugged on CPU, but full training requires GPU. Cloud services are recommended.

Q: How does this differ from the original CS336?

We retain the original technical depth while adding Chinese explanations, detailed reference sources, domestic model examples (Qwen, DeepSeek), and complete assignment implementations.

Reader Community

Join the Diy-LLM reader community to discuss the course, ask questions, and learn together:

Group 1

Group 2

If group 1 is full, join group 2. If both QR codes have expired or are full, add any maintainer below on WeChat (note: diy-llm) and we'll add you to the group:

Maintainer	WeChat ID
Xu Hu (徐虎)	xuhu96736
Li Youzhen (黎又榛)	zydsx111
Li Shengkang (李盛康)	muzichengminguangli

Contributors

Xu Hu (徐虎) Project Lead · Datawhale · SJTU Chapters: 1, 3, 9, 12, 14, 15; HW5, 6; Content review

Li Shengkang (李盛康) Project Lead · Datawhale · XUPT Chapters: 4, 6, 7, 8, 13; HW2, 4

Li Youzhen (黎又榛) Project Lead · Datawhale Chapters: 2, 5, 10, 11, 13; HW1, 3

• Thanks to @aimetrics for adding MPS support for MacBooks in HW1
• Thanks to @FuTseYi (Datawhale) for website migration, deployment, and refactoring
• Thanks to @jiangyinhe for contributing to Ch16 "LLM Future — LeCun"; and for translating the first chapter and preface into English based on the Chinese version.

License

This work is licensed under CC BY-NC-SA 4.0.

Acknowledgments

• Stanford CS336 course team for the excellent original course
• @Sm1les for support and guidance
• All contributors and the open-source community

Star History

---

Empowering everyone to systematically learn large language model construction

Made with ❤️ by Datawhale