Foreword
In late 2022, large language models, led by ChatGPT, rapidly entered the public eye, accelerating the pace of AI development. Since then, models such as GPT-4, DeepSeek-R1, and Qwen have continually pushed the boundaries of what people believe AI is capable of. Large Language Models (LLMs) have gradually become the core of research in natural language processing and the broader field of artificial intelligence. From academia to industry, and from internet companies to traditional sectors, large-model technology is profoundly influencing the nature of technological innovation and is gradually reshaping paradigms of how people learn, work, and create.
From the perspective of technological evolution, LLMs did not appear in a vacuum, but are built upon a long history of accumulated research. The field of natural language processing has gone through stages such as symbolism, statistical learning, and deep learning, and has gradually evolved into pre-trained language models represented by BERT and GPT, through large-scale unsupervised data training, these models have developed powerful language comprehension capabilities. As model parameters, data scale, and computing power have continued to grow, researchers have further explored new training paradigms such as instruction fine-tuning and reinforcement learning alignment, enabling models to continuously break through in understanding, reasoning, and generation, ultimately ushering in the current era of large language models.
However, the rapid development of large language models has also brought new technical challenges. Compared to traditional machine learning tasks, building and training an LLM involves not only model architecture design but also multiple complex steps, including data engineering, distributed training, GPU performance optimization, inference system design, and model evaluation. For learners, understanding these technologies often presents two major challenges: first, the theoretical knowledge foundation is vast and complex; second, there is a lack of systematic, practical learning paths.
It is against this backdrop that we hope this CS336 Chinese course will become not merely a translation or “Chinese version” of the original course, but a tailor-made “LLM Alchemy Workshop” for Chinese learners. Here, you can not only understand the core theory behind large models but also personally build and train models, mastering the complete LLM development pipeline through continuous experimentation and iteration. Our goal is that, through systematic content design, learners can build their own LLM step by step—just like alchemy, from preparing materials and controlling the heat to finally producing the elixir.
While retaining the technical depth of the original course, this course has reorganized and localized the knowledge framework. The content covers core theories of large language models—such as the Transformer architecture, scaling laws, training workflows, and alignment techniques—while also providing an in-depth exploration of key engineering challenges, including GPU programming, high-performance optimization, and distributed training and inference systems. Additionally, we have tailored the course for the practical environment of developers in China by incorporating more real-world examples and discussions into the case studies and discussions, such as the domestic open-source model ecosystem and common engineering practices.
In terms of learning methods, this course emphasizes a “code-driven, theory-and-practice-integrated” teaching model. Theoretical study is only the first step; the key to truly understanding large language models lies in hands-on practice. Therefore, this course has designed a series of progressive hands-on assignments—ranging from implementing a tokenizer and building a language model architecture to performing data processing, system optimization, and model alignment—to gradually guide learners through the complete training and evaluation process for large language models. Through these practical exercises, you will not only understand “what” large language models are but also truly master “how” to build them.
After completing this course, you will gain a complete knowledge framework for large models: you will understand the core principles of LLMs and be able to participate in practical large model engineering development. From data preparation to model training, and from system optimization to evaluation and deployment, these capabilities will help you establish a solid technical foundation in the current AI era, and also prepare you well for future scientific research or industrial-grade large model development.
I. Course Objectives and Target Audience
This course focuses on the core principles and engineering practices of large language models, covering topics such as model architecture, training methods, system optimization, data engineering, and model evaluation. The course emphasizes both theoretical foundations and the development of engineering skills, aiming to help learners systematically master the technology stack of large language models and acquire the practical skills to build and train language models from scratch.
This course is suitable for learners with some technical background. Before you begin, you should be proficient in the Python programming language and possess basic software engineering skills, such as code organization, debugging, and experiment management. You should also have a basic understanding of the principles of deep learning, including neural network architectures, backpropagation, and optimization algorithms. If you are familiar with the PyTorch framework and have a foundation in linear algebra, probability and statistics, and calculus, you will find the course easier to follow. Furthermore, familiarity with the basic concepts of GPU computing or CUDA will help you understand the course’s section on system optimization, but this is not a prerequisite. The course will gradually introduce the relevant fundamentals and practical methods, so there is no need to worry.
II. Course Structure and Advantages
Currently, most tutorials are limited to building the basic components of large models, without touching on data processing, resource estimation, GPU principles, distributed training, model evaluation, reinforcement learning, and other aspects. Our goal is to enable readers to learn and experience the complete pipeline of building a large model. This is the advantage of this tutorial.
This course is organized according to the complete workflow of building a large model system and can be broadly divided into three layers. The first layer covers fundamental components, introducing the core building blocks required for constructing a language model, such as tokenizers, the Transformer architecture, and training primitives, laying the foundation for subsequent content. The second layer covers system implementation, gradually delving into more complex system-level topics, including Mixture of Experts (MoE), high-performance GPU programming, distributed training, and model scaling laws. The third layer covers the complete training pipeline, presenting the full process of large model training and application, including data engineering, model training workflows, model alignment methods, and evaluation and inference optimization. Through this progressive learning path, you will gradually build a complete large model technology framework.
III. Suggestions for Learning
In terms of learning methods, we strongly recommend that you combine theoretical study with hands-on practice. The most effective way to understand large models is not simply by reading documentation, but by continuously deepening your understanding through writing code, running experiments, and analyzing results. The practical assignments in this course start with basic components—such as building a word segmenter and a language model from scratch—and then gradually progress to system optimization, data processing, model tuning, and model evaluation tasks. We recommend that you complete each assignment thoroughly and experiment with different configurations or parameters; this hands-on experience will help you gain a deeper understanding of the relationship between model design and system implementation.
At the same time, the field of large language models is evolving at a rapid pace, with new methods, architectures, and engineering tools constantly emerging. Therefore, while studying the course material, we also encourage you to stay up to date on cutting-edge technologies—for example, by reading the latest research papers, following open-source communities, and participating in relevant technical discussions. Through interaction with the community, you will not only gain more practical experience but also develop a better understanding of current trends in large language model technology.
IV. Conclusion
Finally, we hope you will maintain patience and a spirit of exploration throughout this course. Building a large model is a complex and systematic engineering task that requires continuous experimentation, debugging, and optimization. Encountering difficulties and challenges along the way is completely normal, but it is precisely this process that will help you truly understand the principles behind the technology. When you are eventually able to build and train a language model from scratch, what you will gain is not only knowledge but also the ability to solve complex engineering problems.
Large model technology is still developing rapidly, with new breakthroughs emerging every year, from architectures to methods. We hope this course will help you not only understand current technology but also cultivate the ability to continuously explore and build complex AI systems.
Welcome to the ranks of large model developers. May this course serve as a stepping stone into the world of LLMs, helping you to continuously explore, create, and grow in the future AI era.
Welcome to this “LLM Alchemy Workshop.” May you gradually master the craft of building large models through continuous experimentation, debugging, and reflection, and create your own achievements in the era of large models.