AN OPEN RESEARCH NOTEBOOK
From a paper,
to a real training run.
Hands-on LLM and agent reinforcement learning with PyTRIO.
16 runnable experiment notes: from GRPO, DAPO and GSPO to Search-R1, ReTool and multimodal agents.
git clone datawhalechina/agentic-rl-lab && uv syncQuickstart THE BOARD
Every experiment, right here.
Loss Functions
An intuition for reinforcement learning losses
Understand importance sampling, PPO clipping and CISPO through intuitive examples, equations and the policy updates used in this lab.
GRPO
Learn from comparisons within a group
Follow GRPO from group sampling and rewards to policy updates on GSM8K, with runnable PyTRIO code and a comparison of three loss functions.
Vision GRPO
Bring images into reinforcement learning
Connect image inputs, geometry reasoning and verifiable rewards on GeoQA, with multimodal training code and evaluation details.
PyTrio-Jev
Train your own classification and decision model
Fine-tune Qwen3.5-4B on 2,160 records with LoRA, then run restricted-softmax inference, temperature calibration and five benchmark evaluations.
General OPD
Teach the student on its own trajectories
Build a minimal on-policy distillation loop on DeepMath-103K: student sampling, teacher scoring and reverse KL, with code and evaluation notes.
Medical OPD
Balance specialist and general capabilities
Start from medical SFT and study SAR-OPD and IDT-OPD, with specialist and general-task evaluations and explicit experimental limitations.
DAPO
Unpack four changes to training stability
Examine DAPO’s four core changes and the practical time cost of dynamic sampling through training code, equations and recorded experiments.
GSPO
Optimize at the sequence level
Implement sequence-level importance ratios and clipping, and distinguish the core GSPO loss reproduced here from a complete training reproduction.
OPSD
Distill from self-generated trajectories
Study self-distillation with a fixed step-0 teacher and trajectories sampled by the student, including the training loop and mathematical reasoning results.
AgentOPSD
Assign credit to each interaction turn
Use a skill-guided self-teacher to provide turn-level learning signals, then examine multi-turn agent training and paired evaluation in ALFWorld.
Search-R1
Learn to search while reasoning
Train multi-turn search with the official DeepSeek Search API or Wikipedia, and compare Base with the 30-step checkpoint using fixed evaluation questions.
ReTool
Make Python part of the reasoning process
Connect a local Python sandbox and study reinforcement learning with interleaved reasoning and code execution, including rollout handling and results.
ALFWorld
Train an agent to complete household tasks
Train a household agent in a real TextWorld environment, with long trajectories, environment feedback, group-relative advantages and evaluation notes.
TEMPO
Learn long tasks through shorter trajectory segments
Explore macro-step optimization and a generative critic in ALFWorld, with clear boundaries between the public TEMPO description and this reproduction.
Spec-o3
Search the sky with reinforcement learning
Move from SFT to GRPO for astronomical candidate review, interleaving spectrum images, reasoning and tools with explicit evaluation settings.
Tournament-GRPO
Put research reports through a tournament
Compare research reports for the same query in tournaments, turn wins into GRPO rewards, and examine a 20-step PyTRIO run with independent evaluation.
TRAINING PLATFORMS
From training tools to experiments.
LEARN IN THE OPEN
Document the limits. Keep asking questions.
Implementations, results and things that did not work. Bring your questions, reproduction notes and ideas.