An Agentic RL learning path
Follow the sequence below or enter through a question you already have. Running one small experiment and changing one variable is often a better way to build intuition than reading every paper at once.
01 · Understand a policy update
Loss Functions → GRPO → DAPO → GSPO
Begin with log probabilities, importance ratios and advantages. The GRPO, DAPO and GSPO comparison explains what each chapter covers. After completing a GRPO training loop, examine how dynamic sampling, clipping and a sequence-level objective change it.
Your first change
In the GRPO experiment, keep other parameters fixed and switch among importance_sampling, ppo and cispo. Compare the training behavior. See the GRPO quick start for commands.
02 · Turn teacher feedback into a learning signal
General OPD → Medical OPD → OPSD
Start with student sampling and teacher scores to understand token-level feedback. Then examine how specialist and general capabilities fit into one training process, before exploring self-distillation.
03 · Let a model interact with the world
Introduce search, a Python sandbox and a TextWorld environment in turn. Watch how multi-turn trajectories are stored, how environment observations enter the context, and how training distinguishes model actions from observations.
04 · Explore more complex tasks
After GRPO and tool interaction, continue with one of these directions:
| Direction | Reading order | Key question |
|---|---|---|
| Multimodal tool use | Vision GRPO → Spec-o3 | How do images and tool outputs enter a training trajectory together? |
| Long-horizon learning and credit assignment | TEMPO → AgentOPSD | Which segment or turn of a long trajectory should receive a learning signal? |
| Fixed-option classification and decisions | PyTrio-Jev | How can one forward pass produce candidate probabilities, and how do we check and calibrate confidence? |
The TEMPO chapter is an algorithm-level reproduction in ALFWorld based on a public technical blog. Each method’s environment, budget and implementation boundaries are documented in its note; read those alongside the results.
Check your understanding with an experiment
After each chapter, answer four questions:
- Where does the reward come from, and can it be verified?
- Which tokens or interaction turns participate in optimization?
- What variable did this experiment actually change?
- Which conclusions still need stronger controls?
Continue to environment setup or return to all experiments.