AN OPEN RESEARCH NOTEBOOK
From a paper,
to a real training run.
Hands-on LLM and agent reinforcement learning with PyTRIO.
15 runnable experiment notes: from GRPO, DAPO and GSPO to Search-R1, ReTool and multimodal agents.
git clone KMnO4-zx/agentic-rl-lab && uv syncQuickstart THE BOARD
Every experiment, right here.
Loss Functions
An intuition for reinforcement learning losses
Understand importance sampling, PPO clipping and CISPO through intuitive examples, equations and the policy updates used in this lab.
GRPO
Learn from comparisons within a group
Follow GRPO from group sampling and rewards to policy updates on GSM8K, with runnable PyTRIO code and a comparison of three loss functions.
General OPD
Teach the student on its own trajectories
Build a minimal on-policy distillation loop on DeepMath-103K: student sampling, teacher scoring and reverse KL, with code and evaluation notes.
Medical OPD
Balance specialist and general capabilities
Start from medical SFT and study SAR-OPD and IDT-OPD, with specialist and general-task evaluations and explicit experimental limitations.
Search-R1
Learn to search while reasoning
Put online search inside the rollout loop and train a model to retrieve evidence across turns, with switchable search backends and evaluation notes.
OPSD
Distill from self-generated trajectories
Study self-distillation with a fixed step-0 teacher and trajectories sampled by the student, including the training loop and mathematical reasoning results.
ReTool
Make Python part of the reasoning process
Connect a local Python sandbox and study reinforcement learning with interleaved reasoning and code execution, including rollout handling and results.
DAPO
Unpack four changes to training stability
Examine DAPO’s four core changes and the practical time cost of dynamic sampling through training code, equations and recorded experiments.
GSPO
Optimize at the sequence level
Implement sequence-level importance ratios and clipping, and distinguish the core GSPO loss reproduced here from a complete training reproduction.
ALFWorld
Train an agent to complete household tasks
Train a household agent in a real TextWorld environment, with long trajectories, environment feedback, group-relative advantages and evaluation notes.
Vision GRPO
Bring images into reinforcement learning
Connect image inputs, geometry reasoning and verifiable rewards on GeoQA, with multimodal training code and evaluation details.
TEMPO
Learn long tasks through shorter trajectory segments
Explore macro-step optimization and a generative critic in ALFWorld, with clear boundaries between the public TEMPO description and this reproduction.
AgentOPSD
Assign credit to each interaction turn
Use a skill-guided self-teacher to provide turn-level learning signals, then examine multi-turn agent training and paired evaluation in ALFWorld.
Spec-o3
Search the sky with reinforcement learning
Move from SFT to GRPO for astronomical candidate review, interleaving spectrum images, reasoning and tools with explicit evaluation settings.
PyTrio-Jev
Train your own classification and decision model
Fine-tune Qwen3.5-4B on 2,160 records with LoRA, then run restricted-softmax inference, temperature calibration and five benchmark evaluations.
TRAINING PLATFORMS
From training tools to experiments.
LEARN IN THE OPEN
Document the limits. Keep asking questions.
Implementations, results and things that did not work. Bring your questions, reproduction notes and ideas.