LARGER TEACHERS
Early loss plateaus
25.1%Mean final loss reduction · 5 student–teacher pairs · 200 updates
ON-POLICY DISTILLATION · ARXIV 2026
Why On-Policy Distillation
Sometimes FailsVanishing Learning Signals
Larger teachers can plateau. Nearby self-RL teachers recover far more loss.
1 University of Pennsylvania2 Tsinghua University3 University of California, Riverside4 Marquette University
01
THE KEY FINDING
Larger teachers can still lead to loss plateaus.
Across our code and math experiments, we observe two contrasting outcomes.
SELF-RL TEACHERS
Much deeper recovery
96.2%Mean final loss reduction · 2 student–teacher pairs · 200 updates
All 7 student–teacher pairs
Table 2 · Final-checkpoint accuracy; maximum observed loss reduction
| Student / task | OPD teacher | Max. loss reduction ↑ | Accuracy gain | Gap recovery |
|---|
Loss recovery and accuracy gain differ. Qwen3-4B-RL-Math shows the largest accuracy gain in this table despite its loss plateau.
Bars show maximum loss reduction on a shared 0–100% scale, using the lowest observed training loss. Labels summarize the training curves. Accuracy gain is measured in percentage points (pp). Gap recovery is the fraction of the initial teacher–student accuracy gap closed.
Abstract
On-policy distillation (OPD) enables effective capability transfer between language models, yet the mechanisms underlying its failures are not fully understood. Across code generation and mathematical reasoning, OPD with larger-scale teachers exhibits early loss plateaus, with an average final loss reduction of 25.1% after 200 updates, compared with 96.2% for self-RL teachers, obtained by further reinforcement learning (RL) training of the initial student. To understand this difference, we analyze OPD as an idealized continuous-time dynamical system in the small-learning-rate limit. Our training-log diagnostics associate these plateaus with an early decline in a gradient-based learning-signal proxy while substantial loss remains; these measurements do not establish why the underlying gradient weakens. We further prove a local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions, offering a conditional explanation for the success of self-RL teachers in our experiments. Across runs with and without loss plateaus, we observe small relative parameter changes (0.025–0.098%) and high similarity between the student’s representations before and after OPD (linear CKA >0.98 across layers). These observations suggest that limited representation adaptation may contribute to learning-signal collapse, a hypothesis that remains to be tested. Code is available on GitHub.
THE PHENOMENON
Same student. Different outcomes.
Compare larger teachers with self-RL teachers—the initial student, further trained with RL.
Remaining OPD loss
% of first logged loss ↓Validation accuracy
Mean across 3 benchmarks ↑Loading the published experiment data…
THE QUESTION
Why does OPD keep learning from some teachers, but stall with others?
Raw top-16 OPD loss, normalized to update 1; validation every 10 updates. Hover to inspect, or replay training. Self-RL = the initial student after further RL training.
02
UNDERSTANDING OPD PLATEAUS
When the learning signal fades.
THE LOSS DYNAMICS
The rate of loss reduction
\frac{d\mathcal{L}}{dt}=-2\,\mu\,(1+\alpha)\,\mathcal{L}
- \muLearning signal
- Update strength relative to the remaining loss. A small μ means a weak learning signal.
- 1+\alphaRollout effect
- Changing rollouts can reinforce (α > 0) or offset (α < 0) the update. At α ≤ −1 the loss stops falling, even with a nonzero gradient.
We track a logged gradient-based proxy for μ against the remaining loss below.
THE EXPERIMENTAL PATTERN · FIGURE 2
The signal weakens before the loss is gone.
Larger teachers: weaker signal, substantial loss. Self-RL teachers: much lower loss before the signal fades.
→ Less remaining loss↓ Weaker signal proxy μ̂Each dot = one update
μ̂ is a noisy logged proxy, and this pattern is an association, not a causal test. What μ̂ does and does not measure →
How is the learning signal defined?
\hat{\mu}=\frac{\lVert\hat{g}\rVert^{2}}{2\,\hat{\mathcal{L}}}
Squared minibatch gradient norm divided by twice the OPD loss. Every raw update is included, without smoothing.
Loss is relative to update 1. The proxy uses a log scale, with different axis ranges for the two students. These measurements do not measure α.
03
THEORY
Nearby teachers. A local guarantee.
For a sufficiently close teacher in a shared parameterization, idealized OPD reduces loss exponentially, up to a residual. The guarantee is local and assumes regularity conditions.
LOCAL RECOVERY GUARANTEEIdealized OPD flow · local time interval
\mathcal{L}(\theta(t))\le e^{-\delta t}\,\mathcal{L}(\theta_0)+E_\Delta(\delta)+O\!\left(\rho^{3}+\frac{\rho^{4}}{\delta}\right)
\begin{aligned}\mathcal{L}(\theta(t))\le{}& e^{-\delta t}\,\mathcal{L}(\theta_0)\\&+E_\Delta(\delta)\\&+O\!\left(\rho^{3}+\frac{\rho^{4}}{\delta}\right)\end{aligned}
- Decaying initial loss
- The starting loss shrinks exponentially with time.
- Low-curvature contribution
- Residual from displacement along low-curvature directions.
- Higher-order remainder
- Controlled by teacher distance and the curvature threshold.
\rho Absolute teacher–student parameter distance.
\delta>0 Curvature threshold defining E_\Delta(\delta).
ALIGNMENT WITH EXPERIMENTS
Self-RL recovery in practice.
| Self-RL teacher / initial student | Relative teacher distance | Max. loss reduction | Final loss reduction |
|---|---|---|---|
| Qwen3-4B-RL-CodeCode · Qwen3-4B (Non-thinking) studentTop-16 · 1 rollout / prompt | 0.030% | 99.5% | 99.4% |
| JustRL-1.5BMath · R1-Distill-1.5B studentTop-16 · 1 rollout / prompt | 0.497% | 96.3% | 92.9% |
| JustRL-1.5BMath · R1-Distill-1.5B studentFull logits · 1 rollout / prompt | 0.497% | 96.2% | 91.8% |
| JustRL-1.5BMath · R1-Distill-1.5B studentTop-16 · 8 rollouts / prompt | 0.497% | 96.2% | 31.1%Late loss increase |
Consistent with the local picture: shared parameterizations, small relative distances, and strong loss recovery in code and math.
Relative distance = \frac{\lVert\theta_T-\theta_0\rVert}{\lVert\theta_0\rVert}. These measurements do not verify the theorem’s absolute-distance or curvature conditions.
04
REPRESENTATION CHANGES
Representations barely move.
Linear CKA > 0.98 at all measured layers after 200 updates—in both successful and stalled runs.
A possible explanation. Limited adaptation may suffice for nearby self-RL teachers while leaving larger teacher–student differences unresolved. Its role in signal collapse remains untested.
CKA ≈ 1: highly similar representations. Panels (a–c): before vs. after OPD.
05
ABLATIONS
Do training changes help?
Full logits, more rollouts, and an SFT warm start.
The teacher-dependent pattern persists.
Qwen3-4B-RL-Math outperforms Qwen3-4B in final accuracy and loss reduction across all three Qwen3-1.7B configurations.
Read the figure. Each column fixes a teacher. Top: validation accuracy. Bottom: loss relative to the first update. Line styles distinguish top-16, full logits, and eight rollouts.
Scope: one run per setting, 200 OPD updates. Conclusions apply to these recipes and budgets.
06
IN PRACTICE
Check your own OPD runs.
Both diagnostics use quantities most OPD trainers already log.
Watch the learning signal, not just the loss.
Compute \hat{\mu}_m=\hat{G}_m^{2}/(2\hat{\mathcal{L}}_m) from the logged gradient norm and loss at every step. With our larger teachers, it fell by about an order of magnitude while over 60% of the loss remained, and the loss then changed little. With self-RL teachers, the loss fell by more than 90% as it declined gradually.
Read the average decay rate off the loss curve.
Integrating Eq. (1) gives \bar{\gamma}(t)=\tfrac{1}{2t}\log\bigl(\mathcal{L}(\theta_0)/\mathcal{L}(\theta(t))\bigr), so the average rate needs only the loss curve. A plateau is a small \bar{\gamma} over that interval, and \gamma=\mu(1+\alpha) says it comes from a small μ or from α near or below −1.
Teacher proximity is what the guarantee covers.
The local guarantee applies to any teacher in the student’s own parameter space that is close enough and differs mostly along high-curvature directions, however it was trained. Our self-RL teachers sit 0.030% and 0.50% from their students; the larger teachers have different parameterizations and fall outside it.
Common training changes did not remove the gap.
In our ablations, full logits and eight rollouts per prompt left the teacher-dependent pattern in place (with eight rollouts, the JustRL loss later rose again), and an SFT warm start did not resolve the plateaus.
These patterns describe our runs (one run per setting, 200 updates); they are not tuned recommendations.
Scope & FAQ
Is the plateau just a learning-rate or training-budget effect?
We cannot rule this out. Every run uses AdamW with a constant learning rate of 1e-6 for a fixed 200 updates, with one run per setting. Controlled interventions across learning rates, training budgets and seeds are needed to separate optimization limits from representational limits.
Does μ̂ measure the true learning signal?
Not exactly. It uses a noisy, pre-clipping minibatch gradient, so its squared norm includes sampling variance, and it does not measure the size of an AdamW update. A decline in μ̂ alone does not establish an equal decline in ‖g‖². Cross-fitted estimates of ‖g‖² decline from early to middle training in Math, but stay roughly stable for Qwen3-14B after step 20.
Is the changing rollout distribution (α) to blame?
Not consistently. 1 + α̂ is positive at 19 of 20 checkpoints for Qwen3-14B, and at 12 and 13 of 20 for Skywork-7B and R1-Distill-7B. With only 20 checkpoints per teacher, these occupancy estimates are tentative.
Does the theory say only self-RL teachers work?
No. The proof does not use how the teacher was trained. It assumes an idealized gradient flow and a shared parameterization, while our runs use AdamW; small relative distances alone do not verify its absolute-distance and curvature conditions.
Do the results measure the effect of teacher size?
No. They describe the runs studied and are not a controlled estimate of the effect of teacher size or initialization. The association between μ̂ and the plateau does not establish causation.
Do the small parameter changes explain the stalls?
Not on their own. After 200 updates, the relative parameter change is 0.025–0.098% and CKA stays at least 0.985 at every layer, in stalled and successful runs alike. Whether the direction of these changes limits teacher matching is a hypothesis for controlled interventions.
07
CHECKPOINTS, CODE & REPRODUCTION
Reproduce the results.
Released checkpoints, training recipes, evaluation tools, and the data behind every figure.
Released checkpoints.
The final student of every run, after 200 OPD updates, in one 🤗 Hugging Face collection ↗. Teacher weights are not included. Accuracy is Code avg@4 / Math avg@8, archived from the paper runs; each model card lists the settings, per-benchmark scores, provenance and checksums.
| Checkpoint | Final loss reduction | Accuracy student → step 200 |
|---|---|---|
| Qwen3-4B-OPD-RL-Code-Step200 ↗Code · Qwen3-4B studentTeacher: Qwen3-4B-RL-Code (self-RL) | 99.4% | 52.7 → 60.4 |
| Qwen3-4B-OPD-Qwen3-14B-Step200 ↗Code · Qwen3-4B studentTeacher: Qwen3-14B | 25.9% | 52.7 → 53.4 |
| Qwen3-1.7B-OPD-RL-Math-Step200 ↗Math · Qwen3-1.7B studentTeacher: Qwen3-4B-RL-Math | 55.4% | 21.1 → 43.9 |
| Qwen3-1.7B-OPD-Qwen3-4B-Step200 ↗Math · Qwen3-1.7B studentTeacher: Qwen3-4B | 6.2% | 20.7 → 22.7 |
| R1-Distill-Qwen-1.5B-OPD-JustRL-Step200 ↗Math · R1-Distill-1.5B studentTeacher: JustRL-1.5B (self-RL) | 92.9% | 39.9 → 51.9 |
| R1-Distill-Qwen-1.5B-OPD-Skywork-7B-Step200 ↗Math · R1-Distill-1.5B studentTeacher: Skywork-7B | 23.9% | 39.9 → 46.1 |
| R1-Distill-Qwen-1.5B-OPD-R1-7B-Step200 ↗Math · R1-Distill-1.5B studentTeacher: R1-Distill-7B | 13.9% | 39.9 → 42.7 |
Load a checkpoint · Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "leizhao7/Qwen3-1.7B-OPD-RL-Math-Step200"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
# Qwen3 students: enable_thinking=False. R1-Distill students: omit it.From setup to a complete experiment.
Set up the environment, select a student–teacher pair, and compare with the reported results.
GPU required for training · Figures can be rebuilt on CPU
08
CITATION
Cite this work.
The paper, experiment code, diagnostics, and plotting assets are available.
BibTeX
@article{zhao2026opd,
title = {Why On-Policy Distillation Sometimes Fails:
Vanishing Learning Signals},
author = {Zhao, Lei and Zhao, Qichao and Zuo, Bowen and Zhan, Qishi},
journal = {arXiv preprint arXiv:2610.11247},
year = {2026},
url = {https://arxiv.org/abs/2610.11247}
}