DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

On-policy distillation (OPD) trains on student-generated trajectories using teacher feedback. DTDD adds a simple switch: keep OPD on the student prefix; learn a teacher-generated recovery suffix through behavior cloning, the usual cross-entropy loss on teacher tokens.

1. A. Student rollout. Generate a response and divide it into fixed-length token windows.
2. B. Divergence trigger. Average the student–teacher log-probability gap within each window and compare it with a threshold.
3. C. Teacher recovery. Retain the student prefix and regenerate from the start of the triggering window.
4. D. Role-split training. OPD trains the prefix; cross-entropy trains the recovery suffix. No trigger means ordinary OPD.

Why can token-level OPD feedback be insufficient?

A base student may need coherent demonstrations to acquire a compatible reasoning pattern. Long reasoning can become unstable, and an incorrect tool call can change all later observations. Local teacher feedback on the resulting trajectory may no longer be enough to show the student how to recover.

We study when teacher recovery stabilizes training, accelerates learning from a base initialization, and helps interactive agents.

Long-horizon stability

Student R1-Distill-1.5B · Teacher JustRL-1.5B · Data DAPO-Math-17K

With 15K- and 18K-token response budgets, DTDD suppresses the late gradient spikes seen in OPD while keeping held-out accuracy within one percentage point.

Peak gradient norm decreases from 12.02 to 1.40 at 15K and from 12.81 to 1.84 at 18K.

Measurement and stability controls

The full model names are DeepSeek-R1-Distill-Qwen-1.5B and JustRL-DeepSeek-1.5B. Peak gradient norms are measured from update 51 onward, after the initial optimization transient. The 7K response-budget setting serves as a stable OPD control.

Clipping the per-token OPD signal to [−1, 1] also stabilizes these runs. This control shows that teacher recovery is one way to suppress the late optimization event; it is not required for stability alone.

Learning from a base initialization

Student Qwen3-1.7B-Base · Teacher Qwen3-4B-Base-GRPO · Data DAPO-Math-17K

With frequent teacher intervention, DTDD reaches the 21.3% held-out accuracy reference level near update 70, compared with approximately 180 for direct OPD. Both later reach around 22%: the gain is in how early the student learns.

One training stage, no offline SFT pipeline

DTDD reaches comparable accuracy with fewer teacher-written tokens and less measured serial time than generating a teacher dataset, running SFT, and then training with OPD: 22.0 versus 42.8 hours and 43M versus 95M teacher-written tokens.

What is being compared?

The 2.6× comparison counts optimizer updates against direct OPD, not elapsed time. The serial-time comparison is a separate run: DTDD reaches 21.34% at update 80 in 22.0 hours; the offline SFT-then-OPD pipeline reaches 21.55% at update 20 in approximately 42.8 hours, including data generation, SFT, checkpoint merging, and OPD.

The offline stages use A40 and A100 GPUs; this DTDD run uses four L40S GPUs. These are measured pipeline times, not hardware-normalized compute savings: the corresponding totals are approximately 66 and 88 GPU-hours. The 95M and 43M figures count teacher-written tokens, not teacher scoring of student tokens.

The figure compares different token-window sizes and intervention controllers. Its fixed-threshold control starts around 25% takeover and adapts thereafter; it is not a constant 25% rate. The reasoning implementation generates the full student response before choosing the teacher recovery boundary, so takeover does not save student rollout computation.

Multi-turn agents

ALFWorld with LoRA

Student Qwen2.5-3B-Instruct · Teacher GiGPO-7B

Best-observed average success over the seen and unseen splits increases from 76.49% to 81.32%.

Agentic code

Student Qwen3-1.7B-SFT · Teacher DemyAgent-GRPO-4B · Data Open-AgentRL-30K, code subset

Best-observed accuracy averaged across four evaluation splits increases from 36.49% to 40.59%.

In ALFWorld, takeover falls from about 90% to near zero as training progresses, then reactivates around update 180. In agentic code, a sparse 2.8% teacher-token share improves all four splits at the selected best checkpoints.

Evaluation splits, selected checkpoints, and controls

Best-observed results select one checkpoint per method by the average across complete evaluations. ALFWorld uses updates 280 and 140 for OPD and DTDD; code uses updates 80 and 60. Code averages HO-200 full, HO-200 subset, LiveCodeBench v1, and LiveCodeBench v6.

At the last shared ALFWorld checkpoint (400), success is 70.06% for OPD and 77.59% for DTDD. Under full fine-tuning, the best-observed gain is smaller: +0.34 percentage point, compared with +4.83 with LoRA.

At the last shared code checkpoint (130), the four-split average is 33.93% for OPD and 34.96% for DTDD. The two LiveCodeBench splits average 42.26% and 38.43%, respectively; the held-out code splits average 25.60% and 31.50%. The best-observed and endpoint views therefore answer different questions about training.

We vary two choices: whether takeover is triggered by a segment average or a token spike, and whether teacher-written tokens receive cross-entropy or the OPD surrogate. The segment-triggered, cross-entropy combination performs best among the tested online variants at updates 60–120.

Same base-student DAPO setting as above. Direct OPD: 14.54%. DTDD and OPD are two-seed means; other variants are single runs.

Across 23 arms in 13 settings, 19 follow a common pattern: takeover tends to help when the teacher still outperforms the OPD-trained student. That gap is informative, but a larger gap does not guarantee a larger gain.

• A valid teacher. Can the teacher recover in this environment? In rebuilt WebShop, the task-specific teacher falls below OPD, making takeover unhelpful.
• A learnable recovery. Can the student learn the continuation? A 0.5B ALFWorld student does not reliably benefit despite a large gap to its 7B teacher.
• An appropriate dose. How much teacher text is useful? Base-model math benefits from frequent intervention; excessive intervention hurts agentic code.

@misc{he2026dtdddivergencetriggereddynamicdistillation,
  title={DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision},
  author={Qijia He and Zequn Li and Spencer Luo and Tianle Chen and Pradyumna Narayana and Zixian Ma and Ranjay Krishna and Lan Nie and Sugato Basu},
  year={2026},
  eprint={2610.19302},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2610.19302},
}