⚗️ Awesome On-Policy Distillation

A curated collection of papers, technical reports, frameworks, and tools for on-policy distillation (OPD) of large language models.

On-policy distillation trains a student on samples from its own evolving policy, while a teacher (external, privileged, or self-conditioned) provides dense supervision on those same samples.

On-policy distillation (OPD) trains a student on trajectories sampled from its own policy while a teacher scores the student-visited prefixes with dense token-level guidance. This on-policy data collection reduces the train-inference distribution gap that affects off-policy KD/SFT on fixed traces. Depending on the estimator, OPD looks like GKD on student rollouts or policy-gradient/RL with teacher-defined per-token KL/log-prob rewards, making the natural contrast sparse outcome-reward RL rather than RL as a whole. As of 2026, OPD is a standard post-training primitive at Alibaba (Qwen3), DeepSeek (V4), Xiaomi (MiMo), Zhipu (GLM-5), NVIDIA (Nemotron-Cascade 2), and others.

Shipping today? Jump to Frameworks and Implementations. New to OPD? Read Start Here.

Contents

• Start Here
• Surveys and Essays
• Core OPD Papers

• Taxonomy
• Adjacent and Enabling Work
• Domain Extensions

• Technical Reports and Industrial Recipes
• Frameworks and Implementations
• Acknowledgments
• Contributing
• Citation

Start Here

A fast path through the field:

• Survey. OPD Survey — taxonomy, methods, and open problems in one place.
• Foundations. MiniLLM, GKD, and ExOPD — the core student-rollout plus teacher-supervision loop, including its dense KL-constrained RL framing.
• Practical intuition. Thinking Machines blog — the clearest end-to-end explanation of why and when OPD applies.
• When OPD works and when it breaks. Revisiting OPD, Entropy-Aware OPD, and Rethinking OPD — failure modes (instability, diversity collapse, tokenizer mismatch) and success conditions (compatible thinking patterns, novel teacher capability).
• No teacher logits. Black-Box OPD — discriminator-based reward when the teacher is API-only.
• No teacher at all. OPSD and SDFT — same model as student and self-teacher.
• Context and experience. OPCD and OEL — distill prompts and deployment traces into weights.
• Industrial recipes. Qwen3, DeepSeek-V4, MiMo-V2-Flash, GLM-5 — how labs ship OPD in production.

Surveys and Essays

Surveys and Position Papers

• A Survey of On-Policy Distillation for Large Language Models (2026) — First dedicated OPD survey; organizes methods by feedback signal, teacher access mode, and loss scope.
• A Brief Overview: On-Policy Self-Distillation in Large Language Models (2026) — Beginner-oriented overview of on-policy self-distillation, cataloguing privileged-context designs where a single model is both teacher and student.
• Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillation (2026) — Reframes SFT/RL/OPD by training-state source rather than loss, explaining why OPD's student-sampled states beat a degraded teacher.
• On the Geometry of On-Policy Distillation (2026) — Parameter-space diagnostics placing OPD in a relaxed off-principal regime with early subspace locking, a distinct update geometry between SFT and RLVR.
• Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation (2026) — Analyzes OPD checkpoint deltas, showing dense teacher supervision still yields sparse, off-principal weight edits resembling RLVR rather than dense supervised rewriting. (Code)
• A Formula-Driven Survey and Research Agenda for On-Policy Distillation (2026) — Formula-driven OPD survey and research agenda splitting temporal credit from vocabulary routing.
• Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning (2026) — Unifies post-training by data-policy mismatch, with a dedicated framework for on-policy teacher-guided transfer.

Essays, Blog Posts, and Walkthroughs

• Thinking Machines: On-Policy Distillation (2025) — Best single-article introduction. Covers concepts, intuition, and practical use cases.
• Unlocking On-Policy Distillation for Any Model Family (GOLD) (2025) — Cross-tokenizer OPD walkthrough with TRL code.
• The Magic of LLM Distillation — Rishabh Agarwal, Google DeepMind (2025) — Latent Space talk tracing distillation from logits to on-policy student rollouts, framed as the DAgger-style fix for exposure bias.
• Distilling 100B+ Models 40x Faster with TRL (2026) — HF engineering walkthrough of TRL's DistillationTrainer scaling tricks; ~40× speedup, validated on Qwen3-235B → Qwen3-4B math.
• Multi-Teacher On-Policy Distillation: A New Post-Training Primitive (2026) — Yumo Xu surveys MOPD as a post-training primitive across MiMo-V2-Flash, GLM-5, Nemotron-Cascade 2, DeepSeek-V4.
• On-Policy Distillation: Theory & Practice in Model Merging (2026) — ByteDance Seed framing OPD as entropy-regularized RL; cross-tokenizer pitfalls and reward hacking in agent merging.
• On SFT, RL, and on-policy distillation (2026) — Will Brown's essay on OPD via SFT-vs-RL compounding and gradient geometry; pointers toward an optimal teacher.
• SFT, RL, and OPD Through a Distributional Lens (2026) — wh's distributional-geometry framing; experiment shows OPD students from SFT and RL teachers converge and forget less.
• On Policy Self Distillation (2026) — KL-geometry study showing OPSD inverts OPD's per-token sign and suffers larger KL shocks that GEPA hint evolution roughly halves.
• What Apple found out about On-Policy Distillation (2026) — AVB's tutorial-style breakdown of "Unmasking OPD"; training-free gradient-alignment for predicting student-teacher fit.
• OPD深度解析：从数学推导到DeepSeek V4、SWIFT与verl实践 / OPD Deep Dive: From Mathematical Derivation to DeepSeek V4, SWIFT, and verl Practice (2026) — Chinese-language Zhihu deep-dive deriving OPD's sequence- and token-level reverse-KL; maps variants to MiniLLM, GKD, verl, DeepSeek V4.
• 重温 On-Policy Distillation / Revisiting On-Policy Distillation (2026) — Chinese-language notes deriving OPD as both a SeqKD student-rollout mirror and RL with token-level teacher supervision.
• The Imitation Game: State of Policy Distillation in Language Model training (2026) — Long-form OPD/OPSD survey with a four-axis failure-modes taxonomy; argues hybrid OPSD and cross-tokenizer OPD as the highest-leverage open problems.
• Scaling SDPO (2026) — Trajectory field note extending self-distillation policy optimization to stale off-policy production rollouts for single-trajectory continual learning on live interactions.
• My Continuous Learning Pipeline with Self-Distillation on Agent Traces (2026) — HuggingFace continuous-learning walkthrough that self-distills coding-agent traces with TRL's SDPO trainer, using an evaluator model's targeted feedback as privileged context.
• On-Policy Distillation: Promise, Pitfalls, and Prospects (2026) — Li Jiang organizes recent OPD failure analyses into three connected mechanisms — local teacher noise, horizon coverage decay, and myopic per-token supervision.
• Solving OPSD (basically) (2026) — Continuation arguing OPSD's positive teacher-agreement pressure carries the useful signal while negative pressure drives length collapse, so keeping only positive matches OPD.
• Why On-Policy Distillation Works and Naive Self-Distillation Doesn't (2026) — Frames on-policy distillation as KL-regularized RL toward a reward-tilted teacher, explaining why naive self-distillation distills a feedback-hallucinating template.
• Self-Distilled Reasoner: On-Policy Self-Distillation (2026) — Author walkthrough deriving on-policy self-distillation from a privileged self-teacher as dense token-wise distribution matching, with a policy-gradient interpretation.
• On-policy Distillation (2026) — Accessible long-form guide framing on-policy distillation as teacher-graded student trajectories that fix exposure bias, built around the GKD loop.
• Understanding Self-Distillation and Privileged Information Distillation (2026) — Walkthrough deriving self-distillation and privileged-information distillation through an RL-as-variational-inference lens, spanning reverse-KL self-teachers to reward-tilted joint training.

Core OPD Papers

The papers that define on-policy distillation for LLMs.

Scope rule: A paper belongs here if its primary contribution is a new component of the OPD training loop itself — an objective, divergence formulation, stability fix, teacher access-mode variant, self-distillation variant, context-internalization mechanism, or systems/efficiency/privacy constraint applied to that loop — with student rollouts central to the learning signal, evaluated on LLM text generation or reasoning. Operational test: if removing the OPD-loop component leaves a working contribution (a working RL recipe, preference method, or KD baseline), the OPD piece is auxiliary → Adjacent. Papers that enable OPD (cross-tokenizer alignment, calibration), compose with OPD as one component of a larger RL/preference structure, or apply OPD to non-text-reasoning substrates live in Adjacent and Enabling Work or Domain Extensions.

Foundations

• MiniLLM: On-Policy Distillation of Large Language Models (2023) — Reverse-KL framing for generative LMs; the paper that named the field. (Code)
• GKD: On-Policy Distillation of Language Models — Learning from Self-Generated Mistakes (2023) — Unifying formulation spanning on-/off-policy mixtures with flexible divergences.

Gap-Bridging

• Speculative Knowledge Distillation (2024) — Interleaved teacher/student sampling mitigates poor student rollout quality. (Code)
• Black-Box On-Policy Distillation of Large Language Models (2025) — GAD: discriminator-based reward on student rollouts; no teacher logits required. (Code)
• SOD: Step-wise On-policy Distillation for Small Language Model Agents (2026) — Reweights teacher guidance by step-level divergence to avoid tool-induced cascade drift.
• MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate (2026) — Multi-agent debate consensus as the OPD teacher; extends to agentic tasks via step-level sampling.
• ROPD: Rubric-based On-policy Distillation (2026) — Black-box OPD using prompt-specific rubrics distilled from teacher-student contrasts to score rollouts. (Code)
• Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation (2026) — Backtracks straying student rollouts to the last safe state for teacher correction, targeting the reversed exposure bias on-policy distillation introduces. (Code)
• Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation (2026) — Counteraction-aware multi-teacher OPD that decouples conflicting recovery and preservation gradients, recovering general capability from proxy prompts without teacher-aligned prompt coverage.
• Trust-Region Behavior Blending for On-Policy Distillation (2026) — Warmup samples early prefixes from a teacher-blended behavior policy within a student-centered KL trust region, annealed to zero by warmup's end.
• Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance (2026) — Spreads teacher guidance across a near-future token window, using trajectory drift to find true reasoning forks rather than high-loss single tokens.
• Trust Region On-Policy Distillation (2026) — Restricts reverse-KL distillation to teacher-reliable trust regions on student rollouts, applying forward-KL to mismatched outlier tokens instead.
• OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification (2026) — Replaces teacher logits with chunk-level semantic verification from Monte Carlo rollouts, enabling on-policy distillation from black-box teachers.
• Trajectory-Refined Distillation (2026) — Teacher revises the student's failed rollout within on-policy support before distilling, correcting prefix failure at the trajectory rather than token level. (Code)
• SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models (2024) — Teacher selectively intervenes during student generation at high token-probability discrepancies, curbing teacher misguidance over long student rollouts.
• Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients (2026) — Embeds teacher candidate answers into RL prompts so the student learns on-policy, recovering signal on hard questions where every rollout fails.
• SGPO: Strategy-Guided Policy Optimization for LLM Reasoning (2026) — Distills reusable strategy descriptions from teacher responses via forward-KL on student trajectories, replacing instance-level trajectory imitation.
• MOPD: Multi-Teacher On-Policy Distillation for Capability Integration (2026) — Distills multiple per-domain RL teachers into one student on its own rollouts, integrating their specialized capabilities into a single model.
• Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection (2026) — Student perplexity steers the teacher's chunk-by-chunk generation toward learnable reasoning paths, selecting branches during generation instead of filtering afterward.
• Weak-to-Strong Generalization via Direct On-Policy Distillation (2026) — Applies a weak teacher's pre/post-RL log-ratio as a dense reward on student rollouts, transferring the RL-induced shift rather than the final policy.
• CoPD: Co-Evolving Policy Distillation (2026) — Parallel expert training with bidirectional OPD; experts co-evolve as mutual teachers during RLVR.
• CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation (2026) — Schedules a per-token forward-to-reverse-KL mixture on student rollouts, gating the transition on student coverage of teacher tokens rather than time.
• Weak-to-Strong On-Policy Distillation (2026) — Synthesizes a proxy teacher from two weak models' logit difference anchored on the student's base, lifting a student stronger than both. (Code)
• Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold (2026) — Reliability-gated multi-teacher on-policy distillation whose teachers are deliberately trained on examples the accumulated teacher union fails to cover.
• Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models (2026) — Builds a corrective teacher from how successful versus failed weak-model patches shift continuations at the strong model's prefixes, never imitating the patches.
• RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection (2026) — Contrasts rollout-conditioned teacher guidance with retention anchors, injecting knowledge without erasing existing capabilities.
• Pass the Baton: Trajectory-Relayed On-Policy Distillation (2026) — Lets the teacher repair failed prefixes before returning control, keeping supervision near the student's reachable trajectory. (Code)
• CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning (2026) — Finds the first causal error with a teacher, then repairs short continuations through on-policy reinforcement.

Stability and Objective Design

• DistiLLM: Towards Streamlined Distillation for Large Language Models (2024) — Skew-KL divergence with adaptive off-policy use of student-generated outputs; foundational OPD objective formulation. (Code)
• DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs (2025) — Contrastive extension of skew-KL; student-generated outputs collected per epoch. (Code)
• Veto: Stable On-Policy Distillation through Adaptive Target Reformulation (2026) — Intermediate target distribution in logit space stabilizes training.
• Entropy-Aware On-Policy Distillation of Language Models (2026) — Forward-KL on high-entropy teacher tokens preserves output diversity.
• ExOPD: Learning beyond Teacher via Generalized On-Policy Distillation with Reward Extrapolation (2026) — Casts OPD as dense KL-constrained RL; reward scaling enables teacher-surpassing behavior. (Code)
• Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention (2026) — Black-box oracle edits student failures into proximal rollouts for reward-based distillation that preserves prior knowledge. (Code)
• REOPOLD: Scaling Reasoning Efficiently via Relaxed On-Policy Distillation (2026) — Relaxes imitation with reward clipping, entropy-based dynamic sampling, and explore-to-refine training.
• PACED: Distillation at the Frontier of Student Competence (2026) — Pass-rate weighting focuses learning on the student's competence frontier.
• Revisiting On-Policy Distillation — Empirical Failure Modes and Simple Fixes (2026) — Truncated reverse-KL with teacher top-K support matching; fixes imbalanced signals and tokenizer mismatch. (Code)
• Rethinking On-Policy Distillation — Phenomenology, Mechanism, and Recipe (2026) — Identifies compatible thinking patterns and novel teacher capability as OPD success conditions.
• The Illusion of Certainty — Decoupling Capability and Calibration in OPD (2026) — Diagnoses OPD-induced overconfidence; CaOPD replaces confidence targets with student-grounded empirical success rates.
• Demystifying OPD — Length Inflation and Stabilization Strategies (2026) — Repetition-driven length inflation in iterative OPD; Stable-OPD adds divergence constraints and a rollout-mixture anchor.
• SCOPE: Signal-Calibrated On-Policy Distillation with Dual-Path Adaptive Weighting (2026) — Routes correct student rollouts to student-PPL-weighted MLE and incorrect to teacher-PPL-weighted KL; dual-path OPD loss design.
• HPD: Hybrid Policy Distillation for LLMs (2026) — Unified reweighted-log-likelihood framework combining forward/reverse KL with off-policy and on-policy sampling. (Code)
• Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe (2026) — Offline difficulty-aware and online correctness-aware data balancing with outcome-guided margin calibration. (Code)
• AOPD: Asymmetric On-Policy Distillation (2026) — Replaces ineffective negative reinforcement with localized teacher-distribution matching in non-positive advantage regions.
• vOPD: On-Policy Distillation with a Control Variate Baseline (2026) — Closed-form per-token reverse-KL value baseline; unbiased lower-variance single-sample estimator with no extra critic.
• Unmasking On-Policy Distillation — Where It Helps, Where It Hurts, and Why (2026) — Training-free gradient-alignment diagnostic; best teacher flips with student capacity and task; wrong demos hurt self-distillation except on hard math.
• The Many Faces of On-Policy Distillation — Pitfalls, Mechanisms, and Fixes (2026) — Names three failure modes (student-prefix teacher-state mismatch, biased Top-K gradients, PI-free OPSD aggregation) and three stabilizers (stop-grad Top-K KL, RLVR teachers, SFT-stabilized students).
• Rock Tokens — Deciphering High-Loss Tokens in On-Policy Distillation (2026) — High-loss tokens (up to 18%) persist after apparent convergence; masking them streamlines alignment. (Code)
• BRTS: On-Policy Distillation with Best-of-N Teacher Rollout Selection (2026) — Auxiliary teacher-context branch alongside standard OPD; selects best-of-N teacher rollouts by correctness then student-alignment. (Code)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation (2026) — Dynamic release rule truncates dense supervision where the teacher's local margin collapses; counters suffix degradation in strong-to-weak OPD.
• Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for LLM Post-Training (2026) — Sparse-to-dense post-training workflow framing OPD as the dense teacher-induced reward between GRPO stages.
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs (2026) — Reward-extrapolation OPD collapses past a clip threshold on near-deterministic structured outputs, mapping where teacher-surpassing reward scaling stops working.
• MOPD: Multi-Rollout On-Policy Distillation via Peer Successes and Failures (2026) — Conditions the teacher on successful and failed peer rollouts from the student's local group, sharpening token-level supervision over independent per-rollout distillation.
• Teacher-Guided Policy Optimization for LLM Distillation (2026) — Feeds teacher tokens conditioned on the student's rollout as explicit on-policy-SFT targets, replacing reverse-KL's uninformative negative feedback under large teacher gaps.
• Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation (2026) — Applies OPD loss only to "teachable" tokens where the teacher's corrective mass lands within the student's support, separating learnable from incompatible disagreement.
• AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment (2026) — Reflection-bottlenecked privileged self-distillation converting diagnostics into ReLU-gated token-level advantages, preventing the late-stage collapse of raw-oracle conditioning.
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation (2026) — Rewards the student's top-K candidate tokens by the teacher confidence they induce one step ahead, countering supervision-fidelity decay over long reasoning chains.
• OPD+: Rethinking the Advantage Design for On-Policy Distillation (2026) — Corrects on-policy distillation's biased stop-gradient advantage estimator, generalizing the objective to any f-divergence beyond the usual reverse KL.
• SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment (2026) — Confines reverse-KL on-policy distillation to a mined sparse subset of safety tokens, aligning behavior while sidestepping the alignment tax.
• FiRe-OPD: Filter, Then Reweight — Rethinking Optimization Granularity in On-Policy Distillation (2026) — Drops low teacher-likelihood trajectories, then soft-reweights remaining tokens by teacher confidence and student confusion, replacing hard token selection. (Code)
• Reinforcement Learning from Rich Feedback with Distributional DAgger (2026) — Distills a feedback-conditioned teacher via forward cross-entropy on student-visited states, provably guaranteeing the monotonic policy improvement reverse-KL self-distillation lacks. (Code)
• OPRD: On-Policy Representation Distillation (2026) — Supervises student rollouts by aligning intermediate hidden states with the teacher across layers, replacing high-variance output-space logit matching. (Code)
• Physics-Guided Policy Optimization with Self-Distillation (2026) — Modulates self-distillation step size by student-teacher mutual information, stabilizing privileged-feedback updates that otherwise collapse late in training.
• Constitutional On-Policy Safe Distillation (2026) — Recalibrates a constitution-conditioned teacher via cross-SFT cold-start before on-policy distillation, preventing reverse-KL collapse into over-cautious safety refusals.
• When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation (2026) — Gates each historical-teacher refresh on consolidation evidence, replacing clock-driven updates to prevent state-oblivious collapse in self on-policy distillation.
• SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling (2026) — Routes each distillation token by sign agreement between a binary verifier and the teacher, extrapolating consensus tokens and interpolating conflicts.
• Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation (2026) — Unifies SFT, DAgger, offline RL, and on-policy distillation by decomposing the distillation KL along prefix-source and direction axes.
• Escaping the KL Agreement Trap in On-Policy Distillation (2026) — Terminates student rollouts trapped in persistent low-KL teacher agreement, filtering degenerate supervision that yields no corrective on-policy signal.
• Validity-Calibrated Reasoning Distillation (2026) — Reweights per-token distillation updates by the relative local validity of teacher and student next-step proposals, instead of uniform trajectory imitation.
• PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation (2026) — Replaces on-policy distillation's unbounded log-ratio reward with a natively bounded Box–Cox power transformation, curbing high-variance gradients that post-hoc scaling cannot. (Code)
• On the Position Bias of On-Policy Distillation (2026) — Importance-Weighted OPD upweights each token by its cumulative student-teacher prefix discrepancy, correcting position bias where later rollout tokens drift off-teacher. (Code)
• Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillation (2026) — Finds decision tokens via student entropy and evidence tokens via hidden-state similarity plus teacher-student divergence for selective OPD loss.
• ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation (2026) — Reweights each student trajectory by a student-teacher log-ratio score over pivotal tokens, emphasizing likely-negative rollouts without observing the final answer. (Code)
• Blockwise Policy-Drift Gating for On-Policy Distillation (2026) — Reweights OPD position losses with student-only block-level policy-drift gates, stabilizing reused rollouts without altering teacher targets.
• SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (2026) — Uses joint teacher-student entropy as a competence probe to tailor token-level supervision, addressing the competence-dependent degradation uniform on-policy distillation ignores.
• DOPD: Dual On-Policy Distillation (2026) — Routes each token's supervision between a privileged teacher and privileged student by advantage gap, disentangling closable capability gaps from un-replicable information asymmetry.
• Reward-Gated On-Policy Distillation (2026) — Gates teacher distillation on each rollout by agreement between verifier reward sign and teacher–student likelihood gap, trusting only directionally reliable teacher signal.
• Trust Region Policy Distillation (2026) — Interpolates teacher and student probabilities into a proximal teacher, replacing on-policy distillation's unbounded log-ratio reward with a provably variance-bounded signal.
• Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation (2026) — Anchors suppression-prone tokens to a frozen-base reference prior via an asymmetric divergence gate, preventing OPSD's collapse of native reasoning.
• On-Policy Delta Distillation (2026) — Rewards on-policy distillation by a delta signal, the teacher minus its pre-tuning base, transferring reasoning traces rather than inherited preferences.
• Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations (2026) — Diagnoses length-exploitation shortcuts in token-level on-policy distillation, regulating the advantage with hard clipping or order-preserving log-scale compression.
• Distilled Reinforcement Learning for LLM Post-training (2026) — Weights student rollouts by clipped teacher-to-student token ratios inside the RL objective, replacing on-policy distillation's unconditional logit matching.
• Outcome-Confounded Local Supervision in On-Policy Distillation (2026) — Crosses token-level teacher-student divergence with final-answer correctness, finding agreement-on-failure dominates because local divergence cannot localize where trajectories became unrecoverable.
• β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation (2026) — Distills a scheduled reference-to-teacher logit interpolant derived as the optimal policy of a KL-regularized family generalizing vanilla self-distillation.
• Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models (2026) — Subtracts the cross-rollout-predictable part of teacher-reference disagreement, relaxing the requirement that the distillation teacher generated the fine-tuning data.
• SAF-OPD: Stable Advantage Fusion for On-Policy Distillation (2026) — Rescales the token-level distillation advantage into a bounded, time-decaying signal before fusing it with GRPO's, preventing fixed-coefficient entropy collapse.
• Not Every Divergence Should Be Suppressed: Counterfactual Recoverability in On-Policy Distillation (2026) — Labels each student error state as recoverable or not by replaying teacher-continuation and rollback branches, replacing divergence-based supervision selection.
• SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation (2026) — Reshapes teacher targets at sparsely probed positions using verifier-scored student continuations, correcting reverse-KL's neglect of plausible alternatives.
• On-Policy Delta Distillation for Multilingual Math Reasoning (2026) — Empirical study of delta-signal on-policy distillation for multilingual math reasoning, finding English-only training transfers accuracy but drags responses toward English.
• DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models (2026) — Weights self-distillation tokens by backward multi-step aggregation of divergence gaps, restoring the order dependence vanilla OPSD's uniform coefficients discard. (Code)
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance (2026) — Applies teacher guidance only to zero-reward rollout groups, restoring gradients where group-relative reinforcement learning stalls.
• DAPD: Dual-Anchored Policy Distillation (2026) — Dual self-teacher anchors constrain privileged targets, preventing unreachable guidance from destabilizing student rollouts. (Code)
• When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation (2026) — Filters teacher signals by input grounding plus distributional extremity, suppressing confident but spurious guidance. (Code)

Self-Distillation

• OPSD: Self-Distilled Reasoner (2026) — Single model as both teacher and student via privileged information; no external teacher.
• SDFT: Self-Distillation Enables Continual Learning (2026) — Demonstration-conditioned self-teaching for continual learning with less forgetting. (Code)
• SDPO: Reinforcement Learning via Self-Distillation (2026) — Converts textual feedback into dense self-teacher signals for RL-like training. (Code)
• Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? (2026) — Traces failures to suppression of epistemic verbalization; task coverage determines whether conciseness helps. (Code)
• OPSDC: On-Policy Self-Distillation for Reasoning Compression (2026) — Compresses verbose reasoning using concise privileged self-teachers. (Code)
• BIRD: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning (2026) — SFT-bootstraps the rollout distribution toward cleaner prefixes before on-policy reverse-KL self-distillation, fixing OPSD's cold-start prefix bottleneck. (Code)
• GATES: Self-Distillation under Privileged Context with Consensus Gating (2026) — Consensus-gated asymmetric-context self-distillation without labels or rewards.
• HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation (2026) — Privileged self-distillation on cliff prompts where RL gradients vanish; recovers KL-regularized optimal policy.
• RLSD: Self-Distilled RLVR (2026) — Self-distillation as token-level credit assignment within GRPO; OPSD-style matching leaks privileged information.
• SDZero: Self-Revision Turns Binary Rewards into Dense Supervision (2026) — Generator-reviser dual roles; reviser converts binary feedback into token-level supervision with no external teacher.
• OPSDL: On-Policy Self-Distillation for Long-Context Language Models (2026) — Short-context distribution of the same model as co-evolving reverse-KL teacher under long context.
• PBSD: Preference-Based Self-Distillation — Beyond KL Matching via Reward Regularization (2026) — DPO-style preference learning between context-augmented teacher positives and on-policy student negatives.
• UniSD: Towards a Unified Self-Distillation Framework for Large Language Models (2026) — Unifies self-distillation across supervision reliability, representation alignment, and training stability. (Code)
• OPSD Compresses What RLVR Teaches — A Post-RL Compaction Stage (2026) — Correct-only OPSD preserves accuracy and shortens responses; proposes SFT → RLVR → OPSD as post-RL compaction.
• ATESD: Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning (2026) — Treats teacher reveal ratio as a learnable control variable via Beta-policy controller with discounted learning-progress reward.
• OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering (2026) — Contrasts averaged teacher logits over correct vs. incorrect rollouts to form outcome-guided steering on anchor logits.
• RLRT: Rebellious Student — Reversing Teacher Signals for Reasoning Exploration (2026) — Upweights student tokens that diverged from teacher but still succeeded as a "valuable exploration" signal added to GRPO.
• EGRSD: Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning (2026) — Teacher-entropy confidence gate over RLSD's direction-magnitude signal; causal-lookahead variant preserves transient pivot tokens.
• CREDIT: From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation (2026) — Recasts the self-distillation token reward as Bayesian filtering; batch-contrastive teacher baseline strips input-generic shortcuts.
• OPHSD: Training with Harnesses — On-Policy Harness Self-Distillation for Complex Reasoning (2026) — Generalizes self-distillation privileged context from a static variable (reference solution, environment trace) to a harness-driven workflow (draft-verify, plan-solve); harness is a removable training scaffold. (Code)
• MixSD: Mixed Contextual Self-Distillation for Knowledge Injection (2026) — Per-token Bernoulli mix of fact-conditioned and naive-conditioned base-model samples; replaces SFT for knowledge injection without collapsing held-out capability.
• AntiSD: Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information (2026) — Identifies the OPSD token reward as a PMI that suppresses deliberation tokens, then reverses its sign under an entropy-triggered gate. (Code)
• TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment (2026) — Routes self-distillation KL only to annotator-marked spans to cure the all-token "distillation tax" of SDPO/SRPO.
• AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals (2026) — Multi-view privileged self-distillation that gates teacher-specific residuals so they can adjust update magnitude but cannot reverse the cross-view consensus direction. (Code)
• It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs (2026) — Two same-model teachers conditioned on self-generated feedback supply complementary reverse-KL targets on the student's own rollouts, intersected as a product-of-experts.
• VPD: Learning from Language Feedback via Variational Policy Distillation (2026) — Variational-EM self-distillation refines a feedback-conditioned self-teacher in the E-step before distilling it back via token-level KL in the M-step.
• RMSD: Bringing Capabilities in Distribution via Relevance-Masked Self-Distillation (2026) — Applied Compute's OPSD variant masking the reverse-KL loss to LLM-judge-selected behavior-relevant tokens; preserves capabilities where SFT collapses.
• SPD: Self-Policy Distillation via Capability-Selective Subspace Projection (2026) — Decode-time KV-subspace projection biases self-rollout generation toward capability-relevant directions, then LoRA-SFTs on those rollouts without any external verifier or teacher.
• Multilingual Safety Alignment via Self-Distillation (2026) — Same-model OPSD transfers English safety reasoning to low-resource languages without any response data.
• COPSD: Crosslingual On-Policy Self-Distillation for Multilingual Reasoning (2026) — Uses English translations and reference solutions as privileged teacher context for low-resource multilingual reasoning OPSD. (Code)
• EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation (2026) — Guides a fraction of student rollouts with the privileged context, then distills only positive-evidence tokens, internalizing rare identities OPSD never samples.
• When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning (2026) — Weights OPSD token supervision by within-sequence position, the strongest tested predictor of privileged-teacher reliability, rather than ambiguous teacher entropy.
• Ditto: Reinforcing Human Behavior Simulation via Verbal Feedback (2026) — Jointly GRPO-optimizes a draft rollout and its judge-feedback-conditioned refinement so the policy internalizes verbal guidance, targeting subjective human-simulation rather than verifiable rewards.
• OISD: On-Policy Internal Self-Distillation of Language Models (2026) — Distills the detached final layer into an intermediate layer across model depth via advantage-weighted Jensen–Shannon alignment — needs no privileged context.
• ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domains (2026) — Reflection-guided OPSD restricting self-teacher distillation to a rollout's erroneous span, targeting cross-domain reasoning generalization.
• SGSD: Skill-Conditioned Gated Self-Distillation for LLM Reasoning (2026) — Skill-conditioned OPSD whose retrieved-skill teachers are outcome-validated before distillation, extending privileged self-distillation to unreliable experience-derived context. (Code)
• Distilling LLM Feedback for Lean Theorem Proving (2026) — Distills a self-teacher conditioned on LLM-generated critique of the student's attempt, injecting external knowledge through natural-language feedback rather than logits or solutions.
• CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO (2026) — Answer-free correctness-conditioned self-teacher bidirectionally flips GRPO token-advantage signs, unlike the privileged-context teachers of related self-distillation methods.
• SC-SDPO: Restoring the Sweet Spot via Pass-Rate Weighted Self-Distillation (2026) — Reweights SDPO's self-distillation loss by an on-the-fly pass-rate term, restoring the difficulty sweet spot that pure self-distillation discards.
• Self-Supervised On-Policy Distillation for Reasoning Language Models (2026) — Conditions a self-teacher on a successful peer completion to densely supervise failed on-policy prefixes within each GRPO group.
• Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation (2026) — On-policy self-distillation for safety using a privileged-context self-teacher, with flip-rate prompt search selecting contexts that activate latent refusal.
• Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning (2026) — Distills a temperature-scaled copy of the model's own logits to restore entropy in RL-collapsed policies before continued training.
• SDPG: Self-Distilled Policy Gradient (2026) — Couples GRPO verifier rewards with exact full-vocabulary privileged self-distillation, casting its reverse-KL term as an equivalent centered log-ratio policy gradient. (Code)
• Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning (2026) — Routes signed-KL self-distillation by student entropy, pulling low-entropy tokens toward the privileged self-teacher while pushing high-entropy tokens away to preserve exploration.
• AR-OPD: Anchored Residual Guidance for Privileged On-Policy Distillation (2026) — Splits privileged supervision into a locally reachable partial-oracle anchor plus a scaled full-view residual, curbing hindsight leakage from monolithic full-privilege imitation.
• The Role of Feedback Alignment in Self-Distillation (2026) — Compares self-teacher context forms, showing step-aligned critique matched to the solver's reasoning trace localizes distillation credit at errors unlike reference solutions.
• RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation (2026) — Contrasts the privileged-hint self-teacher gap under correct versus wrong hints, cancelling style-token drift to concentrate distillation signal on task-bearing tokens. (Code)
• RGSD: Rubric-Guided Self-Distillation (2026) — Conditions the frozen self-teacher on per-prompt grading rubrics, distilling dense token-level targets into the prompt-only student to replace verifier-scored rubric reinforcement learning.
• PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners (2026) — Privileged self-distillation re-scoring student rollouts under an adaptively masked reference solution, calibrating only high-entropy-mismatch tokens. (Code)
• OPCoD: Be My Tutor — On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback (2026) — Bidirectional co-distillation where paired students condition each self-teacher on peer natural-language feedback, targeting mutual cross-domain Pareto improvement without external teachers.
• Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation (2026) — Splices a student's own wrong rollouts with self-generated correct references into explicit error-correction trajectories, replacing logit-level self-distillation.
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation (2026) — Conditions the privileged teacher on learned rubrics to give criterion-aware token-level guidance on student rollouts, replacing reference-trajectory supervision.
• Self-Distillation as a Performance Recovery Mechanism for LLMs (2026) — Anchors a degraded model to a frozen earlier checkpoint via on-policy self-distillation, recovering capabilities lost to fine-tuning or compression.
• On-Policy Self-Distillation Reduces Output Diversity (2026) — Sampled-demonstration teacher tilts the base distribution by pointwise conditional mutual information, collapsing rollout diversity unlike on-policy RL.
• PHF: Privileged Hidden Flow for On-Policy Self-Distillation (2026) — Distills how the privileged teacher's hidden states move along the rollout, supervising internal computation that output-only self-distillation leaves untouched.
• Learning from Partial Chain-of-Thought via Truncated-Reasoning Self-Distillation (2026) — Student matches a frozen self-teacher's answer distribution while conditioned only on a truncated reasoning prefix, building robustness to inference-time truncation.
• Purified OPSD: On-Policy Self-Distillation Without Losing How to Think (2026) — Subtracts a reference-only teacher probe to isolate the transferable residual, distilling a closed-form PMI target that preserves long-CoT reflection.
• DemoPSD: Disagreement-Modulated Policy Self-Distillation (2026) — Targets a disagreement-modulated reverse-KL barycenter of privileged teacher and student, pulling supervision toward the student where high divergence signals privileged-information leakage.
• Neuron-OPSD: Neuron-Aware Data Selection for Annotation-Free LLM Self-Distillation (2026) — Derives both training-data selection and teacher context from internal neuron activations, driving annotation-free on-policy self-distillation without external oracles.
• Denser ≠ Better: Limits of On-Policy Self-Distillation for Continual Post-Training (2026) — Empirical-theoretical critique showing dense on-policy self-distillation forgets more than GRPO during continual post-training, as teacher projection induces excess parameter drift. (Code)
• Rethinking On-Policy Self-Distillation for Thinking Models (2026) — Diagnostic study showing privileged self-distillation degrades thinking models on long reasoning, as teacher context lowers fork rates at high-entropy positions.
• CANON: Consensus as Privileged Context for Label-Free Self-Distillation (2026) — Turns agreement among a model's own samples into privileged context for a frozen self-teacher, densely supervising fresh rollouts without labels.
• Enhancing Rubric-based RL via Self-Distillation (2026) — Distills a rubric-conditioned self-teacher into the policy via forward-KL on the model's own rollouts, rescuing criteria that scalar reward aggregation suppresses.
• PromptSD: One Student, Many Teachers — Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context (2026) — On-policy self-distillation via a soft-prompt teacher, routing each example in a merged multi-task corpus to its own per-task prompt-teacher.
• H²SD: Hybrid Hindsight Self-Distillation (2026) — Correctness-aware on-policy self-distillation that reweights updates on correct rollouts but applies reference-hint reverse-KL correction where the student's reasoning fails.
• Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning (2026) — Downweights credit for tokens whose likelihoods shift most under opposite correctness assertions, recasting privileged self-teacher shifts as sensitivity rather than supervision.
• Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation (2026) — Traces self-distillation's failure on hard tasks to privileged-information bias pulling per-token targets toward one reference solution rather than correctness.
• On-Policy Self-Distillation without Any Supervision (2026) — Uses a majority-vote consensus rollout as privileged teacher context for distillation onto disagreeing rollouts, removing OPSD's last dependence on external supervision.
• RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer (2026) — Routes multilingual token supervision by how much an English reference shifts the teacher view, targeting reasoning pivots rather than surface realization. (Code)
• Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning (2026) — Replaces privileged solution traces with problem-structure guidance, preserving reasoning cues while reducing shortcut imitation.
• Rubrics as Privileged Information for Open-Ended Generation (2026) — Conditions a self-teacher on task rubrics, transferring open-ended evaluation criteria through student-generated responses.

Context and Experience Internalization

• OPCD: On-Policy Context Distillation for Language Models (2026) — Context-conditioned teacher on student rollouts; distills system prompts and experiential knowledge. (Code)
• OEL: Online Experiential Learning for Language Models