Awesome-Knowledge-Distillation-of-LLMs

This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.

Last updated Aug 8, 2026

1.3k

Stars

Forks

Issues

Stars/day

Attention Score

View on GitHub ↗ Open in github.dev

Topics

#alignment #compression #data-augmentation #data-synthesis #feedback #instruction-following #kd #knowledge-distillation #large-language-model #llm #multi-modal #self-distillation #self-training #supervised-finetuning #survey

Language breakdown

No language data available.

▸ Files click to expand

README

Awesome Knowledge Distillation of LLM Papers

[

]()

A Survey on Knowledge Distillation of Large Language Models

Xiaohan Xu¹ Ming Li² Chongyang Tao³ Tao Shen⁴ Reynold Cheng¹ Jinyang Li¹ Can Xu⁵ Dacheng Tao⁶ Tianyi Zhou²

¹ The University of Hong Kong ² University of Maryland ³ Microsoft ⁴ University of Technology Sydney ⁵ Peking University ⁶ The University of Sydney

*A collection of papers related to knowledge distillation of large language models (LLMs). If you want to use LLMs for benefitting your own smaller models training, or use self-generated knowledge to achieve the self-improvement, just take a look at this collection.*

We will update this collection every week. Welcome to star ⭐️ this repo to keep track of the updates.

❗️Legal Consideration: It's crucial to note the legal implications of utilizing LLM outputs, such as those from ChatGPT (Restrictions), Llama (License), etc. We strongly advise users to adhere to the terms of use specified by the model providers, such as the restrictions on developing competitive products, and so on.

💡 News

2024-2-20: 📃 We released a survey paper "A Survey on Knowledge Distillation of Large Language Models". Welcome to read and cite it. We are looking forward to your feedback and suggestions.

Update Log

- 2024-3-19: Add 14 papers.

Contributing to This Collection

Feel free to open an issue/PR or e-mail shawnxxh@gmail.com, minglii@umd.edu, hishentao@gmail.com and chongyangtao@gmail.com if you find any missing taxonomies or papers. We will keep updating this collection and survey.

📝 Introduction

KD of LLMs: This survey delves into knowledge distillation (KD) techniques in Large Language Models (LLMs), highlighting KD's crucial role in transferring advanced capabilities from proprietary LLMs like GPT-4 to open-source counterparts such as LLaMA and Mistral. We also explore how KD enables the compression and self-improvement of open-source LLMs by using them as teachers.

KD and Data Augmentation: Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts.

Taxonomy: Our analysis is meticulously structured around three foundational pillars: algorithm, skill, and verticalization -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields.

KD Algorithms: For KD algorithms, we categorize it into two principal steps: "Knowledge Elicitation" focusing on eliciting knowledge from teacher LLMs, and "Distillation Algorithms" centered on injecting this knowledge into student models.

Figure: An illustration of different knowledge elicitation methods from teacher LLMs.

Skill Distillation: We delve into the enhancement of specific cognitive abilities, such as context following, alignment, agent, NLP task specialization, and multi-modality.

Verticalization Distillation: We explore the practical implications of KD across diverse fields, including law, medical & healthcare, finance, science, and miscellaneous domains.

Note that both Skill Distillation) and Verticalization Distillation employ Knowledge Elicitation and Distillation Algorithms in KD Algorithms to achieve their KD. Thus, there are overlaps between them. However, this could also provide different perspectives for the papers.

Why KD of LLMs?

In the era of LLMs, KD of LLMs plays the following crucial roles:

| Role | Description | Trend | |:--------|:--------:|:--------:| | ① Advancing SLMs | Transferring advanced capabilities from proprietary LLMs to smaller SLMs, such as open source LLMs or other smaller models. | Most common | | ② Compression | Compressing LLMs to make them more efficient and practical. | More popular with the prosperity of open-source LLMs | | ③ Self-Improvement | Refining open-source LLMs' performance by leveraging their own knowledge, i.e. self-knowledge. | New trend to make open-source LLMs more competitive |

📒 Table of Contents

KD Algorithms

- Knowledge Elicitation - Labeling - Expansion - Curation - Feature - Feedback - Self-Knowledge - Distillation Algorithms - Supervised Fine-Tuning - Divergence and Similarity - Reinforcement Learning - Rank Optimization

Skill Distillation

- Context Following - Instruction Following - Multi-turn Dialogue - RAG Capability - Alignment - Thinking Pattern - Preference - Value - Agent - Tool Using - Planning - NLP Task Specialization - NLU - NLG - Information Retrieval - Recommendation - Text Generation Evaluation - Code - Multi-Modality - Summary Table

Verticalization Distillation

- Law - Medical & Healthcare - Finance - Science - Misc.

Encoder-based KD

Citation

KD Algorithms

Knowledge Elicitation

Distillation Algorithms

Supervised Fine-Tuning

Due to the large number of works applying supervised fine-tuning, we only list the most representative ones here.

Divergence and Similarity

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning | EMNLP Findings| 2024-02 | Github | Data | Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models | arXiv | 2024-04 | | Weight-Inherited Distillation for Task-Agnostic BERT Compression | NAACL | 2024-03 | Github | | BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation | arXiv | 2024-02 | Github | | DISTILLM: Towards Streamlined Distillation for Large Language Models | arXiv | 2024-02 | Github | | Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs | arXiv | 2024-02 | Github | Data| | Revisiting Knowledge Distillation for Autoregressive Language Models | arXiv | 2024-02 | | Knowledge Distillation for Closed-Source Language Models | arXiv | 2024-01 | | Knowledge Fusion of Large Language Models | ICLR | 2024-01 | Github | Improving In-context Learning via Bidirectional Alignment | arXiv | 2023-12 | Towards the Fundamental Limits of Knowledge Transfer over Finite Domains | NeurIPS | 2023-10 | | Baby Llama: Knowledge Distillation from an Ensemble of Teachers Trained on a Small Dataset with No Performance Penalty | CoNLL | 2023-08 | Github | Data| | f-Divergence Minimization for Sequence-Level Knowledge Distillation | ACL | 2023-07 | Github | Data| | f-Divergence Minimization for Sequence-Level Knowledge Distillation | ACL | 2023-07 | Github | Data| | MiniLLM: Knowledge Distillation of Large Language Models | ICLR | 2023-06 | Github | Data | | On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes | ICLR | 2023-06 | | LLM-QAT: Data-Free Quantization Aware Training for Large Language Models | arXiv | 2023-05 | Github| Data| | Less is more: Task-aware layer-wise distillation for language model compression | PMLR | 2022-10 | Github | DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter | NeurIPS | 2019-10

Reinforcement Learning

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | Direct Language Model Alignment from Online AI Feedback | arXiv | 2024-02 | | Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint | arXiv | 2024-01 | Github | Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Models | CoRL | 2023-11 | | Motif: Intrinsic Motivation from Artificial Intelligence Feedback | ICLR | 2023-10 | Github | | Ultrafeedback: Boosting language models with high-quality feedback | arXiv | 2023-10 | Github | Data| | Eureka: Human-Level Reward Design via Coding Large Language Models | arXiv | 2023-10 | Github | Rlaif: Scaling Reinforcement Learning from Human Feedback with AI Feedback | arXiv | 2023-09 | | Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct | arXiv | 2023-08 | Github| | On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes | ICLR | 2023-06 | | Aligning Large Language Models through Synthetic Feedback | EMNLP | 2023-05 | Github| Data| | Language Model Self-improvement by Reinforcement Learning Contemplation | arXiv | 2023-05 | Consitutional AI: Harmlessness from AI Feedback | arXiv | 2022-12 |

Rank Optimization

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering | arXiv | 2024-03 | | KnowTuning: Knowledge-aware Fine-tuning for Large Language Models | arXiv | 2024-02 | Github | | Self-Rewarding Language Models | arXiv | 2024-01 | Github| | Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models | arXiv | 2024-01 | Github | Data| | Zephyr: Direct Distillation of Language Model Alignment | arXiv | 2023-10 | Github | Data| | CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment | arXiv | 2023-10

Skill Distillation

Context Following

Instruction Following

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models | arXiv | 2024-02 | | Revisiting Knowledge Distillation for Autoregressive Language Models | arXiv | 2024-02 | | Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning | arXiv | 2024-02 | Github | Data| | Phi-2: The surprising power of small language models | - | 2023-12 | | What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning | ICLR | 2023-12 | Github | Data| | MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Following | arXiv | 2023-12 | Github | Data| | Instruction Fusion: Advancing Prompt Evolution through Hybridization | arXiv | 2023-12 | | Orca 2: Teaching Small Language Models How to Reason | arXiv | 2023-11 | | Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning | NIPS Workshop | 2023-10 | Github | Data| | Textbooks Are All You Need II: Phi-1.5 Technical Report | arXiv | 2023-09 | | Orca: Progressive Learning from Complex Explanation Traces of GPT-4 | arXiv | 2023-06 | | Textbooks Are All You Need: A Large-Scale Instructional Text Data Set for Language Models | arXiv | 2023-06 | | SelFee: Iterative Self-Revising LLM Empowered by Self-Feedback Generation | arXiv | 2023-05 | ExpertPrompting: Instructing Large Language Models to be Distinguished Experts | arXiv | 2023-05 | Github | Data| | LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | arXiv | 2023-04 | Github | Data| | Wizardlm: Empowering large language models to follow complex instructions | ICLR | 2023-04 | Github| Data
Data| | Koala: A Dialogue Model for Academic Research | - | 2023-04 | Github| Data| | Alpaca: Aligning Language Model with Human Preferences | - | 2023-03 | Github| Data| | Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\%* ChatGPT Quality | - | 2023-03 | Github| Data| | Self-instruct: Aligning language model with self generated instructions | ACL | 2022-12 | Github| Data |

Multi-turn Dialogue

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | Zephyr: Direct Distillation of LM Alignment | arXiv | 2023-10 | Github | Data| | OPENCHAT: ADVANCING OPEN-SOURCE LANGUAGE MODELS WITH MIXED-QUALITY DATA | ICLR | 2023-09 | Github | Data| | Enhancing Chat Language Models by Scaling High-quality Instructional Conversations | arXiv | 2023-05 | Github | Data| | Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data | EMNLP | 2023-04 | Github| Data| | Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\%* ChatGPT Quality | - | 2023-03 | Github| Data|

RAG Capability

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection | NIPS | 2023-10 | Github | Data| | SAIL: Search-Augmented Instruction Learning | arXiv | 2023-05 | Github | Data| | Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks | NIPS | 2023-05 | Github | Data|

Alignment

Thinking Pattern

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | Aligning Large and Small Language Models via Chain-of-Thought Reasoning | EACL | 2024-03 | Github | | Divide-or-Conquer? Which Part Should You Distill Your LLM? | arXiv | 2024-02 | | Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning | arXiv | 2024-02 | Github | Data| | Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statements | arXiv | 2024-02 | Github | Data| | Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering | arXiv | 2023-11 | Github | | Orca 2: Teaching Small Language Models How to Reason | arXiv | 2023-11 | | Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning | NIPS Workshop | 2023-10 | Github | Data| | Orca: Progressive Learning from Complex Explanation Traces of GPT-4 | arXiv | 2023-06 | | SelFee: Iterative Self-Revising LLM Empowered by Self-Feedback Generation | arXiv | 2023-05

Preference

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | Ultrafeedback: Boosting language models with high-quality feedback | arXiv | 2023-10 | Github | Data| | Zephyr: Direct Distillation of LM Alignment | arXiv | 2023-10 | Github | Data| | Rlaif: Scaling Reinforcement Learning from Human Feedback with AI Feedback | arXiv | 2023-09 | | OPENCHAT: ADVANCING OPEN-SOURCE LANGUAGE MODELS WITH MIXED-QUALITY DATA | ICLR | 2023-09 | Github | Data| | RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment | arXiv | 2023-07 | Github| | Aligning Large Language Models through Synthetic Feedbacks | EMNLP | 2023-05 | Github|Data| | Reward Design with Language Models | ICLR | 2023-03 | Github| | Training Language Models with Language Feedback at Scale | arXiv | 2023-03 | | Constitutional AI: Harmlessness from AI Feedback | arXiv | 2022-12 |

Value

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | Ultrafeedback: Boosting language models with high-quality feedback | arXiv | 2023-10 | Github | Data| | RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment | arXiv | 2023-07 | Github| | Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision | NeurIPS | 2023-05 | Github | Data| | Training Socially Aligned Language Models on Simulated Social Interactions | arXiv | 2023-05 | | Constitutional AI: Harmlessness from AI Feedback | arXiv | 2022-12 |

Agent

Tool Using

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | Toolformer: Language Models Can Teach Themselves to Use Tools | arXiv | 2023-02 | | Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT | arXiv | 2023-04 | Github | Data| | Gorilla: Large Language Model Connected with Massive APIs | arXiv | 2023-05 | Github | Data| | GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction | arXiv | 2023-05 | Github | Data| | ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases | arXiv | 2023-06 | Github | Data| | ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs | arXiv | 2023-07 | Github | Data| | Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum | arXiv | 2023-08 | Github | | CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets | arXiv | 2023-09 | Github | | MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning | arXiv | 2024-01 | Github | Data| | Small LLMs Are Weak Tool Learners: A Multi-LLM Agent | arXiv | 2024-01 |Github | | EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction | arXiv | 2024-01 |Github |

Planning

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | AUTOACT: Automatic Agent Learning from Scratch via Self-Planning | arXiv | 2024-01 | Github | Lumos: Learning Agents with Unified Data, Modular Design, and Open-Source LLMs | arXiv | 2023-11 | Github | Data| | TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems | arXiv | 2023-11 | | Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld | arXiv | 2023-11 | | Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Models | CoRL | 2023-11 | | Motif: Intrinsic Motivation from Artificial Intelligence Feedback | ICLR | 2023-10 | Github | | FireAct: Toward Language Agent Fine-tuning | arXiv | 2023-10 | Github | Data| | AgentTuning: Enabling Generalized Agent Abilities for LLMs | arXiv | 2023-10 | Github | | Eureka: Human-Level Reward Design via Coding Large Language Models | arXiv | 2023-10 | Github | Language Instructed Reinforcement Learning for Human-AI Coordination | PMLR | 2023-04 | | Guiding Pretraining in Reinforcement Learning with Large Language Models | PMLR | 2023-02 | | Distilling Internet-Scale Vision-Language Models into Embodied Agents | ICML | 2023-01 |

NLP Task Specialization

NLU

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | LLM vs Small Model? Large Language Model Based Text Augmentation Enhanced Personality Detection Model | arXiv | 2024-03 | | Evolving Knowledge Distillation with Large Language Models and Active Learning | arXiv | 2024-03 | | Mixed Distillation Helps Smaller Language Model Better Reasoning | arXiv | 2023-12 | | PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation | EMNLP | 2023-10 | Github | | TinyLLM: Learning a Small Student from Multiple Large Language Models | arXiv | 2024-02 | | Targeted Data Generation: Finding and Fixing Model Weaknesses | ACL | 2023-05 | Github| | Distilling ChatGPT for Explainable Automated Student Answer Assessment | arXiv | 2023-05 | Github | | ChatGPT outperforms crowd workers for text-annotation tasks | arXiv | 2023-03 | | Annollm: Making large language models to be better crowdsourced annotators | arXiv | 2023-03 | | AugGPT: Leveraging ChatGPT for Text Data Augmentation | arXiv | 2023-02 | Github| | Is GPT-3 a Good Data Annotator? | ACL | 2022-12 | Github| | SunGen: Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning | ICLR | 2022-05 | Github | ZeroGen: Efficient Zero-shot Learning via Dataset Generation | EMNLP | 2022-02 | Github| | Generating Training Data with Language Models: Towards Zero-Shot Language Understanding | NeurIPS | 2022-02 | Github | Towards Zero-Label Language Learning | arXiv | 2021-09 | | Generate, Annotate, and Learn: NLP with Synthetic Text | TACL | 2021-06

NLG

| Title | Venue | Date | Code | Data| |:--------|:--------:|:--------:|:--------:|:--------:| | Tailoring Self-Rationalizers with Multi-Reward Distillation | arXiv | 2023-11 | Github

🔗 More in this category