🛸 AgentArk

Distilling Multi-Agent Intelligence into a Single LLM Agent.

Yinyi Luo¹ · Yiqiao Jin² · Weichen Yu¹ · Mengqi Zhang³ · Srijan Kumar² · Xiaoxiao Li⁵ · Weijie Xu⁴ · Xin Chen⁴; Jindong Wang³

¹Carnegie Mellon University   ²Georgia Institute of Technology   ³William & Mary   ⁴Amazon   ⁵University of British Columbia

At a glance

| Metric | Value | | --- | --- | | Avg. accuracy lift over single-agent baseline | +4.8% | | Total experiments across Qwen3 / Gemma 3 / Llama 3 | 120 | | Hierarchical distillation strategies | 3  (R-SFT · DA · PAD) | | Distillation questions / reasoning trajectories | ~342K / ~2M |

Abstract

While large language model (LLM) multi-agent systems achieve superior reasoning performance through iterative debate, practical deployment is limited by their high computational cost and error propagation. This paper proposes AgentArk, a novel framework to distill multi-agent dynamics into the weights of a single model, effectively transforming explicit test-time interactions into implicit model capabilities. This equips a single agent with the intelligence of multi-agent systems while remaining computationally efficient. Specifically, we investigate three hierarchical distillation strategies across various models, tasks, scaling, and scenarios: reasoning-enhanced fine-tuning; trajectory-based augmentation; and process-aware distillation. By shifting the burden of computation from inference to training, the distilled models preserve the efficiency of one agent while exhibiting strong reasoning and self-correction performance of multiple agents. They further demonstrate enhanced robustness and generalization across diverse reasoning tasks. We hope this work can shed light on future research on efficient and robust multi-agent development.

Architecture

AgentArk distills multi-agent debate into a single model through three hierarchical strategies — Reasoning-Enhanced SFT (R-SFT), Reasoning Trajectory-based Data Augmentation (DA), and Process-Aware Distillation (PAD) — moving the cost of collective reasoning from inference time into training time.

Highlights

• Single-agent efficiency, multi-agent reasoning. A distilled student matches most of the gain of a full debate ensemble at a fraction of the inference cost.
• PRM capacity matters more than student size. A stronger process reward model lifts smaller students disproportionately; student capacity bounds the multi-agent gain.
• Reasoning quality outweighs quantity. Curated, higher-fidelity trajectories beat naive scale-up of distillation data.
• Process-aware distillation improves reasoning behavior, not just accuracy. Students internalize critique-and-revise dynamics rather than memorizing answers.
• Robust and general. Gains transfer to out-of-distribution and robustness benchmarks (e.g., TruthfulQA).
• Extends across modalities and model families. Validated on Qwen3, Gemma 3, Llama 3, and Qwen2.5-VL (multimodal).

Installation

Requirements

| | | | --- | --- | | Python | 3.10+ | | CUDA | 12.5 | | GPU memory | 40 GB+ recommended for inference |

Setup

# Clone the repository
git clone <repository-url>
cd AgentArk
Create virtual environment
conda create -n agentark python=3.12
conda activate agentark
Install dependencies
pip install -r requirements.txt

Key Dependencies

| Category | Packages | |----------|----------| | LLM Inference | transformers, vllm, flash-attn | | RL Training | deepspeed, trl, torch | | Evaluation | rougescore, bertscore, sympy | | Utilities | datasets, accelerate, peft, wandb |

Quick Start

# Run inference with LLM Debate on QMSum dataset
python inference.py \
    --methodname llmdebate \
    --testdatasetname QMSum \
    --model_name Qwen/Qwen3-8B \
    --use_vllm \
    --tensorparallelsize 2
Evaluate results
python -m eval.shortanswereval \
    --inputfile results/QMSum/Qwen/Qwen3-8B/llmdebate_infer.jsonl \
    --dataset_name QMSum

Supported Methods

Each method lives under methods/ with its own YAML config in methods/<name>/configs/.

| Method | Directory | Description | | --- | --- | --- | | AgentVerse | methods/agentverse | Collaborative role-play with critic feedback rounds | | AutoGen | methods/autogen | Conversable multi-agent framework | | CAMEL | methods/camel | Role-playing communicative agents | | ChatDev | methods/chatdev | Software-development-oriented multi-agent pipeline | | CoT | methods/cot | Single-agent chain-of-thought baseline | | DyLAN | methods/dylan | Dynamic agent network with listwise ranking | | EvoMAC | methods/evomac | Evolutionary multi-agent collaboration | | LLM Debate | methods/llm_debate | Iterative debate among peer agents | | MacNet | methods/macnet | Macro-network of communicating agents | | MAD | methods/mad | Multi-Agent Debate | | MapCoder | methods/mapcoder | Code-generation pipeline with planner/coder roles | | MAS Base | methods/mas_base | Shared base utilities for multi-agent systems | | MAV | methods/mav | Multi-Agent Verifier | | Self-Consistency | methods/self_consistency | Parallel sampling with majority vote |

Supported Datasets

| Dataset | Task type | | --- | --- | | MATH | Mathematical reasoning | | GSM8K | Grade-school math | | MetaMathQA | Augmented math | | MedMCQA | Medical multiple choice | | QASPER | Long-context scientific QA | | HotpotQA | Multi-hop QA | | QMSum | Query-based meeting summarization | | TruthfulQA | Robustness / truthfulness |

Supported Models

| Family | Models | Typical role | | --- | --- | --- | | Qwen 3 | Qwen3-32B, Qwen3-8B, Qwen3-1.7B, Qwen3-0.6B | Teacher (32B) / Students | | Gemma 3 | Gemma3-27B-it, Gemma3-7B | Teacher / Student | | Llama 3 | Llama3-8B-Instruct | Student | | Qwen2.5-VL (multimodal) | Qwen2.5-VL-32B-Instruct, Qwen2.5-VL-3B | Teacher / Student |

Usage

Inference

Run multi-agent inference on a dataset:

python inference.py \
    --method_name <method> \
    --testdatasetname <dataset> \
    --modelname <modelpathorname> \
    --use_vllm \
    --tensorparallelsize <num_gpus>

Key Arguments:

| Argument | Description | Default | |----------|-------------|---------| | --method_name | Multi-agent method to use | Required | | --testdatasetname | Dataset for evaluation | Required | | --model_name | HuggingFace model or local path | Required | | --model_temperature | Sampling temperature | 0.5 | | --modelmaxtokens | Maximum tokens per generation | 4096 | | --use_vllm | Enable vLLM for efficient batching | False | | --tensorparallelsize | Number of GPUs for tensor parallelism | 1 | | --usemodalbatch | Use Modal for cloud deployment | False |

python inference.py \
    --method_name dylan \
    --testdatasetname MATH \
    --model_name Qwen/Qwen3-32B \
    --use_vllm \
    --tensorparallelsize 4 \
    --model_temperature 0.7

# First deploy the Modal model
modal deploy modal/launch_modal.py
Then run inference
python inference.py \
    --methodname llmdebate \
    --testdatasetname QMSum \
    --usemodalbatch \
    --model_name Qwen/Qwen3-8B

Solution Labeling

Label generated solutions for correctness (required for PRM training):

python label.py \
    --input results/QMSum/Qwen/Qwen3-32B/llmdebateinfer.jsonl \
    --dataset_name QMSum \
    --model Qwen/Qwen2.5-72B-Instruct \
    --tensorparallelsize 4

This produces labeled data with the format:

{
    "query": "...",
    "gt": "ground truth answer",
    "solutions": [
        {"id": 1, "text": "solution text", "is_correct": true},
        {"id": 2, "text": "solution text", "is_correct": false}
    ],
    "labels": [true, false]
}

Process Reward Model Training

Train a PRM to score intermediate reasoning steps:

PYTHONPATH=$PYTHONPATH:$(pwd) python prm/finetune2.py \
    --modelnameor_path Qwen/Qwen3-8B \
    --traindatapath results/QMSum/labeled.jsonl \
    --outputdir outputs/prmqmsum \
    --numtrainepochs 3 \
    --perdevicetrainbatchsize 64 \
    --perdeviceevalbatchsize 16 \
    --gradientaccumulationsteps 1 \
    --learning_rate 1e-4 \
    --weight_decay 0.1 \
    --adam_beta2 0.95 \
    --warmup_ratio 0.0 \
    --logging_steps 1 \
    --save_strategy steps \
    --save_steps 500 \
    --savetotallimit 3 \
    --bf16 True \
    --gradient_checkpointing True \
    --fix_llm True \
    --enablenanmonitoring True

RL Finetuning with GRPO

Finetune the policy model using Group Relative Policy Optimization:

python -m openrlhf.cli.train_grpo \
    --pretrain Qwen/Qwen3-0.6B \
    --rewardpretrain outputs/prmqmsum \
    --savepath outputs/grpoqmsum \
    --temperature 0.5 \
    --nsamplesper_prompt 8 \
    --advantage_estimator rloo \
    --reward_baseline token \
    --reward_mode PRMVR \
    --verifiablerewardcoef 1.0 \
    --microrolloutbatch_size 4 \
    --rolloutbatchsize 64 \
    --microtrainbatch_size 2 \
    --trainbatchsize 128 \
    --actorlearningrate 5e-7 \
    --initklcoef 0.001 \
    --max_epochs 1 \
    --num_episodes 1 \
    --promptmaxlen 40960 \
    --generatemaxlen 2048 \
    --zero_stage 2 \
    --bf16 \
    --flash_attn \
    --gradient_checkpointing \
    --save_steps 20 \
    --logging_steps 1

GRPO Key Arguments:

| Argument | Description | |----------|-------------| | --pretrain | Base model to finetune | | --reward_pretrain | Trained PRM checkpoint | | --nsamplesper_prompt | Group size for RLOO baseline (keep >= 4) | | --advantage_estimator | rloo or gae | | --reward_mode | Reward computation mode (PRMVR, ORM, etc.) | | --microrolloutbatch_size | Prompts per GPU during rollout | | --microtrainbatch_size | Samples per GPU during training |

Memory Optimization Tips:

• Lower microrolloutbatchsize and microtrainbatchsize to save GPU memory
• Keep nsamplesper_prompt >= 4 for stable GRPO performance
• Total samples = rolloutbatchsize x nsamplesper_prompt

Evaluation

Short-Answer Evaluation (ROUGE, BERTScore, F1)

python -m eval.shortanswereval \
    --modelnameor_path Qwen/Qwen3-8B \
    --dataset_name QMSum \
    --split validation \
    --output_dir outputs \
    --temperature 0.7 \
    --use_vllm \
    --applychattemplate

Batch Evaluation Across Models

for MODEL in Qwen/Qwen3-0.6B Qwen/Qwen3-1.7B Qwen/Qwen3-8B; do
    for DATASET in QMSum QASPER HotpotQA; do
        python -m eval.shortanswereval \
            --modelnameor_path "$MODEL" \
            --dataset_name "$DATASET" \
            --split validation \
            --output_dir outputs \
            --use_vllm
    done
done

Math Evaluation (Exact Match)

python -m eval.math_eval \
    --inputfile results/MATH/Qwen/Qwen3-8B/mavinfer.jsonl \
    --dataset_name MATH

Configuration

Each method has YAML configuration files in methods/<method_name>/configs/.

Example: DyLAN Configuration

# methods/dylan/configs/config_main.yaml
random_seed: 0
num_agents: 4           # Number of agents in the network
num_rounds: 3           # Communication rounds
activation: "listwise"  # Agent ranking strategy
roles:
    - "Assistant"
    - "Assistant"
    - "Assistant"
    - "Assistant"

Example: AgentVerse Configuration

# methods/agentverse/configs/config_main.yaml
cnt_agents: 2               # Number of collaborative agents
max_turn: 3                 # Maximum conversation turns
maxcriticizingrounds: 3   # Critic feedback iterations

Example: Self-Consistency Configuration

# methods/selfconsistency/configs/configmain.yaml
parallel_num: 5  # Number of parallel solution paths

Example: LLM Debate Configuration

# methods/llmdebate/configs/configmain.yaml
num_agents: 3           # Number of debating agents
num_rounds: 2           # Debate rounds

Citation

If you find AgentArk useful for your research, please cite:

@article{luo2026agentark,
  title={AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent},
  author={Luo, Yinyi and Jin, Yiqiao and Yu, Weichen and Zhang, Mengqi and Kumar, Srijan and Li, Xiaoxiao and Xu, Weijie and Chen, Xin and Wang, Jindong},
  journal={arXiv preprint arXiv:2602.03955},
  year={2026}
}

Acknowledgments

AgentArk is built on top of excellent open-source work, including OpenRLHF, vLLM, TRL, and HuggingFace Transformers.

License

This project is released under the Apache License 2.0.