Topics

#inference-efficiency #lvlms #mllms #token-compression #visual-token-pruning

Language breakdown

Python 52.0%

Jupyter Notebook 45.1%

Shell 2.0%

JavaScript 0.5%

HTML 0.4%

CSS 0.1%

▸ Files click to expand

README

GlimpsePrune

English | 简体中文

A Dynamic Visual Token Pruning Framework for Large Vision-Language Models

GlimpsePrune dynamically prunes a large number of irrelevant visual tokens before answering questions, reducing the model's inference overhead.

GlimpsePrune is a dynamic visual token pruning framework designed for Large Vision-Language Models (LVLMs). Through fast training on a small amount of data (e.g., less than 1 hour on 20K GQA data), GlimpsePrune enables Qwen2.5-VL-7B to prune an average of 92.6% of visual tokens before generating a response, while maintaining performance comparable to the original model.

For more technical details, please refer to our paper.

If you find our work inspiring or helpful, please give us a star ⭐. Thank you for your attention and support:

✨ Key Features

High Pruning Rate: Prunes over 90% of visual tokens on average with almost no performance loss, effectively reducing computational and memory overhead.
Robust Performance: Stable performance when processing high-resolution images and handling complex free-form VQA tasks.
Lightweight Training: Only a few extra parameters (Glimpse token and VIP) need to be trained, completed in less than 1 hour on a single A100 GPU.
Broad Compatibility: Supports single and multi-image inputs, is compatible with KV-Cache and Flash Attention 2, and provides a fair comparison benchmark with other mainstream visual compression methods.

🚀 News

2026.06.08: Our revised paper has been accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT).
2025.08.05: Paper are publicly released!
2025.08.03: Code and Models are publicly released!

🖼️ Framework Overview

The core idea of GlimpsePrune is to introduce a glimpse token and a lightweight Visual tokens Important Predictor (VIP) that can quickly identify and retain the visual regions most relevant to the text prompt, pruning the remaining redundant information.

The core code implementation is located in:

Qwen2.5-VL: transformersgp/models/qwen25vl/model_gp.py
LLaVA-1.5: llavagp/model/languagemodel/llava_llama.py

📊 Performance Results

We evaluated GlimpsePrune on multiple VQA benchmarks. The results show that it achieves a high pruning rate while maintaining performance on par with the original model, outperforming other visual compression methods.

Free-form VQA Benchmarks

Short-form VQA Benchmarks

Throughput under batched inference

Across batch sizes (bs=1→12), GlimpsePrune prunes visual tokens, so under the same peak GPU memory it sustains substantially higher prefilling/decoding throughput than the Qwen2.5-VL baseline and scales to larger batches.

✅ Roadmap

[x] Support for Qwen2.5-VL
[x] Support for single-image input
[x] Support for multi-image input
[x] Video input support (Qwen2.5-VL)
[x] InternVL2.5 support
[x] Provide a local Gradio Demo
[x] Support for LLaVA-1.5
[x] Provide evaluation scripts for various visual token compression methods (PyramidDrop, VisionZip, etc.) on the free-form VQA
[x] Support for batch input (Batch Inference)

🛠️ Installation

Clone the repository

git clone https://github.com/HVision-NKU/GlimpsePrune.git
    cd GlimpsePrune

Create an environment and install dependencies

We recommend create seperated virtual environment for different models: For Qwen2.5-VL: - python=3.10 - torch==2.7.0 - flash-attn==2.7.4.post1 - pip install -r qwen_requirements.txt - pip install qwen-vl-utils[decord] For LLaVA-1.5 (Optional):

Click to expand LLaVA dependency installation

- python=3.10 - torch==2.1.2 - flash-attn=2.7.3 - pip install -r llava_requirements.txt

Additional dependencies for Evaluation and Demo (Optional):

# Evaluation     pip install lmms-eval==0.3.5 vllm==0.9.0.1     # Demo     pip install gradio==5.39.0

📦 Models and Data

Model Download

All models can be automatically downloaded from the Hugging Face Hub. If you encounter network issues, you can download them manually to a local directory. <new_module> are the weights of the extra glimpse token and VIP modules we trained.

Data Preparation

Training and Free-form VQA evaluation use the Visual-CoT dataset.

# Download the dataset (approx. 128GB)
huggingface-cli download --repo-type dataset --local-dir datas deepcs233/Visual-CoT cotimagestar_split
Extract
cd datas/cotimagestar_split
cat cotimages* | tar -xvf - -C ../cot
cd ../.. # Return to the project root directory

After extraction, the datas directory structure should be as follows:

GlimpsePrune/
├── datas/
│   └── cot/
│       ├── cub/
│       ├── gqa/
│       └── ...
└── ...

▶️ How to Use

Local Demo

We provide a Gradio Demo to intuitively experience the effects of GlimpsePrune.

python demo_gp.py \
    --base_model Qwen/Qwen2.5-VL-7B-Instruct \
    --newmodulesdir ashun989/GlimpsePrune_Qwen2.5-VL-7B-Instruct

Inference

For a detailed example of how to load the model and perform inference, please refer to the Jupyter Notebook: ➡️ notebook/gpqwen_tutorial.ipynb

Evaluation

We provide convenient evaluation scripts.

Free-form VQA

# Default settings (no retention rate limit)
BASEMODEL=<basemodel> bash scripts/inferqwengpcot.sh <newmodules_dir>
Set a maximum retention rate (e.g., 11.1%)
BASEMODEL=<basemodel> MAXREMAINRATIO=0.111 bash scripts/inferqwengpcot.sh <newmodules_dir>

Short-form VQA

# Default settings
BASEMODEL=<basemodel> bash scripts/evalqwengp.sh <newmodulesdir>
Set a maximum retention rate
BASEMODEL=<basemodel> MAXREMAINRATIO=0.111 bash scripts/evalqwengp.sh <newmodulesdir>

Efficiency

# Download V* bench
hf download https://huggingface.co/datasets/craigwu/vstarbench --repo-type dataset --local-dir datas/vstarbench
Test GlimpsePrune under 4096 visual tokens with 11.1% retention ratio. 
TASKS="vstar" BATCHSIZEPERDEVICE=4 WARMUPITERS=3 TIMELOGGER=1 MEMORYLOGGER=1 FIXEDREMAINRATIO=0.111 MAXPIXELS=3211264 BASEMODEL=$basemodel bash scripts/inferqwengpcot.sh $newmodulesdir

Training

Train GlimpsePrune

Training on Qwen2.5-VL-3B-Instruct requires at least two 24GB GPUs (e.g., RTX 3090) and takes about 1 hour.

# Train Qwen2.5-VL
CUDAVISIBLEDEVICES=0,1 \
bash scripts/trainqwengp.sh
Train LLaVA-1.5
CUDAVISIBLEDEVICES=0,1 \
bash scripts/trainllavagp.sh

Train GlimpsePrune+ (Optional)

Training on Qwen2.5-VL-7B-Instruct requires four 80GB A100 GPUs, plus an additional 48GB of VRAM to run the reward model, and takes about 24 hours.

# 1. Deploy the reward model
bash scripts/vllm_serve.sh
2. Test the API
python test_api.py
3. Start training
CUDAVISIBLEDEVICES=0,1,2,3 \
bash scripts/trainqwengp_plus.sh

🙏 Acknowledgements

This project is based on the following excellent open-source work, and we express our sincere gratitude:

Qwen2.5-VL / LLaVA: Powerful Large Vision-Language Models.
Visual-CoT: A VQA dataset with rich domains, diverse object sizes, and bounding box annotations.
PyramidDrop, VisionZip, DivPrune, CDPruner, VScan: Other exploratory works in the field of visual token compression.
lmmseval: A evaluation toolkit of large multimodal models.

🖊️ Citation

If you find our work helpful, please consider citing our paper:

@ARTICLE{11557376,   author={Zeng, Quan-Sheng and Li, Yunheng and Wang, Qilong and Jiang, Peng-Tao and Wu, Zuxuan and Cheng, Ming-Ming and Hou, Qibin},   journal={IEEE Transactions on Circuits and Systems for Video Technology},    title={A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models},    year={2026},   volume={},   number={},   pages={1-1},   keywords={Modeling;Visualization;Large language models;Videos;Decoding;Memory;Training;Computers;Conferences;Computer vision},   doi={10.1109/TCSVT.2026.3702147} }