How many Are in This Image? A Safety Evaluation Benchmark for Vision LLMs

Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, Cihang Xie (equal contribution)

Our paper is online now: https://arxiv.org/abs/2311.16101

Installation

For different VLLMs, please refer to their specific envirnments for installation.

• LLaVA: https://github.com/haotian-liu/LLaVA
• MiniGPT4: https://github.com/Vision-CAIR/MiniGPT-4
• InstructBLIP: https://huggingface.co/Salesforce/instructblip-vicuna-7b
• LLaMA-Adapter: https://github.com/OpenGVLab/LLaMA-Adapter
• mPLUG-Owl1&2: https://github.com/X-PLUG/mPLUG-Owl
• PandaGPT: https://github.com/yxuansu/PandaGPT
• Qwen-VL-Chat: https://huggingface.co/Qwen/Qwen-VL-Chat
• CogVLM: https://github.com/THUDM/CogVLM
• InternLM-Xcomposer: https://huggingface.co/internlm/internlm-xcomposer-7b
• Fuyu: https://huggingface.co/adept/fuyu-8b

Datasets

We host our datasets here, containing both OOD and redteaming attack datasets. The full dataset should looks like this:

.
├── ./safetyevaluationbenchmark_datasets//                    
    ├── gpt4vchallengingset # Contains the challenging test data for GPT4V
        ├── attack_images
        ├── sketchy_images
        ├── oodcv_images
        ├── misleading-attack.json
        ├── sketchy-vqa-challenging.json
        └── oodcv-vqa-counterfactual.json
    ├── redteaming # Contains the test data for redteaming tasks
        ├── misleading_attack
            ├── gaussian_noise
            ├── mixattack_eps32
            ├── mixattack_eps64
            ├── sinattackeps64dog
            ├── sinattackeps64coconut
            ├── sinattackeps64spaceship
            └── annotation.json
        ├── jailbreak_vit # adversarial images for jailbreaking VLLM through ViT
        └── jailbreak_llm # adversarial suffixes for jailbreaking VLLM through LLM
    └── ood # Contains the test data for OOD scenarios
        ├── sketchy-vqa
            ├── sketchy-vqa.json
            ├── sketchy-challenging.json
        └── oodcv-vqa
            ├── oodcv-vqa.json
            └── oodcv-counterfactual.json

Out-of-Distribution Scenario

For $\texttt{OODCV-VQA}$ and its counterfactual version, please download images from OODCV, and put all images in ood/oodcv-vqa.

For $\texttt{Sketchy-VQA}$ and its challenging version, please first download images from here, put the zip file into ood/sketchy-vqa/skechydata/, then unzip it.

Redteaming Attack

For the proposed misleading attack, the full datasets and all trained adversarial examples are in redteaming/misleading_attack, including images with gaussian noise, Sin.Attack and MixAttack with two pertubation budgets $\epsilon=32/255$ (eps32) or $\epsilon=64/255$ (eps64).

For jailbreaking methods, please refer to their respective repositories for more dataset details: Jailbreak through ViT, Jailbreak through LLM.

Testing

Before you start, make sure you have modified the CACHEDIR (where you store all your model weights) and DATADIR (where you store the benchmark data) in baselines/config.json according to your local envirnment.

cd baselines
python ../modeltestingzoo.py --model_name LLaVA

Choose --modelname from ["LlamaAdapterV2", "MiniGPT4", "MiniGPT4v2", "LLaVA", "mPLUGOwl", "mPLUGOwl2", "PandaGPT", "InstructBLIP2", "Flamingo", "LLaVAv1.5", "LLaVAv1.5-13B", "LLaVAllama2-13B", "MiniGPT4llama2", "Qwen-VL-Chat", "MiniGPT413B", "InstructBLIP2-FlanT5-xl", "InstructBLIP2-FlanT5-xxl", "InstructBLIP2-13B", "CogVLM", "Fuyu", "InternLM"].

$\texttt{OODCV-VQA}$ and its Counterfactual Variant

For $\texttt{OODCV-VQA}$:

cd baselines python ../safetyevaluations/oodscenarios/evaluation.py --modelname LLaVA --evaloodcv

For the counterfactual version:

cd baselines
python ../safetyevaluations/oodscenarios/evaluation.py --modelname LLaVA --evaloodcv_cf

$\texttt{Sketchy-VQA}$ and its Challenging Variant

For $\texttt{Sketchy-VQA}$:

cd baselines python ../safetyevaluations/oodscenarios/evaluation.py --modelname LLaVA --evalsketch

For the challenging version:

cd baselines
python ../safetyevaluations/oodscenarios/evaluation.py --modelname LLaVA --evalsketch_challenging

Misleading Attack

For training the misleading adversarial images:

cd safetyevaluations/redteaming/misleadingvision_attack
python misleadingvisattack.py --lr 1e-3 --misleadingobj dog --inputfolder path/to/attack-bard/NIPS2017 --outputfolder ./misleadingadversarial_attack

Change --inputfolder to the path of adversarial examples you want to test. If you want to use the MixAttack, add --mixobj argument to the command.

For testing the VLLMs:

cd baselines
python ../safetyevaluations/redteaming/misleadingvisionattack/testmisleading.py --imagefolder redteaming/misleadingattack/mixattackeps64 --outputname misleadingattackeps64 --humanannotpath redteaming/misleading_attack/annotation.json

Jailbreaking Methods

Please refer to these two repositories for detailed attack settings: Jailbreak through ViT, Jailbreak through LLM. We give our trained adversarial images and suffixes for jailbreaking ViTs and LLMs in redteaming/jailbreakvit and redteaming/jailbreakllm in the data folder.

Usage and License Notices

The data, code and checkpoint is intended and licensed for research use only. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.

Citation

If you find our work useful to your research and applications, please consider citing the paper and staring the repo :)

@article{tu2023how,
  title={How Many Unicorns Are In This Image? A Safety Evaluation Benchmark For Vision LLMs},
  author={Tu, Haoqin and Cui, Chenhang and Wang, Zijun and Zhou, Yiyang and Zhao, Bingchen and Han, Junlin and Zhou, Wangchunshu and Yao, Huaxiu and Xie, Cihang},
  journal={arXiv preprint arXiv:2311.16101},
  year={2023}
}

Acknowledgement

This work is partially supported by a gift from Open Philanthropy. We thank Center for AI Safety and Google Cloud for supporting our computing needs. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.