$M\text{-}Attack$: A Simple Baseline Achieving Over 90% Success Rate Against GPT-4.5/4o/o1

This repository is the official implementation of A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1.

Illustration of our proposed framework. Our method is based on two components: Local-to-Global or Local-to-Local Matching (LM) and Model Ensemble (ENS). LM is the core of our approach, which helps to refine the local semantics of the perturbation. ENS helps to avoid overly relying on single models embedding similarity, thus improving attack transferability.

Requirements

Dependencies: To install requirements:

pip install -r requirements.txt
wandb login

or run the follwoing code to install up-to-date libraries

conda create -n mattack python=3.10
conda activate mattack
pip install hydra-core
pip install salesforce-lavis
pip install -U transformers
pip install gdown
pip install wandb
pip install pytorch-lightning
pip install opencv-python
pip install --upgrade opencv-contrib-python
pip install -q -U google-genai
pip install anthropic
pip install scipy
pip install nltk
pip install timm==1.0.13
pip install openai
python -m spacy download encoreweb_sm
pip install git+https://github.com/openai/CLIP.git
wandb login

Note: you might need to register a Weight & Bias account, then fill wandb.entity in config/ensemble3models.yaml

Images: We have already included the dataset used in our paper, located in resources/images

• resources/images/bigscale/nips17 for clean images
• resources/images/target_images/1 for target images
• resources/images/target_images/1/keywords.json for labeled semantic keywords

API Keys: You need to register API keys for the following APIs for evaluation:

• OpenAI
• Google
• Anthropic

# API Keys for different models
DO NOT commit this file to git!
gpt4v: "youropenaiapi_key" claude: "youranthropicapi_key" gemini: "yourgoogleapi_key"  gpt4o: "youropenaiapi_key"

Note: DO NOT LEAK YOUR API KEYS!

Quick Start

python generateadversarialsamples.py
python blackboxtextgeneration.py -m blackbox.model_name=gpt4o,claude,gemini
python gptevaluate.py -m blackbox.modelname=gpt4o,claude,gemini
python keywordmatchinggpt.py -m blackbox.model_name=gpt4o,claude,gemini

Then you can find corresponding results in wandb. Below is our detailed instructions for each step. We also provide our generated adversarial samples in Hugging Face.

Note: to reproduce the results on Claude 3.5 Sonnet, set model.usetargetcrop=false, model.cropscale=[0.5, 1.0], optim.alpha=0.75 in config/ensemble3models.yaml.

1. Generate Adversarial Samples

python generateadversarialsamples.py

The config is managed by Hydra. To change the config, either directly changing config/ensemble3models.yaml or use commanline override. For example, to scale up to 1000 image, change data.cledatapath and data.tgtdatapath in the config, either directly changing config/ensemble_3models.yaml or use commanline override:

python generateadversarialsamples.py data.cledatapath=resources/images/bigscale1000 data.tgtdatapath=resources/images/targetimages_1000

It is the same if you want to change $\alpha$ or $\epsilon$:

python generateadversarialsamples.py optim.alpha=0.5 optim.epsilon=16

2. Evaluation

The evaluation is seperated into two parts:

• generate descriptions for clean and adversarial images on target blackbox commercial model
• evaluate KMRScore or GPTScore-based ASR

python blackboxtextgeneration.py -m blackbox.model_name=gpt4o,claude,gemini {CONFIG IN STEP 1}

The line -m blackbox.modelname=gpt4o,claude,gemini is used to start Hydra Multi-Run to automatically run multiple setting for generating descriptions with different blackbox commercial models.

python gptevaluate.py -m blackbox.modelname=gpt4o,claude,gemini {CONFIG IN STEP 1}

python keywordmatchinggpt.py -m blackbox.model_name=gpt4o,claude,gemini {CONFIG IN STEP 1}

python evaluation_metrics.py {CONFIG IN STEP 1}

@article{li2025mattack,
  title={A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1},
  author={Zhaoyi Li and Xiaohan Zhao and Dong-Dong Wu and Jiacheng Cui and Zhiqiang Shen},
  journal={arXiv preprint arXiv:2503.10635},
  year={2025},
}