Interpretable Generative Models through Post-hoc Concept Bottlenecks (CVPR 2025)

Paper | Project Page

This is the official repository for the CVPR 2025 paper: Interpretable Generative Models through Post-hoc Concept Bottlenecks

• We propose two novel methods to enable interpretability for generative models:

* CB-AE: The 1st method for post-hoc interpretable generative models. CB-AE can be trained efficiently with a frozen pretrained generative model, without real concept-labeled images. * Concept Controller: An optimization-based concept intervention method with improved steerability and higher image quality.

• We show our methods have higher steerability (+31% and +28% better than prior SOTA) and lower cost (4-15x faster to train) on deep generative models including GANs and diffusion models.

Table of Contents

• Setup

* Environment setup instructions * Download base model and CB-AE/CC weights * Download concept classifier weights

• Demo
• Training
• Evaluation
• Results
• Sources
• Cite this work

Setup

Environment setup instructions

• Conda environment installation:

conda create -n posthocgencbm python=3.8
    conda install nvidia/label/cuda-11.7.0::cuda-nvcc cudatoolkit
    pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
    pip install -r requirements.txt

• Download the CelebA-HQ-pretrained StyleGAN2 base model from below and test the environment using python3 eval/test_stygan2.py. It should save a StyleGAN2 generated image in images/. If you get CUDA runtime errors (during "Setting up PyTorch plugin..."), use this:

export CUDAHOME=$CONDAPREFIX
    export CPLUSINCLUDEPATH=$CUDAHOME/include:$CPLUSINCLUDE_PATH
    export LIBRARYPATH=$CUDAHOME/lib:$LIBRARY_PATH

Download base model and CB-AE/CC weights

• We use models/checkpoints for saving/loading CB-AE/CC checkpoints

mkdir models/checkpoints
    cd models/checkpoints

• CelebA-HQ-pretrained StyleGAN2 (from [[2]](#sources)):

## base model weights
    wget https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan2/versions/1/files/stylegan2-celebahq-256x256.pkl
    ## CB-AE weights
    gdown https://drive.google.com/uc?id=1RBdjcBDbpAoW5qOkG-rBonIpcBApBF-q
    ## CC weights
    gdown https://drive.google.com/uc?id=1fh2XV2ttrCc88-SgfR9f-JcwG1eent_U

• CUB-pretrained StyleGAN2 (trained using [[4]](#sources)):

## base model weights
    gdown https://drive.google.com/uc?id=1sW7WgvUFH2REZPQx88BjFneoItP9C0XB

• CelebA-HQ-pretrained DDPM-256x256 (from [[3]](#sources)):

## base model weights get downloaded automatically via HuggingFace when using "-e cbae_ddpm" (i.e. any config using this DDPM model)
    ## CB-AE weights
    gdown https://drive.google.com/uc?id=1kl5pDqzm0M73r8H74AfSokgDFGAF0szb

Download concept classifier weights

• ResNet18 CelebA-HQ-based classifiers for training and visualization.
• ViT-L-16 CelebA-HQ-based classifiers for quantitative evaluation.

cd models/checkpoints
    ## ResNet18 CelebA-HQ
    gdown https://drive.google.com/uc?id=1xbR7MbERV7wMnU4WcsNSDriYXBqsy_jZ
    unzip celebahqrn18conclsf.zip
    ## ViT-L-16 CelebA-HQ (relatively large file of ~8.4 GB, so download only if you want to do evaluations)
    gdown https://drive.google.com/uc?id=1XD6Badmf4QwRrdy6MbOr-mefyu1k_OIy
    unzip celebahqvitl16conclsf.zip
    ## ResNet18 CelebA (64x64)
    gdown https://drive.google.com/uc?id=15m6xCI5JPZaz-BaSoCjHCJCeof53G4rC
    unzip celeba64rn18conclsf.zip
    ## ResNet50 CUB (256x256)
    gdown https://drive.google.com/uc?id=1vW5Q41FGHXdTqbraz54AXQ2uoBKispLD
    unzip cubrn50conclsf.zip
    ## ResNet50 CUB (64x64)
    gdown https://drive.google.com/uc?id=1vvlWd4MWB62-lyq2sPQAVhf5Pqc5Mnzf
    unzip cub64rn50conclsf.zip

• Other concept classifiers can be trained using train/train_conclsf.py.

Demo

• Follow notebooks/visualize_interventions.ipynb for concept interventions demo with CelebA-HQ-pretrained StyleGAN2 with CB-AE.

* Note: Before the notebook, please download concept classifier weights (ResNet18) for visualization.

Training

• Use bash scripts/train_cbae.sh to train a CB-AE for a CelebA-HQ-pretrained StyleGAN2 with supervised classifiers as pseudo-label source.
• Some important arguments to specify are:

* -e: specify which config file from the config/ folder to use (e.g. cbaestygan2 or cbaeddpm). * -d: specify dataset of base generative model (e.g. celebahq). * -t: specify experiment name to be used as a suffix for saving logs, checkpoints, etc. * -p: specify pseudo-label source $M$ for CB-AE/CC training (e.g. supervised for supervised-trained classifiers, clipzs for zero-shot CLIP classifiers, tipzs for few-shot adapted CLIP).

• The same train_cbae.sh has commented out examples for training Concept Controller (CC) with StyleGAN2 and for CB-AE with DDPM.

* For DDPM training, first run python3 eval/generateddpmtrainset.py to save generated images from DDPM (faster than generating at training time). Also update the path to the real dataset in --base-root as that is used for evaluation (which is not possible in StyleGAN or other GANs).

Evaluation

• Use bash scripts/eval_intervention.sh for an example that runs steerability evaluation for Smiling concept for a CelebA-HQ StyleGAN2 CB-AE.
• Some important arguments to specify are:

* -e, -d, and -t should be the same as from training (or use based on downloaded CB-AE/CC checkpoint, e.g. celebahqcbaestygan2thr90supplcls8cbae.pt would use -d celebahq -e cbaestygan2thr90 -t suppl_cls8). * -c: concept to intervene on (e.g. Smiling or MouthSlightlyOpen). * -v: desired concept value (e.g. 0 or 1 based on if desired target concept is Smiling or Not Smiling). * --optint: use this for optimization-based interventions (not using this will use CB-AE interventions). * --visualize: use this to visualize some examples (not using this will run the full quantitative evaluation).

• The same eval_intervention.sh has commented out examples for evaluating StyleGAN2 CC and DDPM CB-AE.

Results

1. Concept Steerability or Intervention Success Rate

• Our CB-AE and CC improves steerability across GANs (+31%) and diffusion models (+28%) over the prior state-of-the-art method CBGM [[1]](#sources) while being 4-15x faster to train on average.

2. Interpreting generated images

• Our CB-AE (and CC) provide human-understandable concept predictions along with the generated images.

3. Concept intervention examples (CB-AE interventions)

• Concept intervention (modifying concepts) in the CB-AE leads to appropriate changes in the resulting image generation, enabling controllable generation.

4. Concept intervention examples (optimization-based interventions)

Optimization-based interventions also enable controllable generation with improved orthogonality (i.e.* less change in other concepts, closer to the original generation) than CB-AE interventions.

Sources

[1] CBGM (ICLR 2024): https://github.com/prescient-design/CBGM

[2] StyleGAN3 GitHub repo (it has StyleGAN2 pretrained weights for CelebA-HQ and CUB)

[3] CelebA-HQ pretrained DDPM repo

[4] StyleGAN2-Ada PyTorch GitHub repo

Cite this work

A. Kulkarni, G. Yan, C. Sun, T. Oikarinen, and T.-W. Weng, Interpretable Generative Models through Post-hoc Concept Bottlenecks, CVPR 2025

@inproceedings{kulkarni2025interpretable,
    title={Interpretable Generative Models through Post-hoc Concept Bottlenecks},
    author={Kulkarni, Akshay and Yan, Ge and Sun, Chung-En and Oikarinen, Tuomas and Weng, Tsui-Wei},
    booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2025},
}