Applying Evolutionary Computing to Embeddings of Diffusion Models
Evolutionary-Diffusion
Combining Evolutionary Computing with Diffusion ModelsImages
- 🎨 Aesthetics Maximization/Minimization using LAION Aesthetics Predictor V2
- 📊 Multi-Objective Optimization with CLIP-IQA metrics
- 🛡️ Evading AI-image detection by optimizing against a fine-tuned SDXL detector, SSP, or DIRE
- 🧭 Navigating the CLIP-Score Landscape for Prompt-Matching
Audio
- 🔊 Optimize Audiobox Aesthetics from Meta
Try it out in Google Colab
| Notebook | Link | |--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Genetic Algorithm | 
| | Island Genetic Algorithm | | | NSGA | |
Image results will be saved in your Google Drive in the folder evolutionary. Each generation creates a new folder where the images will be saved in. You can change the folders in the notebook.
Sometimes Google Collab causes dependency problems which break the notebook. If you have any issues executing this in a Collab environment, please do not hesitate to create a new issue.
Running locally
Using a virtual environment is recommended. Clone the repo (or download the zip), then install dependencies:python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e ".[all]"The dependency set is validated on Python 3.12 and 3.13. Now you are ready to go with the notebooks or custom code. CUDA and MPS are supported.
Example - Creating the most Aesthetic Image
Optimizing for Aesthetics using the Aesthetics Predictor V2 from LAION with a GA and SDXL-Turbo
Optimizing the aesthetics predictor as a maximization problem, the algorithm came to a max Aesthetics score of 8.67. This score is higher than the examples from the real LAION English Subset dataset have, with the red line showing the limit. A wide variety of prompts (inspired by parti prompts) was used for the initial population.https://github.com/malthee/evolutionary-diffusion-results/assets/18032233/4841d671-639a-4ac4-b7a8-ee5a66fab28d
Parameters:
population_size = 100 num_generations = 200 batch_size = 1 elitism = 1
creator = SDXLPromptEmbeddingImageCreator(pipelinefactory=setuppipeline, batchsize=batchsize, inference_steps=3) evaluator = AestheticsImageEvaluator() crossover = PooledArithmeticCrossover(0.5, 0.5) mutationarguments = UniformGaussianMutatorArguments(mutationrate=0.1, mutationstrength=2, clamprange=(-900, 900)) mutationargumentspooled = UniformGaussianMutatorArguments(mutationrate=0.1, mutationstrength=0.3, clamp_range=(-8, 8)) mutator = PooledUniformGaussianMutator(mutationarguments, mutationarguments_pooled) selector = TournamentSelector(tournament_size=3)
Example - Island GA with Artists on each Island
Performing an Island GA by creating random embeddings and mixing them with artist embeddings to get mixtures of styles and new ideas.| | | | | | --- | --- | --- | --- | | 
| 
| 
| 
| | 
| 
| 
| 
Example - Improving Audiobox Aesthetics Score
Starting from noisy random samples, evolving to better sounds. Using the sum of all fitness criteria Audiobox Aesthetics offers.https://github.com/user-attachments/assets/25be32eb-188b-483a-a787-9cea4d7f5eeb
https://github.com/user-attachments/assets/17f4bef7-1f5a-47a4-8f29-e4f42519a4fe
Detailed Results and Notebooks
More detailed results can be found in a separate repository dedicated to the results of the experiments: https://github.com/malthee/evolutionary-diffusion-resultsEvaluators
- AestheticsImageEvaluator: Uses the LAION Aesthetics Predictor V2. Blog: https://laion.ai/blog/laion-aesthetics/
- AestheticsPredictorV25ImageEvaluator: Uses the Aesthetic Predictor V2.5 from discus0434
- CLIPScoreEvaluator: Using the torchmetrics implementation for CLIP-Score
- (Single/Multi)CLIPIQAEvaluator: Using the torchmetrics implementation for CLIP Image Quality Assessment.
- AIDetectionImageEvaluator: Using the original Version from HuggingFace, or the fine-tuned one for SDXL generated images
- SSPAIDetectionImageEvaluator: Implements SSP from the paper with references from the official repo.
- DIREAIDetectionImageEvaluator: Implements DIRE from the paper with an official ADM/DDIM backend and an explicit SDXL-Turbo experimental comparison mode.
- AudioboxAestheticsEvaluator: Using Audiobox Aesthetics from Meta
Image Creators
Current supported creators working in the prompt embedding space:- SDXLPromptEmbeddingImageCreator: Supports the SDXL pipeline, creates both prompt- and pooled-prompt-embeddings.
- SDPromptEmbeddingImageCreator: Only has prompt-embeddings, is faster but produces less quality results than SDXL.
Audio Creators
Supporting only AudioLDM because it works simply on the CLAP embedding space (suitable for this kind of operation). Other embeddings have shown to not work well with evolutionary operations (like the T5 encoder for example)- AudioLDMSoundCreator: Works with any AudioLDMPipeline, default is audioldm-l-full
Package Structure and Base Classes
(Pre-Testing) Evaluating Models for Evolutionary use
There are multiple notebooks exploring the speed and quality of models for generation and fitness-evaluation. These notebooks also allow for simple inference so that any model can be tried out easily.- diffusionmodelcomparison: tries out different diffusion models with varying arguments (inference steps, batch size) to find out the optimal model for image generation in an evolutionary context (generation speed & quality)
- clip_evaluators: uses torch metrics with CLIPScore and CLIP IQA. CLIPScore could define the fitness for "prompt fulfillment" or "image alignment" while CLIP IQA has many possible metrics like "quality, brightness, happiness..."
- aidetectionevaluator: evaluates AI-detection fitness for evasion using the SDXL detector, SSP, and DIRE.
- aesthetics_evaluator: uses a pre-trained model from the maintainers of the LAION image dataset, which scores an image 0-10 depending on how "aesthetic" it is. Could be used as a maximization criteria for the fitness of images.
- clamprange: testing the usual prompt-embedding min and max values for different models, so that a CLAMP range can be set in the mutator for example. Using the parti prompts.
- crossovermutationexperiments: testing different crossover and mutation strategies to see how they work in the prompt embedding space
- embedding_relations: experimenting with TensorBoard and integrating it into our embedding model
Secured auto-download of models and TLS certificates
Sadly many of these models are hosted somewhere without a proper versioning and release system. Thus we added some hardcoded hashes of the models we use in the code, so that if the auto-download is used, it will be verified against the expected hash. You can enable/disable this verification in the Evaluator usage code.Furthermore, the Model/checkpoint auto-download uses verified TLS. If you hit CERTIFICATEVERIFYFAILED on macOS Python.org builds, run:
/Applications/Python\ 3.13/Install\ Certificates.commandAlternative: set SSLCERTFILE to a valid CA bundle path (for example from certifi). An insecure fallback exists only via explicit opt-in for temporary troubleshooting:
export EVOLUTIONARYDIFFUSIONALLOWINSECURESSL=1Tests (newly added, to be extended)
pytest tests -q
