#Multi-modality
Showing 25 of 25 repositories tagged #multi-modality, ranked by stars
[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP
Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network). Technique was originally created by https://twitter.com/advadnoun
🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions
Algorithms and Publications on 3D Object Tracking
Parsing-free RAG supported by VLMs
Chatbot Arena meets multi-modality! Multi-Modality Arena allows you to benchmark vision-language models side-by-side while providing images as inputs. Supports MiniGPT-4, LLaMA-Adapter V2, LLaVA, BLIP-2, and many more!
The open source implementation of Gemini, the model that will "eclipse ChatGPT" by Google
An open-source implementation for training LLaVA-NeXT.
[CVPR 2025 Highlight] Official code for "Olympus: A Universal Task Router for Computer Vision Tasks"
Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.
[ESSD 2025 & IEEE DFC 2025 & CVPRW 2026] Bright: A globally distributed multimodal VHR dataset for all-weather disaster response
Official code for NeurIPS2023 paper CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection and TPAMI 2025 paper CoDAv2
An all-new Language Model That Processes Ultra-Long Sequences of 100,000+ Ultra-Fast
Seed, Code, Harvest: Grow Your Own App with Tree of Thoughts!
[CVPR 2025] DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
[ICCV 2021] Official implementation of the paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering"
[ICLR 2025] Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations, ICCV 2021
Implementation of Qformer from BLIP2 in Zeta Lego blocks.
[CVPR 2024 Highlight] ImageNet-D
A new novel multi-modality (Vision) RAG architecture
The World's First AI-Enabled Multi-Modality Native Search Engine