#Multi-modal
Showing 60 of 65 repositories tagged #multi-modal, ranked by stars
Build and run agents you can see, understand and trust.
A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone
Open-source framework for conversational voice AI agents
[CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型
ModelScope: bring the notion of Model-as-a-Service to life.
a state-of-the-art-level open visual language model | 多模态预训练模型
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.
Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch
Ecommerce Search and Discovery - marqo.ai
[EMNLP 2022] An Open Toolkit for Knowledge Graph Extraction and Construction
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.
Open-source AI assistant ecosystem with MCP integrations, multimodal workflows, IoT support, and cross-platform voice interaction.
Represent, send, store and search multimodal data
Project Page for "LISA: Reasoning Segmentation via Large Language Model"
[NeurIPS 2023] MotionGPT: Human Motion as a Foreign Language, a unified motion-language generation model using LLMs
A robust, all-in-one GPT interface for Discord. ChatGPT-style conversations, image generation, AI-moderation, custom indexes/knowledgebase, youtube summarizer, and more!
Efficient Retrieval Augmentation and Generation Framework
[pip install medmnist] 18x Standardized Datasets for 2D and 3D Biomedical Image Classification
Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI
Recent Transformer-based CV and related works.
The TypeScript library for building AI applications.
This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.
Parsing-free RAG supported by VLMs
Use late-interaction multi-modal models such as ColPali in just a few lines of code.
[CVPR 2026🔥] 🧑🎨 OmniLottie, an open-sourced multi-modal instructed vector animation generator that produces Lottie JSONs.
Multi-modal OCR pipeline optimized for ML training (text, figure, math, tables, diagrams)
Multi-model DAG-driven parallel AI film generation — parallel speedup scales with scene independence; Generate film scenes simultaneously instead of one by one; "把影视生成的执行图从拓扑序变成关键路径最优调度" ; 唯一把场景叙事依赖建模为 DAG、以 CPM 算法驱动并行调度的影视生成引擎
Democratization of RT-2 "RT-2: New model translates vision and language into action"
🌐 Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
Chat2Graph: Graph Native Agentic System.
A streaming multimodal database for Edge AI, and Edge Computing.
LightRFT: Light, Efficient, Omni-modal & Reward-model Driven Reinforcement Fine-Tuning Framework
This project is the official implementation of 'LLMGA: Multimodal Large Language Model based Generation Assistant', ECCV2024 Oral
[ICCV2023] Official Implementation of "UniTR: A Unified and Efficient Multi-Modal Transformer for Bird’s-Eye-View Representation"
Official implementation of the paper "You Only Need Adversarial Supervision for Semantic Image Synthesis" (ICLR 2021)
[NeurIPS 2024] MeshXL: Neural Coordinate Field for Generative 3D Foundation Models, a 3D fundamental model for mesh generation
Official pytorch repository for "QD-DETR : Query-Dependent Video Representation for Moment Retrieval and Highlight Detection" (CVPR 2023 Paper)
MotionGPT3: Human Motion as a Second Modality, a MoT-based framework for unified motion understanding and generation
[NeurIPS 2024] Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
A Python framework for multi-modal document understanding with Amazon Bedrock
Text-driven human motion generation surveys, datasets and models.
MOVE (Multi-Omics Variational autoEncoder) for integrating multi-omics data and identifying cross modal associations
Official implementation of I2I-Mamba, an image-to-image translation model based on selective state spaces
[InterSpeech 2023] The official PyTorch implementation of: "AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation"
[IV2024] MultiCorrupt: A benchmark for robust multi-modal 3D object detection, evaluating LiDAR-Camera fusion models in autonomous driving. Includes diverse corruption types (e.g., misalignment, miscalibration, weather) and severity levels. Assess model performance under challenging conditions.
Qualcomm Cloud AI SDK (Platform and Apps) enable high performance deep learning inference on Qualcomm Cloud AI platforms delivering high throughput and low latency across Computer Vision, Object Detection, Natural Language Processing and Generative AI models.
The Next Generation Multi-Modality Superintelligence
[ICCV 2021] Official implementation of the paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering"
GenDB, an LLM-Powered Generative Query Engine Built for the Future
VS Code extension that streamlines development workflows through AI-powered task execution, intelligent file management, and automated code generation. Built on Cline, it integrates with various LLMs to enhance productivity and code quality while simplifying complex development tasks.
Implementation of the transformer from the paper: "Real-World Humanoid Locomotion with Reinforcement Learning"
One lib to rule them all (gen ai)
Implementation of the text to video model LUMIERE from the paper: "A Space-Time Diffusion Model for Video Generation" by Google Research
Implementation of Qformer from BLIP2 in Zeta Lego blocks.
Enjoy easier conversations with LLM
Implementation of AutoRT: "AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents"
Stable Diffusion with Text-to-Image and Image-to-Text
HAI Build is Presidio’s human-centric approach to harnessing Generative AI to produce faster, more collaborative, better-quality outcomes across the software development lifecycle. This groundbreaking innovation unlocks the ideal working relationship between human and machine.