#Multimodal
Showing 60 of 262 repositories tagged #multimodal, ranked by stars
Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience
[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
โ๏ธ Build multimodal AI applications with cloud-native stack
YC (S26) | AI that knows what you've seen, said, or heard. Records everything you do, say, hear 24/7, local, private, secure. Connect to OpenClaw, Hermes agent and 100+ apps
Janus-Series: Unified Multimodal Understanding and Generation Models
Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.6, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).
Visualize, query, and stream to train on multimodal robotics data.
Production ready toolkit to run AI locally
A Python library for anomaly detection across tabular, time series, graph, text, image, and audio data. 60+ detectors, benchmark-backed ADEngine orchestration, and an agentic workflow for AI agents.
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
๐ๅคงๆจกๅๅบ็จๅผๅๅฎๆไธ๏ผRAG ๆๆฏๅ จๆ ๆๅ๏ผๅจ็บฟ้ ่ฏปๅฐๅ๏ผhttps://datawhalechina.github.io/all-in-rag/
Deeplake is AI Data Runtime for Agents. It provides serverless postgres with a multimodal datalake, enabling scalable retrieval and training.
Mobile-Agent: The Powerful GUI Agent Family
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple Silicon.
This repository is a curated collection of links to various courses and resources about Artificial Intelligence (AI)
notes for software engineers getting up to speed on new AI developments. Serves as datastore for https://latent.space writing, and product brainstorming, but has cleaned up canonical references under the /Resources folder.
The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/
Open-source framework for building AI-powered apps in JavaScript, Go, and Python, built and used in production by Google
Solve Visual Understanding with Reinforced VLMs
A visual playground for agentic workflows: Iterate over your agents 10x faster
A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines
A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
A framework for efficient model inference with omni-modality models
A Next-Generation Training Engine Built for Ultra-Large MoE Models
Plug in and Play Implementation of Tree of Thoughts: Deliberate Problem Solving with Large Language Models that Elevates Model Reasoning by atleast 70%
Curated tutorials and resources for Large Language Models, AI Painting, and more.
Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.
OpenMMLab Pre-training Toolbox and Benchmark
MOSSโTTS Family is an openโsource speech and sound generation model family from MOSI.AI and the OpenMOSS team. It is designed for highโfidelity, highโexpressiveness, and complex realโworld scenarios, covering stable longโform speech, multiโspeaker dialogue, voice/character design, environmental sound effects, and realโtime streaming TTS.
From Chain-of-Thought prompting to OpenAI o1 and DeepSeek-R1 ๐
Code and models for ICML 2024 paper, NExT-GPT: Any-to-Any Multimodal Large Language Model
The most accurate document search and store for building AI apps
SimpleMem: Efficient Lifelong Memory for LLM Agents โ Text & Multimodal
InternGPT (iGPT) is an open source demo platform where you can easily showcase your AI models. Now it supports DragGAN, ChatGPT, ImageBind, multimodal chat like GPT-4, SAM, interactive image editing, etc. Try it at igpt.opengvlab.com (ๆฏๆDragGANใChatGPTใImageBindใSAM็ๅจ็บฟDemo็ณป็ป)
Foundation Architecture for (M)LLMs
Represent, send, store and search multimodal data
An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions
๐ค Type-safe, provider-agnostic TypeScript AI SDK for streaming chat, tool calling, agents, and multimodal apps across OpenAI, Anthropic, Gemini, React, Vue, Svelte, and Solid.
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
Easily compute clip embeddings and build a clip retrieval system with them
Images to inference with no labeling (use foundation models to train supervised models).
[EMNLP-2024] Build multimodal language agents for fast prototype and production
Comprehensive resources on Generative AI, including a detailed roadmap, projects, use cases, interview preparation, and coding preparation.
mPLUG-Owl: The Powerful Multi-modal Large Language Model Family
HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance
GenAI Processors is a lightweight Python library that enables efficient, parallel content processing.
๐จ NeMo Data Designer: Generate high-quality synthetic data from scratch or from seed data.
Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch
A collection of original, innovative ideas and algorithms towards Advanced Literate Machinery. This project is maintained by the OCR Team in the Language Technology Lab, Tongyi Lab, Alibaba Group.
The Self-Coding System for Your App โ Alan AI SDK for Android
Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ.
The Self-Coding System for Your App โ Alan AI SDK for Flutter
The Self-Coding System for Your App โ Alan AI SDK for Ionic
Meta-Transformer for Unified Multimodal Learning
Unified multimodal backend for AI data apps