#Vllm
Showing 60 of 95 repositories tagged #vllm, ranked by stars
Industrial-grade speech recognition toolkit: 170x realtime, 50+ languages, speaker diarization, emotion detection, streaming, and OpenAI-compatible API.
Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model family and using them on various provider services
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop β all through one unified, production-ready inference API.
A Datacenter Scale Distributed Inference Serving Framework
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines
πA curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.π
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Structured data extraction, instruction calling and agentic workflows with ML, LLM and Vision LLM
Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc
System Level Intelligent Router for Mixture-of-Models at Cloud, Data Center and Edge
A course of learning LLM inference serving on Apple Silicon for systems engineers: build a tiny vLLM + Qwen.
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
Cascading runtime for AI agents. Optimize cost, latency, quality, and policy decisions inside the agent loop.
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
Community maintained hardware plugin for vLLM on Ascend
A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.
OpenAI and Anthropic compatible server for Apple Silicon. Run LLMs and vision-language models (Llama, Qwen-VL, LLaVA) with continuous batching, MCP tool calling, and multimodal support. Native MLX backend, 400+ tok/s. Works with Claude Code.
Open Source Continuous Inference Benchmark Research Platform β Kimi K2.7-Code, MiniMax M3, DeepSeekv4, GLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72 & soonβ’ TPUv6e/v7/Trainium2/3 | εΌζΊζη»ζ¨ηεΊεη η©ΆεΉ³ε° β Kimi K2.7-CodeγMiniMax M3γDeepSeekv4γGLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72οΌε³ε°ζ¨εΊβ’ TPUv6e/v7/Trainium2/3
AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.
π Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.
Evaluate your LLM's response with Prometheus and GPT4 π―
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.
Make Discord your LLM frontend - Supports any OpenAI compatible API (OpenRouter, Ollama and more)
End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS.
A Python library powered by Language Models (LLMs) for conversational data discovery and analysis.
An local, offline (after initial setup), portable OCR software that can process images and PDF files, using DeepSeek-OCR-2 AI (running directly on your machine).
[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.
Accurate, large-scale, and extensible simulator for LLM inference Systems
EfficientSAM3 compresses SAM3 into lightweight, edge-friendly models via progressive knowledge distillation for fast promptable concept segmentation and tracking.
Crater is a cloud-native AI training & inference platform.
Pure Rust + CUDA LLM inference engine β no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
A command-line interface tool for serving LLM using vLLM.
Open Model Engine (OME) β Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
The Runpod worker template for serving our large language model endpoints. Powered by vLLM.
π¨ Ready-to-use DeepSeek-OCR Web UI | Modern Interface | 7 Recognition Modes | Batch Processing | Real-time Logging | Fully Responsive
KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.
Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).
TopicGPT: A Prompt-Based Framework for Topic Modeling [NAACL'24]
Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/aeon-7/aeon-vllm-ultimate:latest container, tuned for long-context draft acceptance on DGX Spark. 6 HF variants (BF16/NVFP4/MTP/MTP-XS), docker-compose, and QuickStart.
Python package for LLM compression
Low latency JSON generation using LLMs β‘οΈ
Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.
LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, supporting hybrid inference for MOE large models.
Setup and run a local LLM and Chatbot using consumer grade hardware.
βΈοΈ Easy, advanced inference platform for large language models on Kubernetes. π Star to support our work!
AI-powered offensive security testing using autonomous agents, directly in your terminal.
The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.
Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime.
An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance.
High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.
Fully-featured, beautiful web interface for vLLM - built with NextJS.
Booster - open accelerator for LLM models. Better inference and debugging for AI hackers
Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU, autoscaling, air-gapped, production-ready.
Automated system for LLM evaluation via agents. Doc as below: