#Llm-serving
Showing 42 of 42 repositories tagged #llm-serving, ranked by stars
A high-throughput and memory-efficient inference and serving engine for LLMs
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
Run, manage, and scale AI workloads on any AI infrastructure. Use one system to access & manage all AI compute (Kubernetes, Slurm, 20+ clouds, on-prem).
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
Superduper: End-to-end framework for building custom AI applications and agents.
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
Community maintained hardware plugin for vLLM on Ascend
MoBA: Mixture of Block Attention for Long-Context LLMs
AICI: Prompts as (Wasm) Programs
Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
RayLLM - LLMs on Ray (Archived). Read README for more info.
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
A throughput-oriented high-performance serving framework for LLMs
A highly optimized LLM inference acceleration engine for Llama and its variants.
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
♾️ Private Agent Fleet with Spec Coding. Each agent gets their own GPU-accelerated desktop. Run Claude, Codex, Gemini and open models on a full private AI Stack ♾️
LLM (Large Language Model) FineTuning
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
Efficient AI Inference & Serving
This is suite of the hands-on training materials that shows how to scale CV, NLP, time-series forecasting workloads with Ray.
The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.
Tensor Fusion is a state-of-the-art GPU virtualization and pooling solution designed to optimize GPU cluster utilization to its fullest potential.
🪶 Lightweight OpenAI drop-in replacement for Kubernetes
SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.
Enterprise-grade LLM automated deployment tool that makes AI servers truly "plug-and-play".
A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
A High-Efficiency System of Large Language Model Based Search Agents
A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes
It is a comprehensive resource hub compiling all LLM papers accepted at the International Conference on Learning Representations (ICLR) in 2024.
[EMNLP'25 Industry] Repo for "Z1: Efficient Test-time Scaling with Code"
npm like package ecosystem for Prompts 🤖
[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI
LLM Security Platform.
The open-source solution that helps SaaS owners track and optimize their AI costs.