#Model-serving
Showing 49 of 49 repositories tagged #model-serving, ranked by stars
A high-throughput and memory-efficient inference and serving engine for LLMs
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
A framework for efficient model inference with omni-modality models
Olares: An Open-Source Personal Cloud to Reclaim Your Data
In this repository, I will share some useful notes and references about deploying deep learning-based models in production.
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
Community maintained hardware plugin for vLLM on Ascend
🏕️ Reproducible development environment for humans and agents
AICI: Prompts as (Wasm) Programs
OpenLake is a high performance storage engine for efficient LLM inference and GPU Training
MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.
An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.
Hopsworks - Data-Intensive AI platform with a Feature Store
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
The simplest way to serve AI/ML models in production
A throughput-oriented high-performance serving framework for LLMs
A highly optimized LLM inference acceleration engine for Llama and its variants.
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
A scalable inference server for models optimized with OpenVINO™
Model Deployment at Scale on Kubernetes 🦄️
Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.
FastAPI Skeleton App to serve machine learning models production-ready.
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).
The missing bridge between your ML models and your AI agents.
BentoDiffusion: A collection of diffusion models served with BentoML
The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.
A multi-functional library for full-stack Deep Learning. Simplifies Model Building, API development, and Model Deployment.
ClearML - Model-Serving Orchestration and Repository Solution
BentoML Example Projects 🎨
MONAI Deploy App SDK offers a framework and associated tools to design, develop and verify AI-driven applications in the healthcare imaging domain.
Route inference across providers.
The official python package for NimbleBox. Exposes all APIs as CLIs and contains modules to make ML 🌸
PyTorch/XLA integration with JetStream (https://github.com/google/JetStream) for LLM inference"
🚀 Stream inferences of real-time ML models in production to any data lake (Experimental)
A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
A collection of model deployment library and technique.
CLIP as a service - Embed image and sentences, object recognition, visual reasoning, image classification and reverse image search
Large-scale Auto-Distributed Training/Inference Unified Framework | Memory-Compute-Control Decoupled Architecture | Multi-language SDK & Heterogeneous Hardware Support
Serve the home! Inference stack for your Nvidia DGX Spark aka the Grace Blackwell AI supercomputer on your desk. Mostly vLLM based for now and single-spark. For the not-so-rich buddies. If you want latest/in-testing, look at the branches
KPilot: Unified control plane for multi-cluster Kubernetes management, GPU compute scheduling, and model serving.
Kubernetes scanner that discovers LLMs running on vLLM and extracts their deployment and runtime facts.