#Serving
Showing 28 of 28 repositories tagged #serving, ranked by stars
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
A flexible, high-performance serving system for machine learning models
A Cloud Native Batch System (Project under CNCF)
An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
In this repository, I will share some useful notes and references about deploying deep learning-based models in production.
Serve, optimize and scale PyTorch models in production
A course of learning LLM inference serving on Apple Silicon for systems engineers: build a tiny vLLM + Qwen.
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
Database system for AI-powered apps
TensorFlow template application for deep learning
A comprehensive guide to building RAG-based LLM applications for production.
DELTA is a deep learning based natural language and speech processing platform. LF AI & DATA Projects: https://lfaidata.foundation/projects/delta/
A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）
A scalable inference server for models optimized with OpenVINO™
Generic and easy-to-use serving service for machine learning models
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
A high-performance inference system for large language models, designed for production environments.
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
ML pipeline orchestration and model deployments on Kubernetes.
A universal scalable machine learning model deployment solution
Blockchain Search with GraphQL APIs
ClearML - Model-Serving Orchestration and Repository Solution
A collection of model deployment library and technique.
Production inference for encoder models - ColBERT, GLiNER, ColPali, embeddings etc. - as vLLM plugins for online and in-process deployment
[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI
LoRA fine-tuned Stable Diffusion Deployment
AsyncIO serving for data science models