#Inference-server
Showing 17 of 17 repositories tagged #inference-server, ranked by stars
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
Turn any computer or edge device into a command center for your computer vision projects.
Open-source inference server and production cluster for all the models your agent needs.
The simplest way to serve AI/ML models in production
An open-source computer vision framework to build and deploy apps in minutes
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
A REST API for Caffe using Docker and Go
Fast GPU OCR server. 270 img/s on FUNSD. TensorRT FP16, PP-OCRv5, HTTP + gRPC.
PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.
This is a repository for an nocode object detection inference API using the Yolov3 and Yolov4 Darknet framework.
Auto-tuned launcher for GGUF models on llama.cpp / ik_llama.cpp — OpenAI-compatible server with multi-GPU tensor-split, MoE expert placement, measured flag tuning (AI Tune), hardware-matched HuggingFace downloads, and crash recovery. An Ollama alternative for multi-GPU rigs.
This is a repository for an nocode object detection inference API using the Yolov4 and Yolov3 Opencv.
ONNX Runtime Server: The ONNX Runtime Server is a server that provides TCP and HTTP/HTTPS REST APIs for ONNX inference.
[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI
Wingman is the fastest and easiest way to run Llama models on your PC or Mac.