#Inference-engine
Showing 39 of 39 repositories tagged #inference-engine, ranked by stars
FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
校招、秋招、春招、实习好项目！带你从零实现一个高性能的深度学习推理库，支持大模型 llama2 、Unet、Yolov5、Resnet等模型的推理。Implement a high-performance deep learning inference library step by step
Rule engine implementation in Golang
Large-scale LLM inference engine
FeatherCNN is a high performance inference engine for convolutional neural networks.
Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.
A highly optimized LLM inference acceleration engine for Llama and its variants.
MLX Studio - Home of JANG_Q - Image Gen/Edit + Chat/Code All in one - + OpenClaw (Anthropic API)
Python Computer Vision & Video Analytics Framework With Batteries Included
Adlik: Toolkit for Accelerating Deep Learning Inference
Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O
A library for high performance deep learning inference on NVIDIA GPUs.
校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。
A Machine Learning System for Data Enrichment.
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
PyKnow: Expert Systems for Python
Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.
The Qualcomm® AI Hub apps are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
docs for search system and ai infra
AMD MIVisionX is a computer vision toolkit built around a highly optimized, conformant open-source implementation of the Khronos OpenVX™ 1.3 specification. As of the 4.0.0 release, MIVisionX ships three components: the AMD OpenVX™ engine, the AMD RPP OpenVX extension, and the RunVX graph executor — across CPU, HIP, and OpenCL backends.
A C# inference engine for running large language models (LLMs) locally using GGUF model files. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/Linux with full GPU capability
The inference engine the open-source world built for itself.
TinyTensor is a tool for running already trained NN (Neural Network) models to be able to use them for inference of various tasks such as image classification, semantic segmentation, etc.
Neural network inference template for real-time cricital audio environments - presented at ADC23
DaisyKit is an easy AI toolkit with face mask detection, pose detection, background matting, barcode detection, face recognition and more. - with NCNN, OpenCV, Python wrappers
An optimized neural network operator library for chips base on Xuantie CPU.
A Machine Learning System for Data Enrichment.
This is a repository for a No-Code object detection inference API using the OpenVINO. It's supported on both Windows and Linux Operating systems.
A small Recursive Language Model: let any LLM run code on its context instead of stuffing it into the prompt.
GPU tensor framework with support for running ONNX models
MINERVA - Minimal Inference Engine for Robust, Verifiable, and Authenticated ML. Encrypted, integrity-verified neural network inference for MCUs down to ATmega328P.
[⛔️ DEPRECATED] Friendli: the fastest serving engine for generative AI
Wingman is the fastest and easiest way to run Llama models on your PC or Mac.
"Very simple but works well" Computer Vision based ID verification solution provided by LibraX.
258 KB WASM runtime for Needle a 26M-parameter tool-calling transformer. Runs in browser, Cloudflare Workers, and Node.js. No backend required.
A local-Llm-first coding agent.