#Llm-inference
Showing 60 of 174 repositories tagged #llm-inference, ranked by stars
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Find secrets with Gitleaks ๐
ๆฌ้กน็ฎๆจๅจๅไบซๅคงๆจกๅ็ธๅ ณๆๆฏๅ็ไปฅๅๅฎๆ็ป้ช๏ผๅคงๆจกๅๅทฅ็จๅใๅคงๆจกๅๅบ็จ่ฝๅฐ๏ผ
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
Official inference library for Mistral models
OpenVINOโข is an open source toolkit for optimizing and deploying AI inference
High-speed Large Language Model Serving for Local Deployment
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
A Datacenter Scale Distributed Inference Serving Framework
Plano is an AI-native proxy and data plane for agentic apps โ with built-in orchestration, safety, observability, and smart LLM routing so you stay focused on your agents core logic.
Open-source implementation of AlphaEvolve
FlashInfer: Kernel Library for LLM Serving
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
โก Pure-Rust WebGPU inference engine โ OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
Low-latency AI engine for mobile devices & wearables
๐A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.๐
Superduper: End-to-end framework for building custom AI applications and agents.
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
RuVector is a High Performance, Real-Time, Self-Learning Ai, Vector GNN, Memory DB built in Rust.
Optimizing inference proxy for LLMs
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
Sparsity-aware deep learning inference runtime for CPUs
A portable accelerated SQL query, search, and LLM-inference engine, written in Rust, for data-grounded AI apps and agents.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
An open coding agent for your terminal, built by a community collective rather than a company. Bring your own model, keep your code on your machine, and owe nothing to anyone.
โก Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platformsโก
AICI: Prompts as (Wasm) Programs
The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect components (LLMs, Tools, vector DBs, memory) to agents that can interact with your data.
Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.
Ultrafast serverless GPU inference, sandboxes, and background jobs
Declarative way to run AI models in React Native on device, powered by ExecuTorch.
Run local LLMs like llama, deepseek-distill, kokoro and more inside your browser
List of software that allows searching the web with the assistance of AI: https://hf.co/spaces/felladrin/awesome-ai-web-search
A curated collection of top-tier penetration testing tools and productivity utilities across multiple domains. Join us to explore, contribute, and enhance your hacking toolkit!
LLMs as Copilots for Theorem Proving in Lean
Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
Local AI app and inference engine for agents. Run open-weight LLMs locally โ private, 100% offline on your computer.
List of awesome hosting sorted by minimal plan price
Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)
A highly optimized LLM inference acceleration engine for Llama and its variants.
LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.
โAI-Compassโๅฐไธบ็คพๅบๆๅผๅจ AI ๆๆฏๆตทๆดไธญ่ช่ก็ๆนๅ๏ผๆ ่ฎบไฝ ๆฏๅๅญฆ่ ่ฟๆฏ่ฟ้ถๅผๅ่ ๏ผ้ฝ่ฝๅจ่ฟ้ๆพๅฐ้ๅพ AI ๅๅคงๆนๅ็่ทฏๅพใๆจๅจๅธฎๅฉๅผๅ่ ็ณป็ปๆงๅฐไบ่งฃ AI ็ๆ ธๅฟๆฆๅฟตใไธปๆตๆๆฏใๅๆฒฟ่ถๅฟ๏ผๅนถ้่ฟๅฎ่ทตๆๆกไป็่ฎบๅฐ่ฝๅฐ็ๅ จ่ฟ็จใ
Llama 3+ inference in pure Java
Open-source VMs-as-a-service
LeaderWorkerSet: An API for deploying a group of pods as a unit of replication
LLMFlows - Simple, Explicit and Transparent LLM Apps
GPU environment and cluster management with LLM support
Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O
Pure Rust Inference Engine
CLI for running large numbers of coding agents in parallel with git worktrees
LLM (Large Language Model) FineTuning