#Gguf
Showing 48 of 48 repositories tagged #gguf, ranked by stars
Hundreds of models & providers. One command to find what runs on your hardware.
Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
Maid is a free and open source application for interfacing with llama.cpp models locally, and with Anthropic, DeepSeek, Ollama, Mistral and OpenAI models remotely.
动手学Ollama，CPU玩转大模型部署，在线阅读地址：https://datawhalechina.github.io/handy-ollama/
Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level
Go manage your Ollama models
A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.
An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.
Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer.
A CLI to estimate inference memory requirements for Hugging Face models, written in Python.
LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.
Llama 3+ inference in pure Java
Local First Ai Agent. Optimized for Local Ai models. Long context window. Proper tools callings. Runs privately on your device.
The most advanced, fully offline client-side AI suite on Android today.
Go with your own intelligence - Go applications that directly integrate llama.cpp for local inference using hardware acceleration.
Run AI ✨ assistant locally! with simple API for Node.js 🚀
A desktop app for running Large Language Models locally.
LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.
PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.
llama.cpp (GGUF LLMs) and llava.cpp (GGUF VLMs) for ROS 2
Auto-tuned launcher for GGUF models on llama.cpp / ik_llama.cpp — OpenAI-compatible server with multi-GPU tensor-split, MoE expert placement, measured flag tuning (AI Tune), hardware-matched HuggingFace downloads, and crash recovery. An Ollama alternative for multi-GPU rigs.
JANG — GGUF for MLX. YOU MUST USE JANG_Q RUNTIME. Adaptive Mixed-Precision Quantization + Runtime for Apple Silicon
Complete guide to running large language models locally on AMD Strix Halo / Ryzen AI MAX+ 395 with Radeon 8060S (gfx1151) and 96GB/128GB unified memory. Covers BIOS config, Ubuntu/kernel setup, Ollama, llama.cpp Vulkan/RADV, ROCm/HIP, vLLM, and 70B/120B GGUF evidence.
A C# inference engine for running large language models (LLMs) locally using GGUF model files. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/Linux with full GPU capability
Run any local LLM engine, auto-tuned to your GPU — polished web UI + OpenAI/Anthropic-compatible API. Point Claude Code at your own machine in one command. No Electron, no Python, offline-first.
Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU, autoscaling, air-gapped, production-ready.
A native LLM client for iOS & Apple Watch. Run local GGUF models offline via llama.cpp, or connect to OpenAI/Claude/Gemini. Features local RAG, Model Context Protocol (MCP) tools, Siri Shortcuts, and cross-device sync. Built with Swift.
lcpp is a dart implementation of llama.cpp used by the mobile artificial intelligence distribution (maid)
🎨 Multiplatform AI image generation app powered by Stable Diffusion • Built with Kotlin Multiplatform & Compose • Supports SDXL, FLUX, SD3 & more • Native performance via C++/JNI • Android/iOS & Desktop ready
Camelid: a Rust-native local inference backend with evidence-gated model compatibility.
InferrLM - On-device AI for iOS & Android
Gradio based tool to run opensource LLM models directly from Huggingface
Own your AI, search the web with it🌐😎
Ruby FFI bindings for llama.cpp to run open-source LLMs such as GPT-OSS, Qwen 3.5, Gemma 4, and Llama 3 locally with Ruby.
一個基於 llama.cpp 的分佈式 LLM 推理程式，讓您能夠利用區域網路內的多台電腦協同進行大型語言模型的分佈式推理，使用 Electron 的製作跨平台桌面應用程式操作 UI。
A fast terminal native app (TUI) and CLI with init wizard for launching local LLMs with zero overhead
AI inference, packed simply. A blazing-fast, zero-dependency WebGPU runtime to run GGUF models directly in the browser. Features a symmetric API for seamless local execution and cloud provider routing. Built with Rust & C++.
Pre-built wheels for llama-cpp-python across platforms and CUDA versions
Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.
Your universal AI text processor, powered by local and cloud LLMs. Edit, refactor, and transform text in any application on Windows, macOS, and Linux.
The Natural Language Shell integrates OpenAI's GPTs, Anthropic's Claude, or local GGUF-formatted LLMs directly into the terminal experience, allowing operators to describe their tasks in either POSIX commands or fluent human language
Android native AI inference library, bringing text, image, video, STT, TTS inference
Core of Artificial Intelligence
MineGPT 是一个基于Kotlin Multiplatform 开发的本地小型语言模型(SLM)对话应用; MineGPT is a lightweight local SLM (Small Language Model) chat application built with Kotlin Multiplatform. It aims to provide a cross-platform and user-friendly AI assistant experience.
Build an AI communication analyzer from scratch to understand how AI products actually work. Learn prompt engineering, reasoning pipelines, and local LLM integration using Node.js - no frameworks, no abstractions, just fundamentals
A local AI search engine: it runs and manages local AI models, searches your files and code, and crawls the web, all in one program. Cited answers, local-first, with an MCP server for your coding agent. TUI, CLI, REST API, and Python library. Works with Ollama and LM Studio.
Semantic code search powered by embeddings. CLI, web dashboard, and AI-agent tooling — search code by meaning, not text. Self-hosted, Go server with embedded llama.cpp.