#Quantization
Showing 60 of 101 repositories tagged #quantization, ranked by stars
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
Faster Whisper transcription with CTranslate2
中文LLaMA&Alpaca大语言模型+本地CPU/GPU训练部署 (Chinese LLaMA & Alpaca LLMs)
[🔥updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. 📃 online docs: https://ufund-me.github.io/Qbot ✨ :news: qbot-mini: https://github.com/Charmve/iQuant
A vector index built on TurboQuant, written in Rust with Python bindings
Accessible large language models via k-bit quantization for PyTorch.
An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
Fast inference engine for Transformer models
[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.
Sparsity-aware deep learning inference runtime for CPUs
A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks
Build, personalize and control your own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
A list of papers, docs, codes about model quantization. This repo is aimed to provide the info for model quantization research, we are continuously improving the project. Welcome to PR the works (papers, repositories) that are missed by the repo.
Mastering Applied AI, One Concept at a Time
Run Mixtral-8x7B models in Colab or consumer desktops
Your Cheat Sheet for AI Engineering Interview – Questions and Answers.
A Python package for extending the official PyTorch that can easily obtain performance on Intel platform
PPL Quantization Tool (PPQ) is a powerful offline neural network quantization tool.
Run a 1-billion parameter LLM on a $10 board with 256MB RAM
INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
Brevitas: neural network quantization in PyTorch
A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.
Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
Build computer vision models in a fraction of the time and with less data.
Neural Network Compression Framework for enhanced OpenVINO™ inference
FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
Dataflow compiler for QNN inference on FPGAs
[ICML2025] SpargeAttention: A training-free sparse attention that accelerates any model inference.
A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.
TinyChatEngine: On-Device LLM Inference Library
[NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning; [NeurIPS 2022] MCUNetV3: On-Device Training Under 256KB Memory
Official implementation of Half-Quadratic Quantization (HQQ)
Palette quantization library that powers pngquant and other PNG optimizers
[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.
Embedded and mobile deep learning research resources
[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.
[ICML 2024] SqueezeLLM: Dense-and-Sparse Quantization
The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment
A Pytorch Knowledge Distillation library for benchmarking and extending works in the domains of Knowledge Distillation, Pruning, and Quantization.
⚡ boost inference speed of T5 models by 5x & reduce the model size by 3x.
QKeras: a quantization deep learning library for Tensorflow Keras
Offline semantic Text-to-Image and Image-to-Image search on Android powered by quantized state-of-the-art vision-language pretrained CLIP model and ONNX Runtime inference engine
Port of MiniGPT4 in C++ (4bit, 5bit, 6bit, 8bit, 16bit CPU inference with GGML)
An Open-Source Package for Deep Learning to Hash (DeepHash)
Awesome machine learning model compression research papers, quantization, tools, and learning material.
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
RDNA-native LLM inference engine in Rust.
Infrastructures™ for Machine Learning Training/Inference in Production.
Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.
KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.
[NeurIPS 2024] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
[ICML 2024] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
大模型/LLM推理和部署理论与实践
Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/aeon-7/aeon-vllm-ultimate:latest container, tuned for long-context draft acceptance on DGX Spark. 6 HF variants (BF16/NVFP4/MTP/MTP-XS), docker-compose, and QuickStart.
Python package for LLM compression
LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.
The Official Repo for "Quick Start Guide to Large Language Models"