#Kv-cache
Showing 19 of 19 repositories tagged #kv-cache, ranked by stars
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
A Golang implemented Redis Server and Cluster. Go 语言实现的 Redis 服务器和分布式集群
Unified KV Cache Compression Methods for Auto-Regressive Models
LLM KV cache compression made easy
LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation (NeurIPS 2025)
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.
KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.
[ICLR'26] The official code implementation for "Cache-to-Cache: Direct Semantic Communication Between Large Language Models"
LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.
Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.
Distributed KV cache pool + control plane for LLM serving
Completion After Prompt Probability. Make your LLM make a choice
This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT) variant. The implementation focuses on the model architecture and the inference process. The code is restructured and heavily commented to facilitate easy understanding of the key parts of the architecture.
First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression with near-zero quality loss.
ConDB: The KV-Cache Native Context Database
A curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.