#Distributed-inference

flashinfer-ai

flashinfer

FlashInfer: Kernel Library for LLM Serving

gpustack

gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

llm-d

llm-d

Achieve state of the art inference performance with modern accelerators on Kubernetes

awslabs

awsome-distributed-ai

Collection of best practices, reference architectures, model training examples and utilities to train large models on AWS.

ADT109119

llamacpp-distributed-inference

一個基於 llama.cpp 的分佈式 LLM 推理程式，讓您能夠利用區域網路內的多台電腦協同進行大型語言模型的分佈式推理，使用 Electron 的製作跨平台桌面應用程式操作 UI。

sutro-sh

sutro

Analyze and generate unstructured data using LLMs, from quick experiments to billion token jobs.