Shepherd Model Gateway

Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. SMG centralizes worker lifecycle management, balances traffic across self-hosted engines and cloud providers, and gives you enterprise-grade control over multi-tenancy, chat-history storage, MCP tooling, and observability — behind one unified endpoint.

Why SMG?

| | | |:--------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------| | 🚀 Maximize GPU Utilization | Cache-aware routing tracks each worker's KV-cache state in radix trees to reuse prefixes across SGLang, vLLM, TensorRT-LLM, TokenSpeed, and MLX — with load modeling that accounts for queued token work and KV pressure. | | 🔌 One API, Any Backend | Route to self-hosted engines over HTTP or gRPC, or to OpenAI, Anthropic, Gemini, and xAI — plus any OpenAI-compatible endpoint — through a single unified gateway. | | ⚡ Built for Speed | Native Rust with streaming gRPC pipelines, cached tokenization with zero-copy cache hits, prefill/decode disaggregation (including a separate encode stage for vision), and DP-aware routing for data-parallel engines. | | 🔒 Enterprise Control | Priority admission scheduling with preemption and per-tenant controls, API-key auth with OIDC on the control plane, WebAssembly plugins for custom logic, and chat history that never leaves your infrastructure. | | 📊 Full Observability | 90+ Prometheus metrics, OpenTelemetry tracing with W3C trace context propagated into the engines over both HTTP and gRPC, and structured JSON logs with request correlation. |

API Coverage: OpenAI Chat Completions, Completions, Embeddings, Rerank, and Classify; Responses and Conversations APIs for agents; Anthropic Messages; Gemini Interactions; Realtime over WebSocket and WebRTC; audio transcription; tokenize/detokenize; and MCP tool execution with approval policies in the Responses and Messages APIs.

Quick Start

Install — pick your preferred method:

# Docker
docker pull lightseekorg/smg:latest
Kubernetes (Helm)
helm install smg oci://ghcr.io/smg-project/charts/smg
Python
pip install smg
Rust (needs protoc)
cargo install smg

Run — point SMG at your inference workers:

# Single worker
smg launch --worker-urls http://localhost:8000
Multiple workers with cache-aware routing
smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware
With high availability mesh
smg launch --worker-urls http://gpu1:8000 --enable-mesh \
  --mesh-advertise-host 10.0.0.1 --mesh-peer-urls 10.0.0.2:39527

Use — send requests to the gateway:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3", "messages": [{"role": "user", "content": "Hello!"}]}'

That's it. SMG is now load-balancing requests across your workers.

Supported Backends

| | | |:--|:--| | Self-Hosted Engines | vLLM · SGLang · TokenSpeed · TensorRT-LLM · MLX (Apple Silicon) · any OpenAI-compatible server (e.g. Ollama) | | Cloud Providers | OpenAI · Anthropic · Google Gemini · xAI · OCI Generative AI · AWS Bedrock · Azure OpenAI · any OpenAI-compatible provider (Groq, Together, …) |

Features

| Feature | Description | |---------|-------------| | 10 Routing Policies | cacheaware, leastload, poweroftwo, consistenthashing, prefixhash, bucket, roundrobin, random, manual, passthrough | | gRPC Pipeline | Native streaming gRPC to the engines with prefill/decode and encode disaggregation and DP-aware routing | | Kubernetes Discovery | Native pod watchers with label selectors, per-role prefill/decode/encode selectors, and router peer discovery | | Model Parsers | 21 tool-call parsers and 16 reasoning parsers with automatic model detection — DeepSeek, Qwen, Kimi, GLM, Llama, Mistral, Command, Nemotron, and more | | MCP Integration | Tool discovery and execution over stdio, SSE, and streamable HTTP, with approval policies and audit logging | | High Availability | Mesh networking with SWIM gossip and CRDT-replicated state for multi-node deployments | | Chat History | Pluggable storage with schema migrations: PostgreSQL, Oracle, Redis, or in-memory | | WASM Plugins | Extend request and response handling with custom WebAssembly middleware | | Resilience | Circuit breakers, retries with backoff and jitter, rate limiting, and priority admission scheduling |

Documentation

Full documentation lives at lightseek.org/smg.

| | | |:--|:--| | Getting Started | Installation and first steps | | Architecture | How SMG works | | Configuration | CLI reference and options | | API Reference | OpenAI-compatible endpoints | | Kubernetes Setup | In-cluster discovery and production setup |

Contributing

We welcome contributions! See the Contributing Guide for details.

• Development Setup
• Code Style