English | 中文

	Ferro Labs AI Gateway
	Open-Source, OpenAI-Compatible LLM Gateway

High-performance AI gateway in Go. Route LLM requests across 30 providers via a single OpenAI-compatible API.

Deploy templates

📖 Documentation: docs.ferrolabs.ai

🔀 30 providers, 2,500+ models — one API
⚡ 13,925 RPS at 1,000 concurrent users (v1.0.0 benchmark)
📦 Single static binary, no external services required, 32 MB base memory

Quick Start

Get from zero to first request in under 2 minutes.

Option A — Binary (fastest)

VER=$(curl -fsSL https://api.github.com/repos/ferro-labs/ai-gateway/releases/latest | grep '"tag_name"' | cut -d'"' -f4)
curl -fsSL "https://github.com/ferro-labs/ai-gateway/releases/download/${VER}/ferrogw${VER#v}linux_amd64.tar.gz" | tar xz
chmod +x ferrogw
./ferrogw init                        # generates config.yaml + MASTER_KEY
export GATEWAY_CONFIG=./config.yaml   # the server reads a config file only when this is set
export OPENAIAPIKEY=sk-your-key     # providers are registered at startup, so export before starting
export MASTERKEY=fgwyour-master-key # the key ferrogw init printed
./ferrogw                             # starts the server

Releases are signed and ship SBOMs — verification steps are in SECURITY.md.

Option B — Docker

docker pull ghcr.io/ferro-labs/ai-gateway:latest
docker run -p 8080:8080 \
  -e OPENAIAPIKEY=sk-your-key \
  -e MASTERKEY=fgwyour-master-key \
  ghcr.io/ferro-labs/ai-gateway:latest

Option C — Go

go install github.com/ferro-labs/ai-gateway/cmd/ferrogw@latest
ferrogw init                          # first-run setup
export GATEWAY_CONFIG=./config.yaml   # the server reads a config file only when this is set
export OPENAIAPIKEY=sk-your-key     # providers are registered at startup, so export before starting
export MASTERKEY=fgwyour-master-key # the key ferrogw init printed
ferrogw                               # start the server

ferrogw init generates the master key and writes a minimal config.yaml. The key is shown once and never written to disk — store it in your .env file or secret manager.

Minimal config

Create config.yaml (or use ferrogw init), then point the gateway at it with export GATEWAY_CONFIG=./config.yaml — a config file is loaded only when that variable names it:

strategy:
  mode: fallback
targets:   - virtual_key: openai     retry:       attempts: 3       onstatuscodes: [429, 502, 503]   - virtual_key: anthropic
aliases:   fast: gpt-4o-mini   smart: claude-3-5-sonnet-20241022

First request

export MASTERKEY=fgwyour-master-key   # the key ferrogw init printed, in whichever shell you curl from
curl http://localhost:8080/v1/chat/completions \   -H "Content-Type: application/json" \   -H "Authorization: Bearer $MASTER_KEY" \   -d '{     "model": "gpt-4o-mini",     "messages": [{"role": "user", "content": "Hello from Ferro Labs AI Gateway"}]   }' | jq

Why Ferro Labs AI Gateway

Most AI gateways are Python proxies that crack under load or JavaScript services that eat memory. Ferro Labs AI Gateway is written in Go from the ground up for real-world throughput — a single binary that routes LLM requests with predictable latency and minimal resource usage.

| Feature | Ferro Labs | LiteLLM | Bifrost | Kong AI | |:-----------------|:------------|:--------|:-----------|:------------| | Language | Go | Python | Go | Go/Lua | | Single binary | ✅ | ❌ | ✅ | ❌ | | Providers | 30 | 100+ | 20+ | 10+ | | MCP support | ✅ | ❌ | ✅ | ❌ | | Response cache | ✅ | ✅ | ✅ | ❌ (paid) | | Guardrails | ✅ | ✅ | ❌ | ❌ (paid) | | OSS license | Apache 2.0 | MIT | Apache 2.0 | Apache 2.0 | | Managed cloud | Coming Soon | ✅ | ✅ | ✅ |

Performance

Every figure in this section comes from one run: **Ferro Labs v1.0.0, measured 2026-03-23**. Benchmarked against Kong OSS, Bifrost, LiteLLM, and Portkey on GCP n2-standard-8 (8 vCPU, 32 GB RAM) using a **60ms fixed-latency mock upstream** — results reflect gateway overhead only. Later releases have not been re-measured; reproduce against the version you intend to run using the commands below.

| VU | RPS | p50 | p99 | Memory | |---:|---:|---:|---:|---:| | 50 | 813 | 61.3ms | 64.1ms | 36 MB | | 150 | 2,447 | 61.2ms | 63.4ms | 47 MB | | 300 | 4,890 | 61.2ms | 64.4ms | 72 MB | | 500 | 8,014 | 61.5ms | 72.9ms | 89 MB | | 1,000 | 13,925 | 68.1ms | 111.9ms | 135 MB |

At 1,000 VU: 13,925 RPS, p50 overhead 8.1ms, memory 135 MB. Against the live OpenAI API, the gateway itself adds 25 microseconds p50 in a typical plugin configuration (2µs bare).

Full methodology, raw results, and flamegraph analysis: ferro-labs/ai-gateway-performance-benchmarks (make setup && make bench reproduces it).

Features

🔀 Routing

• 8 routing strategies: single, fallback, load balance, least latency, cost-optimized, content-based, A/B test, conditional — see internal/strategies/README.md
• Provider failover with configurable retry policies and status code filters
• Cost-optimized routing can explicitly fallback, skip, or allow providers with unknown catalog prices
• Per-request model aliases (fast → gpt-4o-mini, smart → claude-3-5-sonnet)

🔌 Providers (30)

| OpenAI & Compatible | Anthropic & Google | Cloud & Enterprise | Open Source & Inference | |:---|:---|:---|:---| | OpenAI | Anthropic | AWS Bedrock | Ollama, Ollama Cloud | | Azure OpenAI | Google Gemini | Azure Foundry | Hugging Face | | OpenRouter | Vertex AI | Databricks | Replicate | | DeepSeek | | Cloudflare Workers AI | Together AI | | Perplexity | | | Fireworks | | xAI (Grok) | | | DeepInfra | | Mistral | | | NVIDIA NIM | | Groq | | | SambaNova | | Cohere | | | Novita AI | | AI21 | | | Cerebras | | Moonshot / Kimi | | | Qwen / DashScope |

Beyond chat and streaming, providers serve **embeddings, images, rerank, moderations, speech-to-text, text-to-speech, and batch** where the vendor offers them — the full per-provider endpoint matrix is providers/README.md.

🛡️ Guardrails & Plugins

Six plugins ship built-in — word filter, token/message limits, response cache, rate limiting, per-key budgets, and request logging — and the framework is public for writing your own. See plugin/README.md.

🎯 Provider Capabilities

• Capability matrix — one declarative record of which OpenAI parameters each provider forwards, translates, or cannot express
• GET /v1/capabilities — compare providers programmatically before you route to them
• Strict mode — compatibility.onunsupportedparam: warn | drop | reject; a parameter the provider cannot honor is no longer silently discarded
• Conformance-tested — every provider is built through the same seam the gateway uses and asserted against its real upstream payload shape

🤖 MCP (Model Context Protocol)

The gateway connects to MCP tool servers (stdio and Streamable HTTP), injects their tools into chat completions, and drives the agentic tool_calls loop itself — bounded depth, tool filtering, cross-server dedup. See mcp/README.md.

📊 Observability

• OpenTelemetry tracing — OTLP export, W3C propagation, GenAI semantic conventions plus ferro.* cost/routing/MCP attributes; a zero-allocation no-op until enabled. See observability/README.md
• Prometheus metrics at /metrics (authenticated) and deep health checks at /health, /livez, /readyz
• One trace ID threads logs, spans, and the X-Request-ID header; request logs persist to SQLite/PostgreSQL

Dashboard

Every gateway binary serves a built-in operations console at / — same port as the API, compiled in with go:embed, no second image and no second origin. Sign in with your MASTER_KEY or any admin / read-only key and it reads the live gateway: traffic, spend, provider health, routing, plugins, request logs, and the audit trail.

Overview, Analytics (latency/TTFT/cost percentiles), Providers, Routing, Plugins, a Playground over the real routing path, Tracing, Request Logs, the Audit Trail, Configuration with history/rollback, and scoped API key management.

Run the gateway and open . To see it filled like the recording above, bring up the self-contained demo stack:

make up-fullstack   # gateway + Postgres + Jaeger + Prometheus + Grafana + mock upstream + load generator
then open http://localhost:8080

Build and development details are in web/README.md.

Documentation

The root README is the overview; each subsystem keeps its own reference beside its code:

| Reference | Covers | |:---|:---| | providers/README.md | The 30 providers, the per-provider endpoint matrix, and every /v1/* surface | | config/README.md | Config loading, validation, ${VAR} secrets, declared models, trusted proxies | | internal/strategies/README.md | All 8 routing strategies and their failure semantics | | plugin/README.md | The plugin framework and the six built-in plugins | | mcp/README.md | MCP tool servers, transports, the subprocess trust boundary, readiness | | observability/README.md | Tracing setup, managed backends, emitted attributes, privacy levels, exporters | | deploy/README.md | Dockerfiles, Compose files, the fullstack demo stack | | web/README.md | Dashboard development and the embed contract | | AGENTS.md | The complete operator/developer reference: architecture, every env var, request flow | | SECURITY.md | Reporting, security posture, release verification | | CONTRIBUTING.md | Branch strategy, commit conventions, provider/plugin checklists |

Examples

Integration examples for common use cases are in ferro-labs/ai-gateway-examples:

| Example | Description | |:--------|:------------| | basic | Single chat completion to the first configured provider | | fallback | Fallback strategy — try providers in order with retries | | loadbalance | Weighted load balancing across targets (70/30 split) | | with-guardrails | Built-in word-filter and max-token guardrail plugins | | with-mcp | Local MCP server with tool-calling integration | | embedded | Embed the gateway as an HTTP handler inside an existing server |

Configuration

One YAML/JSON file, named by GATEWAY_CONFIG, drives routing, guardrails, MCP tools, and observability:

strategy:
  mode: fallback              # 8 modes — see internal/strategies/README.md
targets:   - virtual_key: openai       # an allowlist: only listed providers are routable     retry: { attempts: 3 }    # per target, honoured under every routing mode     concurrency: { maxconcurrency: 32, queuesize: 1000 }   - virtual_key: anthropic
aliases:   fast: gpt-4o-mini
plugins:                      # guardrails first, then cache — see plugin/README.md   - name: word-filter     type: guardrail     stage: before_request     enabled: true     config: { blocked_words: ["password", "secret"] }
mcp_servers:                  # tool servers — see mcp/README.md   - name: search     url: https://mcp.example.com/mcp     headers: { Authorization: "Bearer ${SEARCH_TOKEN}" }

${VAR} references (braced form only) resolve when a component is constructed, never at file load — so secrets are never stored, served by GET /admin/config, or restored by a rollback. A bare $ is data; an undefined variable is an error.

The full annotated reference with every option is config.example.yaml / config.example.json, and the schema guide is config/README.md. ferrogw validate checks a file without starting the server.

Key environment variables

| Variable | Purpose | |----------|---------| | MASTER_KEY | Bootstrap and break-glass admin credential (generated by ferrogw init); give each operator their own key from POST /admin/keys for day-to-day use | | GATEWAY_CONFIG | Path to config YAML/JSON | | GATEWAYENV | Set to production to enable production-mode safety guards: it refuses to start on ALLOWUNAUTHENTICATEDPROXY=true or a * entry in CORSORIGINS, and warns when per-IP rate limiting is off, pprof is mounted, or the API key store is in-memory | | PORT | Server port (default: 8080) | | ALLOWUNAUTHENTICATEDPROXY | Set to true to disable proxy-route auth (dev only; blocked when GATEWAY_ENV=production) | | CORS_ORIGINS | Comma-separated allowed CORS origins; cross-origin is denied when unset. Matched literally — there is no wildcard, so list each origin explicitly | | TRUSTEDPROXIES | CIDRs of trusted reverse proxies; forwarded headers are honored only from these (default: loopback). See config/README.md | | <PROVIDER>BASEURL | Points a provider at a proxy, self-hosted server, or regional endpoint. It is the API root, used verbatim — write it exactly as the vendor documents it, version segment included (https://api.groq.com/openai/v1); a bare host resolves to the provider's own version segment |

make up-fullstack   # then open Grafana at http://localhost:3000

export OTELEXPORTEROTLP_ENDPOINT=localhost:4317
ferrogw serve

export OPENAIAPIKEY=sk-your-key
export MASTERKEY=fgwyour-master-key
export GATEWAY_CONFIG=./config.yaml
make build && ./bin/ferrogw

make up             # dev: builds from source
IMAGETAG=v1.4.0 CORSORIGINS=https://your-domain.com make up-prod
make down           # tears down either

helm repo add ferro-labs https://ferro-labs.github.io/helm-charts
helm repo update
helm install ferro-gw ferro-labs/ai-gateway \
  --set env.OPENAIAPIKEY=sk-your-key

from litellm import completion
response = completion(     model="gpt-4o",     messages=[{"role": "user", "content": "Hello"}] )

from openai import OpenAI
client = OpenAI(     base_url="http://localhost:8080/v1",     api_key="your-ferro-api-key", )
response = client.chat.completions.create(     model="gpt-4o",     messages=[{"role": "user", "content": "Hello"}], )

from ferrolabs import FerroClient
client = FerroClient(     base_url="http://localhost:8080/v1",     api_key="your-ferro-api-key", )
response = client.chat.completions.create(     model="gpt-4o",     messages=[{"role": "user", "content": "Hello"}], )

import { FerroClient } from "ferrolabs";
const client = new FerroClient({   baseURL: "http://localhost:8080/v1",   apiKey: "your-ferro-api-key", });
const response = await client.chat.completions.create({   model: "gpt-4o",   messages: [{ role: "user", content: "Hello" }], });