AI Gateway

CI

Unified local MCP AI Gateway that routes across Groq, OpenRouter, Mistral, and local Ollama providers, with OpenAI-compatible APIs, MCP tools, fallback/racing router, monitoring, and web dashboard.

🚀 Features

• Async Python gateway with provider abstraction layer
• Built-in providers:
  • Groq (openai/gpt-oss-120b default, plus moonshotai/kimi-k2-instruct-0905)
  • OpenRouter (openai/gpt-4o-mini default)
  • Mistral (codestral-latest default, plus devstral-medium-latest)
  • Ollama
• Intelligent router:
  • priority routing
  • fallback routing
  • parallel racing mode
  • rate-limit aware provider scoring
• MCP server capabilities:
  • tool registration/discovery
  • prompt template discovery/rendering
  • schema validation
  • SSE and HTTP transport
• OpenAI-compatible API:
  • POST /v1/chat/completions
  • POST /v1/completions
  • GET /v1/models
• Web UI (React + Vite + Tailwind):
  • Dashboard
  • Providers
  • Models
  • Chat
  • Logs
  • Stats
  • Runtime gateway API key generation
• Observability:
  • Prometheus metrics (/metrics)
  • structured logs
  • SQLite request statistics
• Extensible plugin system for additional providers
• CLI: mcp-ai start|stop|status|providers|models
• Docker + docker-compose support
• GitHub Actions CI for backend lint/tests and frontend build

🗂️ Repository Structure

mcp-ai-gateway/
  api/
  providers/
  router/
  mcp_server/
  plugins/
  database/
  web_ui/
  config/
  cli/
  tests/
  Dockerfile
  docker-compose.yml
  README.md

⚡ Quick Start (Local)

1) 📦 Prerequisites

• Python 3.11+
• Node.js 20+
• Optional: local Ollama daemon at http://localhost:11434

2) 🐍 Install backend

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e .[dev]

3) 🔐 Configure environment

cp .env.example .env
# edit .env values

4) ⚙️ Edit config

config/config.yaml controls providers, routing, security, and default models.

5) ▶️ Start backend

uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload

6) 🖥️ Start web UI

cd web_ui
npm install
VITE_API_BASE=http://localhost:8000 VITE_API_KEY=change-me npm run dev

Open http://localhost:3000

🐳 Docker Deployment

docker compose up --build

Services:

• Gateway API: http://localhost:8000
• Web UI: http://localhost:3000
• Redis: localhost:6379 (optional cache/rate-limit backend)

🧩 API Overview

OpenAI-Compatible 🤝

• POST /v1/chat/completions
• POST /v1/completions
• GET /v1/models

Use header X-API-Key: <key> (or Authorization: Bearer <key>).

MCP Endpoints 🔌

• JSON-RPC MCP transport: POST /mcp
• SSE streaming output: POST /mcp/stream

Supported MCP methods:

• initialize
• notifications/initialized
• ping
• tools/list
• tools/call
• prompts/list
• prompts/get

Dashboard APIs 📊

• GET /api/dashboard
• GET /api/providers
• POST /api/providers/{name}/toggle
• POST /api/providers/{name}/priority
• POST /api/providers/{name}/config
• GET /api/models
• GET /api/settings
• POST /api/settings/default-model
• POST /api/settings/api-key/generate
• GET /api/logs
• GET /api/stats

Monitoring 📈

• GET /health
• GET /metrics

🛠️ CLI

mcp-ai start
mcp-ai status
mcp-ai providers
mcp-ai models
mcp-ai stop

🧭 Warp MCP Integration

Add this to Warp MCP config:

{
  "mcpServers": {
    "mcp-ai-gateway": {
      "url": "http://localhost:8000/mcp",
      "headers": {
        "Authorization": "Bearer REPLACE_WITH_MCP_GATEWAY_API_KEY"
      }
    }
  }
}

🧱 Configuration Example

providers:
  groq:
    enabled: true
    api_key: ${GROQ_API_KEY}
    priority: 1
    default_model: openai/gpt-oss-120b
  mistral:
    enabled: false
    api_key: ${MISTRAL_API_KEY}
    priority: 12
    default_model: codestral-latest
  openrouter:
    enabled: true
    priority: 2
    default_model: openai/gpt-4o-mini
  ollama:
    enabled: true
    priority: 3
routing:
  fallback_enabled: true
  racing_mode: false

🧪 Plugin Providers

New providers can be added under plugins/ by implementing ProviderPlugin and enabling provider config in config/config.yaml.

Included plugin examples:

• openai_provider.py
• together_provider.py
• openrouter_provider.py

✅ Testing

pytest

CI runs on pushes and pull requests to main via .github/workflows/ci.yml:

• Backend job: ruff check . and pytest -q
• Frontend job: npm install and npm run build in web_ui/

🔄 Release Workflow

• Open PR from dev/* branch to main
• Ensure CI is green
• Merge using GitHub CLI or GitHub UI

📚 Developer Guide

Detailed guide: docs/developer-guide.md