Audio Server

A Model Context Protocol (MCP) server that gives AI agents the ability to process audio files — transcribe speech to text, detect spoken languages, and extract audio metadata. Built with OpenAI Whisper and served over Streamable HTTP transport for seamless integration with any MCP-compatible client.

---

✨ Features

Tool	Description
speech_to_text	Transcribes spoken dialogue from an audio file into structured text using Whisper
detect_audio_language	Analyzes the first 30 seconds of audio to predict the primary spoken language with a confidence score
get_audio_metadata	Extracts technical specs — duration, bitrate, sample rate, channels, format, and file size via ffprobe

Highlights

• 🧠 Thread-safe model caching — Whisper models are loaded once and reused across requests
• 🔒 Strict input validation — All inputs are validated with Pydantic (file existence, extension support, model size)
• 📡 Streamable HTTP transport — served over Streamable HTTP at 127.0.0.1:8000/mcp; binds to loopback by default, so the MCP client must be able to reach that host (clients in Docker or on another machine need extra networking — see below)
• 🎛️ Multiple Whisper models — Choose from tiny, base, small, medium, or large depending on accuracy/speed tradeoff
• 🎵 Wide format support — .mp3, .wav, .flac, .m4a, .ogg, .mp4, .aac

---

📁 Project Structure

mcp-audio-server/
├── server.py              # MCP server entry point — registers tools, runs Streamable HTTP transport
├── audio_processor.py     # Core processing logic — transcription, language detection, metadata
├── models.py              # Pydantic models — request validation & standardized response format
├── requirements.txt       # Python dependencies
├── speech-text-MCP.json   # Pre-built n8n workflow for AI agent integration
└── tests/
    └── test_models.py     # Unit tests for input validation and response serialization

---

🛠️ Prerequisites

• Python 3.10+
• ffmpeg (required for audio metadata extraction and Whisper audio loading)
  • Windows: winget install ffmpeg or download from ffmpeg.org
  • macOS: brew install ffmpeg
  • Linux: sudo apt install ffmpeg
• GPU (optional) — Whisper will use CUDA if available, otherwise falls back to CPU

---

🚀 Getting Started

1. Clone the repository

git clone https://github.com/<your-username>/mcp-audio-server.git
cd mcp-audio-server

2. Create a virtual environment and install dependencies

Using uv (recommended):

uv venv
uv pip install -r requirements.txt

Or with standard pip:

python -m venv .venv

# Windows
.venv\Scripts\activate

# macOS / Linux
source .venv/bin/activate

pip install -r requirements.txt

3. Start the server

python server.py

The server starts on http://127.0.0.1:8000 with the following endpoint:

Endpoint	Purpose
http://127.0.0.1:8000/mcp	Streamable HTTP endpoint for MCP clients

Note: The server binds to 127.0.0.1 (loopback) and maintains per-session state over Streamable HTTP (it issues an Mcp-Session-Id), so it is only reachable from the same machine. If your MCP client runs elsewhere — n8n in Docker, or a different host — it must connect to an address that can reach this machine, not 127.0.0.1. For Docker on the same machine, use http://host.docker.internal:8000/mcp.

---

🧪 Testing

MCP Inspector

The MCP Inspector is the easiest way to test the server interactively:

npx @modelcontextprotocol/inspector

1. Open the Inspector UI in your browser
2. Set Transport Type → Streamable HTTP
3. Set URL → http://127.0.0.1:8000/mcp
4. Click Connect
5. Select any tool and provide an absolute path to an audio file

Unit Tests

pytest tests/ -v

---

🔌 Integration

n8n Workflow

A pre-built n8n workflow is included in speech-text-MCP.json. It sets up a complete AI agent pipeline:

Chat Trigger → AI Agent → Google Gemini LLM
                  ↕              ↕
            MCP Client     Buffer Memory
        (this server)

To import:

1. Start n8n (npx n8n)
2. Go to Workflows → Import from File
3. Select speech-text-MCP.json
4. Configure your Google Gemini API credentials in the Google Gemini Chat Model node
5. Ensure this MCP server is running and reachable from n8n. If n8n runs natively on the same machine, the bundled http://127.0.0.1:8000/mcp endpoint works as-is. If n8n runs in Docker, edit the MCP Client node to use http://host.docker.internal:8000/mcp — inside the container, 127.0.0.1 points at the container itself, not your host.
6. Activate the workflow and start chatting — the AI agent can now transcribe audio, detect languages, and extract metadata on demand

Note on chat models: The agent can only invoke these MCP tools if its chat model supports function/tool calling. The bundled workflow uses Google Gemini, which does. If you swap in another model (e.g. via OpenRouter), pick one that advertises tool/function-calling support — many free or base models don't, and the agent will silently answer without ever calling the tools.

Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "audio-server": {
      "url": "http://127.0.0.1:8000/mcp"
    }
  }
}

Any MCP Client

Connect to http://127.0.0.1:8000/mcp using any MCP-compatible client that supports Streamable HTTP, provided the client can reach the server's host (the server binds to loopback by default — see the note under Start the server). The server exposes three tools that are automatically discoverable through the MCP protocol.

---

📖 API Reference

speech_to_text

Transcribes audio to text using OpenAI Whisper.

Parameters:

Parameter	Type	Default	Description
audio_path	string	required	Absolute path to the audio file
model_size	string	"base"	Whisper model variant: tiny, base, small, medium, large

Response:

{
  "status": "success",
  "data": {
    "text": "The transcribed text content...",
    "language": "en"
  }
}

---

detect_audio_language

Identifies the spoken language from the first 30 seconds of audio.

Parameters:

Parameter	Type	Default	Description
audio_path	string	required	Absolute path to the audio file

Response:

{
  "status": "success",
  "data": {
    "detected_language": "en",
    "confidence_score": 0.9847
  }
}

---

get_audio_metadata

Extracts technical metadata using ffprobe.

Parameters:

Parameter	Type	Default	Description
audio_path	string	required	Absolute path to the audio file

Response:

{
  "status": "success",
  "data": {
    "format_name": "mp3",
    "duration_seconds": 245.67,
    "size_bytes": 3932160,
    "bit_rate": "128000",
    "sample_rate": "44100",
    "channels": 2
  }
}

---

Error Response

All tools return a standardized error format on failure:

{
  "status": "error",
  "message": "Validation failed: The path '/bad/path.mp3' does not exist on this machine."
}

---

⚙️ Architecture

┌─────────────────────────────────────────────────────────┐
│                     MCP Client                          │
│         (Claude, n8n, Inspector, etc.)                  │
└──────────────────────┬──────────────────────────────────┘
                       │ Streamable HTTP
                       ▼
┌──────────────────────────────────────────────────────────┐
│  server.py — FastMCP Server                              │
│  ┌────────────────┬──────────────────┬────────────────┐  │
│  │ speech_to_text │ detect_language   │ get_metadata   │  │
│  └───────┬────────┴────────┬─────────┴───────┬────────┘  │
│          │                 │                 │            │
│          ▼                 ▼                 ▼            │
│  ┌───────────────────────────────────────────────────┐   │
│  │  models.py — Pydantic Validation Layer            │   │
│  │  (AudioPathMixin, TranscriptionRequest, etc.)     │   │
│  └───────────────────────┬───────────────────────────┘   │
│                          ▼                               │
│  ┌───────────────────────────────────────────────────┐   │
│  │  audio_processor.py — Processing Engine           │   │
│  │  ┌─────────────┐  ┌───────────┐  ┌────────────┐  │   │
│  │  │   Whisper    │  │  Whisper   │  │  ffprobe   │  │   │
│  │  │ transcribe() │  │ detect()  │  │  metadata  │  │   │
│  │  └─────────────┘  └───────────┘  └────────────┘  │   │
│  └───────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────┘

---

📝 License

This project is open source. See LICENSE for details.