SSMCP

Model Context Protocol (MCP) server providing web search with content extraction.

Why Use SSMCP?

Many AI models, especially local models or certain cloud-based models, don't have built-in web browsing capabilities. SSMCP bridges that gap by providing a simple, self-hosted solution that gives your AI assistant the ability to:

• Search and Read the Web: Let your AI search for current information, read articles, documentation, or any web content
• Extract Clean Content: Automatically converts messy web pages into clean Markdown format that AI models can easily understand
• Access YouTube Transcripts: Extract subtitles from videos with timestamps for analysis or summarization
• AI-Powered Summarization: Optional LLM summarization extracts only relevant information from web pages based on your search query
• Privacy-Focused: Self-hosted solution - your searches and browsing stay on your infrastructure
• Works with Any Model: Compatible with local models (like Qwen, Llama) and cloud APIs (DeepSeek, Claude, GPT) that support MCP

Example Use Cases:

• Research recent news or developments on a topic
• Read and summarize technical documentation
• Analyze current market trends or product reviews
• Extract information from YouTube tutorials or presentations
• Get up-to-date answers that aren't in the model's training data
• Get concise, query-relevant summaries of web content (with LLM summarization enabled)

Features

• Web Search: Search the web and get results with extracted content in Markdown
• Web Fetch: Fetch and extract content from any URL as clean Markdown
• YouTube Subtitles: Extract subtitles and timestamps from YouTube videos
• LLM Summarization (Optional): Use an LLM to summarize and filter search results by relevance to your query
• Powered by: SearXNG for search, Crawl4AI for content extraction, yt-dlp for subtitles
• Simple API: Easy-to-use interface designed for compatibility with both small local models(like Qwen3 30b) and cloud models lacking web capabilities (e.g., DeepSeek 3.2)
• Container Support: Full containerized deployment with Docker Compose

Quick Start

Prerequisites

• Docker
• Docker Compose

1. Clone this repository

git clone https://github.com/antonsokolskyy/SSMCP.git

2. Create .env file

cd SSMCP/
cp .env.example .env

3. Set Up SearXNG

Start and stop the searxng container to generate settings.yml:

docker compose up searxng

Wait until you see:

"/etc/searxng/settings.yml" does not exist, creating from template...

Then press Ctrl+C to stop it.

Edit deploy/docker/searxng_data/settings.yml and add json to the formats list:

# remove format to deny access, use lower case.
# formats: [html, csv, json, rss]
formats:
  - html
  - json

4. Build the SSMCP image

docker compose build

5. Run the Full Stack

docker compose up -d

YouTube Cookies (Optional)

To access age-restricted or private YouTube videos, and to reduce the chances of hitting captchas or IP blocking, you can provide cookies from your browser.

Note: The cookies file must be in Netscape cookie format.

yt-dlp --cookies-from-browser chrome --cookies cookies.txt https://www.youtube.com/watch?v=VIDEO_ID

deploy/docker/ssmcp/cookies.txt

chmod 600 deploy/docker/ssmcp/cookies.txt

MCP URL

The server uses Streamable HTTP transport. Connect to the MCP server at:

http://{HOST}:{PORT}/mcp

Example:

http://localhost:8000/mcp

Usage with MCP Clients

LM Studio

{
  "mcpServers": {
    "ssmcp": {
      "url": "http://localhost:8000/mcp"
    }
  }
}

Web UI

SSMCP includes an optional Web UI for monitoring requests and responses. This is particularly useful for debugging and inspecting how the model uses the tools.

REDIS_URL=redis://redis:6379

make build

Tools

web_search

Performs a web search and returns relevant results with extracted content.

Parameters:

• query (str): Search query or keywords to find relevant web content

Returns:

• List of results, each containing:
  • url (str): The webpage URL
  • content (str): Page content in Markdown format

web_fetch

Fetches content from a specified URL and converts it to Markdown.

Parameters:

• url (str): The URL to fetch content from

Returns:

• String containing the page content in Markdown format

youtube_get_subtitles

Gets subtitles/captions from a YouTube video and returns the text content.

Parameters:

• url (str): YouTube video URL to get subtitles from

Returns:

• String containing the subtitles with timestamps in format: [HH:MM:SS] text

How Search Works

SSMCP uses a pipeline to deliver clean content from web searches:

1. Search (SearXNG)

• Queries are sent to a local SearXNG instance
• SearXNG aggregates results from multiple search engines
• Returns a list of URLs with titles and snippets

2. Content Extraction (Crawl4AI)

• Each URL is fetched using headless Chromium browser
• Pages are fully rendered (JavaScript executed, dynamic content loaded)
• Raw content is extracted
• All URLs are processed concurrently

3. Content Filtering

Two-stage filtering extracts clean main content:

1. CSS Selector Filter - Tries selectors (article, main, etc.) to find main content area
2. Residual Junk Filter - Removes UI artifacts (tooltips, duplicate text)

If any filter produces output, the filtered HTML is re-processed through Crawl4AI for cleaner output before markdown conversion.

4. Markdown Conversion

• Converts filtered HTML to clean Markdown
• Removes images and external links

5. Optional: LLM Summarization

When enabled, SSMCP uses an LLM to summarize search results before returning them to your AI model. This reduces the token count in your context window - search results can be 40k-60k tokens, but summaries significantly reduce that.

Note: LLM summarization is disabled by default. Enable it via environment variables if needed (see .env.example).

Development

All development tasks are performed inside the Docker container. Nothing needs to be installed on the host machine except Docker and Docker Compose.

Available Make Commands

Run make help to see all available commands:

Configuration

All configuration is managed through environment variables. See .env.example for available options

OpenWebUI OAuth Authentication

SSMCP supports OAuth authentication for use with OpenWebUI and any OIDC-compliant identity provider.

How OpenWebUI OAuth Works

When you select OAuth in OpenWebUI for an MCP server:

• OpenWebUI forwards the system user's OAuth access token in the Authorization: Bearer <token> header
• The token is a JWT containing user information from the identity provider
• SSMCP validates the token and extracts the user identifier from the sub claim

# Enable OAuth authentication
OAUTH_ENABLED=true

# JWKS endpoint URL for your identity provider's public keys
# Examples:
#   Authentik: https://authentik.example.com/application/o/my-app/jwks
OAUTH_JWKS_URL=https://your-idp.example.com/path/to/jwks

# Issuer URL for token issuer verification
# Must match the 'iss' claim in JWT tokens
# Examples
#   Authentik: https://authentik.example.com/application/o/my-app
OAUTH_ISSUER=https://your-idp.example.com/application/o/my-app

# Open WebUI client ID for token audience verification
OAUTH_CLIENT_ID=your-openwebui-client-id

License

Apache License 2.0 - see LICENSE file for details.