Server Metasearch

CI Python 3.10+ License: MIT PyPI

English | 简体中文

A local MCP (Model Context Protocol) server that aggregates 15 web search and extraction tools across 5 providers (Jina, Tavily, Exa, Firecrawl, Bocha) behind a unified interface. Each tool is gated by both an on/off switch and an API key check — AI agents only see tools they can actually use.

Status: v0.2.7 — Production-ready with full test coverage (123 tests, 91% coverage).

Why MCP Server Metasearch?

Instead of configuring 5 separate MCP servers (one per provider), metasearch gives you a single unified interface with:

• Zero-config tool exposure — only tools with valid API keys are visible to AI agents
• Plugin architecture — add a new provider by dropping one Python file, no framework changes
• Shared infrastructure — connection pooling, response caching, and startup diagnostics out of the box

Table of Contents

• Quick Start
• Core Features
• Deployment Guide
  • Claude Code
  • OpenClaw
  • Hermes Agent
• Built-in Tools
• Configuration Reference
• Adding New Tools
• Troubleshooting FAQ
• Development & Testing

Quick Start

1. Configure

mkdir -p ~/.config/mcp-server-metasearch
cp .env.example ~/.config/mcp-server-metasearch/.env
# Edit ~/.config/mcp-server-metasearch/.env with your API keys

2. Run

Recommended — via uvx (no install needed):

uvx mcp-server-metasearch

Or connect via any MCP client — see Deployment Guide below.

pip install mcp-server-metasearch
mcp-server-metasearch

git clone https://github.com/busigui2023/mcp-server-metasearch.git
cd mcp-server-metasearch
uv venv && uv pip install -e ".[dev]"
mcp-server-metasearch

Core Features

• Plugin architecture: Add or remove web tools by dropping a single Python file. No framework code changes.
• Dual validation: Every tool requires both TOOL_*_ENABLED=true and the presence of its API key(s) to be exposed.
• Key-optional support: Some tools may work without an API key; others strictly require one.
• Startup resilience: If zero tools are available, the server fails with a detailed diagnostic report and retries up to 3 times before giving up.
• Local logging: All runtime logs write to both stderr (safe for stdio transport) and logs/mcp-server-metasearch.log.
• Connection pool reuse: Shared httpx.AsyncClient across all tools eliminates per-request connection overhead, with graceful shutdown via atexit.
• Response caching: In-memory TTL cache (600s) with registry-level wrapping — zero tool code changes.
• Optimized config loading: Environment file is read only once per process lifetime, avoiding repeated disk I/O on every tool call.
• Clean protocol handshake: The server does not advertise unused resources or prompts capabilities. This prevents MCP clients (e.g. Hermes) from auto-generating utility stubs for features this server does not implement, keeping the agent's tool list focused on the 15 actual search tools.

Deployment Guide

Prerequisites

• Python 3.10+
• uv installed (provides uvx for zero-install runs)
• API keys for the services you want to enable

1. Configure

See Quick Start above, then continue with your MCP client setup.

2. Connect to an MCP Client

Note: The recommended way is uvx — no pip install needed. uvx auto-downloads and caches the package on first run, then reuses the cache. If you installed from source, use the "From Source" tab instead.

Claude Code

uvx (recommended):

claude mcp add metasearch -- uvx mcp-server-metasearch

{
  "mcpServers": {
    "metasearch": {
      "type": "stdio",
      "command": "uvx",
      "args": ["mcp-server-metasearch"]
    }
  }
}

From Source:

claude mcp add metasearch -- uv run --directory /absolute/path/to/mcp-server-metasearch mcp-server-metasearch

{
  "mcpServers": {
    "metasearch": {
      "type": "stdio",
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/absolute/path/to/mcp-server-metasearch",
        "mcp-server-metasearch"
      ]
    }
  }
}

Restart Claude Code (/quit then re-enter). Run /mcp to verify the server appears.

OpenClaw

uvx (recommended):

openclaw mcp add metasearch --command uvx --arg mcp-server-metasearch

{
  "mcp": {
    "servers": {
      "metasearch": {
        "command": "uvx",
        "args": ["mcp-server-metasearch"],
        "transport": "stdio"
      }
    }
  }
}

From Source:

openclaw mcp add metasearch \
  --command uv \
  --arg run \
  --arg --directory \
  --arg /absolute/path/to/mcp-server-metasearch \
  --arg mcp-server-metasearch

{
  "mcp": {
    "servers": {
      "metasearch": {
        "command": "uv",
        "args": [
          "run",
          "--directory",
          "/absolute/path/to/mcp-server-metasearch",
          "mcp-server-metasearch"
        ],
        "transport": "stdio"
      }
    }
  }
}

Run openclaw mcp probe metasearch to test the connection without starting a full agent turn.

Hermes Agent

uvx (recommended):

# ~/.hermes/config.yaml
mcp_servers:
  metasearch:
    command: "uvx"
    args: ["mcp-server-metasearch"]

From Source:

# ~/.hermes/config.yaml
mcp_servers:
  metasearch:
    command: "uv"
    args:
      - "run"
      - "--directory"
      - "/absolute/path/to/mcp-server-metasearch"
      - "mcp-server-metasearch"

Restart Hermes or run hermes mcp list to verify. Use hermes mcp test metasearch to check connectivity and tool discovery.

Manual Test

uvx (recommended):

uvx mcp-server-metasearch

From Source:

uv run --directory /absolute/path/to/mcp-server-metasearch mcp-server-metasearch

If no tools are enabled, you will see a diagnostic table and the process exits with code 1. Fix your .env and retry.

Troubleshooting FAQ

Q: The diagnostic table shows all tools as "Skipped (switch disabled)" even though I set them to true.

A: The .env file must live at ~/.config/mcp-server-metasearch/.env, not in the project root. The server does not read a local .env file inside the repository. Double-check the path and ensure there are no trailing spaces after true.

Q: The diagnostic table shows "Skipped (missing API key)" but I added the key.

A: Check for these common mistakes:

• The key value is empty after the = sign.
• There is a # comment on the same line that truncates the value.
• The key name is misspelled (e.g. JINA_APIKEY instead of JINA_API_KEY).
• The .env file was saved with Windows line endings (\r\n) in a way that breaks parsing.

Q: The MCP client says "MCP server metasearch failed to start" or the process exits immediately.

A: This usually means zero tools passed validation and the server exited with code 1. Check the client logs (stderr) for the diagnostic table. If you see retry messages (Retry 1/3, Retry 2/3), the server is working as designed — fix your .env before the 3rd retry or the client may stop trying.

Q: I enabled some tools but the client only shows a subset of them.

A: Each tool is validated independently. The missing tools likely failed their own key or switch check. Look at the startup diagnostics — the table lists every tool and the exact reason it was skipped. Common cases:

• JINA_API_KEY missing → all jina tools hidden
• TAVILY_API_KEY missing → all tavily tools hidden
• A tool's switch set to false → that single tool hidden

Since jina tools share the same key, they appear or hide together. Tavily tools also share the same key.

Q: uv run mcp-server-metasearch works manually but fails when launched by the MCP client.

A: MCP clients often launch the server with a different working directory or environment. Ensure:

• You use --directory /absolute/path/to/mcp-server-metasearch so uv can find pyproject.toml and set the correct working directory.
• The .env path ~/.config/mcp-server-metasearch/.env uses the absolute home directory and does not depend on the working directory.

Q: How do I completely reset the startup retry counter?

A: Delete the retry file:

rm ~/.cache/mcp-server-metasearch/startup_retries

This is useful when testing configuration changes and you want a fresh start.

Q: Where do I find logs when the server is launched by a client?

A: Logs are written to stderr (safe for stdio transport) and also persisted to:

logs/mcp-server-metasearch.log

If the client captures stderr, look there. If not, check the log file in the project directory. Note: the log file is created relative to the server's working directory, so if the client changes CWD, the log may appear in an unexpected location.

Q: Can I run the server without uv or uvx?

A: Yes. Install globally with pip install mcp-server-metasearch (preferably in a dedicated virtual environment to avoid polluting your system Python), then run mcp-server-metasearch directly. Note: some Linux distributions (Ubuntu 23.04+) block global pip install due to PEP 668 — in that case, use uvx or a virtual environment.

Built-in Tools

Current total: 15 tools across 5 providers.

Not sure which tool to use? Here's the decision matrix:

jina_reader

Fetch and extract clean, LLM-friendly content from any webpage, PDF, or document URL using jina.ai Reader.

When to use: The AI needs to read a specific URL, parse a PDF link, or extract article content from a JS-heavy site.

Parameters:

API requirement: JINA_API_KEY is required.

Example flow:

1. AI receives a user question about a blog post.
2. AI calls jina_reader with the blog URL.
3. Server requests https://r.jina.ai/http://<url> and returns clean Markdown.

jina_search

Search the web using jina.ai Search and retrieve LLM-friendly results with full page summaries.

When to use: The AI needs current web information, fact-checking, or multi-source summaries.

Parameters:

API requirement: JINA_API_KEY is required.

Example flow:

1. User asks "What are the latest Python 3.14 features?"
2. AI calls jina_search with query="Python 3.14 new features".
3. Server returns up to 5 results, each with title, URL, and full content summary.

jina_deepsearch

Perform multi-step research on a complex topic using jina.ai DeepSearch. Combines web search, page reading, and reasoning into a single comprehensive answer with cited sources.

When to use: The question is broad or exploratory (e.g. "Compare vector databases for RAG in 2026"). DeepSearch autonomously searches multiple sources, reads them, and synthesizes a report.

Parameters:

API requirement: JINA_API_KEY is required.

Example flow:

1. User asks "What are the trade-offs between Weaviate, Qdrant, and Milvus for production RAG?"
2. AI calls jina_deepsearch with the full question.
3. Server sends the query to https://deepsearch.jina.ai/v1/chat/completions.
4. Response is a structured research report with inline citations.

tavily_search

Search the web using Tavily with fine-grained filters, relevance scoring, and optional AI-generated answers.

When to use: You need precise search with time-range filters, domain restrictions, country boosting, or an AI-generated summary answer.

Parameters:

API requirement: TAVILY_API_KEY is required. 1,000 free credits/month.

Example flow:

1. User asks "What happened in AI this week?"
2. AI calls tavily_search with query="AI news", topic="news", time_range="week", include_answer=true.
3. Server returns ranked results plus a concise AI-generated answer.

tavily_extract

Extract clean, LLM-friendly content from one or more URLs using Tavily Extract. Supports batch extraction and query-guided chunk reranking.

When to use: You already know the URLs and want to extract their content in bulk, or you want query-relevant chunks instead of full pages.

Parameters:

API requirement: TAVILY_API_KEY is required.

Example flow:

1. AI finds 3 relevant URLs from a prior search.
2. AI calls tavily_extract with urls=[url1, url2, url3].
3. Server returns clean markdown for all three pages.

tavily_research

Perform comprehensive, multi-step research using Tavily Research. Conducts multiple searches, analyzes sources, and generates a cited research report.

When to use: The question is broad and requires a synthesized report with citations (e.g. "Compare cloud providers for ML workloads in 2026"). Higher credit consumption than search.

Parameters:

API requirement: TAVILY_API_KEY is required. Research consumes significantly more credits than a single search because it runs multiple internal search + extract + synthesis steps.

Example flow:

1. User asks "What are the trade-offs between Weaviate, Qdrant, and Milvus for production RAG?"
2. AI calls tavily_research with the full question.
3. Server submits an async research task and polls until completion.
4. Response is a structured research report with numbered citations and a source list.

exa_search

Search the web using Exa, a search engine optimized for LLMs. Returns highly relevant excerpts (highlights) by default for 10x token efficiency. Supports category filters and deep-reasoning modes.

When to use: You need token-efficient search results, company/people/research-paper category filtering, or the fastest search latency (instant ~250ms).

Parameters:

API requirement: EXA_API_KEY is required.

Example flow:

1. User asks "Find Series A agtech companies in the US."
2. AI calls exa_search with query="agtech companies US Series A", category="company", content_mode="highlights".
3. Server returns up to 10 company pages with key excerpts.

exa_contents

Extract clean, LLM-ready content from one or more URLs using Exa Contents. Handles JS-rendered pages, PDFs, and complex layouts. Supports subpage crawling.

When to use: You already know the URLs and want to extract their content, optionally crawling linked subpages.

Parameters:

API requirement: EXA_API_KEY is required.

Example flow:

1. AI finds a relevant documentation URL.
2. AI calls exa_contents with urls="https://docs.example.com", subpages=10, subpage_target=["api", "reference"].
3. Server returns content from the root page plus up to 10 linked subpages.

exa_answer

Get a direct LLM answer to a question informed by Exa search results. Ideal for quick factual lookups.

When to use: You need a concise answer to a specific question, not a list of search results.

Parameters:

API requirement: EXA_API_KEY is required.

Example flow:

1. User asks "What is SpaceX's latest valuation?"
2. AI calls exa_answer with the question.
3. Server returns "$350 billion" plus cited sources.

firecrawl_scrape

Extract clean, LLM-ready content from any webpage using Firecrawl. Supports multiple output formats and dynamic content rendering.

When to use: The AI needs precise content from a known URL, especially JS-rendered sites, or wants HTML/links in addition to markdown.

Parameters:

API requirement: FIRECRAWL_API_KEY is required.

firecrawl_search

Search the web using Firecrawl and get structured content from results. Supports web, news, and image search with specialized category filtering.

When to use: The AI needs to search with image results, news results, or filter by GitHub/research-paper/PDF categories.

Parameters:

API requirement: FIRECRAWL_API_KEY is required.

Cost note: 2 credits per 10 search results. scrape_content=True adds 1 credit per result page.

firecrawl_map

Discover all URLs on a website quickly. Returns a complete link list with titles and descriptions.

When to use: The AI wants to explore a site's structure before deciding which pages to read, or needs to find specific pages on a large site.

Parameters:

API requirement: FIRECRAWL_API_KEY is required.

Cost note: Always 1 credit per call, regardless of URL count.

firecrawl_crawl

Recursively crawl a website, discovering and scraping all reachable subpages automatically.

When to use: The AI needs to ingest an entire documentation site, blog, or any multi-page resource.

Parameters:

API requirement: FIRECRAWL_API_KEY is required.

Cost note: 1 credit per page crawled. Default limit=10 = max 10 credits.

bocha_web_search

Search the web using Bocha AI, a China-optimized search engine powering DeepSeek's web search. Returns clean, structured results with webpage titles, URLs, snippets, site info, and optional AI summaries.

When to use: The AI needs to search Chinese content, access domestic websites, or needs a search tool that works reliably without proxy.

Parameters:

API requirement: BOCHA_API_KEY is required.

Cost note: ¥0.036 per call. Free tier available for personal use.

bocha_ai_search

Advanced AI-powered search using Bocha AI. Returns web results plus structured modal cards (weather, encyclopedia, stock, train schedules, medical info, etc.) and optional AI-generated answers with follow-up questions.

When to use: The AI needs structured data extraction (e.g., "What's the weather in Beijing?", "Stock price of Alibaba"), or wants an AI-generated summary answer alongside search results.

Parameters:

API requirement: BOCHA_API_KEY is required.

Cost note: ¥0.060 per call. Returns modal cards + AI answer when answer=True.

Configuration Reference

.env Full Example

# ── Jina AI ──
JINA_API_KEY=jina_xxxxxxxxxxxxxxxxxxxxxxxx
TOOL_JINA_READER_ENABLED=true
TOOL_JINA_SEARCH_ENABLED=true
TOOL_JINA_DEEPSEARCH_ENABLED=true

# ── Tavily ──
# TAVILY_API_KEY=tvly-xxxxxxxx
# TOOL_TAVILY_SEARCH_ENABLED=true
# TOOL_TAVILY_EXTRACT_ENABLED=true
# TOOL_TAVILY_RESEARCH_ENABLED=true

# ── Exa ──
# EXA_API_KEY=your_exa_api_key_here
# TOOL_EXA_SEARCH_ENABLED=true
# TOOL_EXA_CONTENTS_ENABLED=true
# TOOL_EXA_ANSWER_ENABLED=true

# ── Firecrawl ──
# FIRECRAWL_API_KEY=fc-xxxxxxxxxxxxxxxx
# TOOL_FIRECRAWL_SCRAPE_ENABLED=true
# TOOL_FIRECRAWL_SEARCH_ENABLED=true
# TOOL_FIRECRAWL_MAP_ENABLED=true
# TOOL_FIRECRAWL_CRAWL_ENABLED=true

# ── 博查 Bocha AI ──
# BOCHA_API_KEY=sk-xxxxxxxxxxxxxxxx
# TOOL_BOCHA_WEB_SEARCH_ENABLED=true
# TOOL_BOCHA_AI_SEARCH_ENABLED=true

How Dual Validation Works

For every discovered tool, the server checks two conditions at startup:

1. Switch check: TOOL_<NAME>_ENABLED must be true.
2. Key check: All required_env_vars must be present and non-empty in .env.

Both must pass for the tool to be registered. If a tool fails either check, it appears in the startup diagnostic table with the exact reason.

Development & Testing

Run Tests

uv run pytest tests/ -v

View Logs

Runtime logs are written to:

• stderr (MCP-safe, visible in client consoles)
• logs/mcp-server-metasearch.log (rotated at 5 MB, 3 backups kept)

tail -f logs/mcp-server-metasearch.log

Startup Diagnostics

If no tools are registered, the server prints a diagnostic table to stderr:

[MCP-Metasearch] STARTUP DIAGNOSTICS
┌────────────────────┬─────────┬─────────────────┬─────────────────────────────┐
│ Tool               │ Switch  │ API Key         │ Result                      │
├────────────────────┼─────────┼─────────────────┼─────────────────────────────┤
│ jina_reader        │ OFF     │ JINA_API_KEY    │ Skipped (switch disabled)   │
│ jina_search        │ ON      │ MISSING         │ Skipped (missing API key)   │
└────────────────────┴─────────┴─────────────────┴─────────────────────────────┘

Adding New Tools

1. Create src/mcp_server_metasearch/tools/my_tool.py.
2. Inherit from BaseTool:

from mcp_server_metasearch.tools.base import BaseTool

class MyTool(BaseTool):
    name = "my_tool"
    description = "What this tool does."
    required_env_vars = ["MY_API_KEY"]   # or [] if no key needed
    enabled_env_var = "TOOL_MY_TOOL_ENABLED"

    async def call(self, param: str) -> str:
        ...

3. Add the key and switch to ~/.config/mcp-server-metasearch/.env.
4. Restart the MCP client. The tool is auto-discovered — no edits to server.py or tool_registry.py needed.

Project Structure

mcp-server-metasearch/
├── .env.example              # Config template
├── pyproject.toml            # Package metadata & dependencies
├── scripts/                  # E2E smoke tests per provider
├── src/mcp_server_metasearch/
│   ├── server.py             # FastMCP instance & startup flow
│   ├── tool_registry.py      # Auto-discovery, dual validation & cache wrapping
│   ├── config.py             # .env loading (single-read) & pydantic-settings
│   ├── http_client.py        # Shared httpx.AsyncClient with graceful shutdown
│   ├── cache.py              # In-memory TTL response cache
│   ├── diagnostics.py        # Startup failure reports
│   ├── retry.py              # Persistent retry counter
│   └── tools/                # One file per tool (plugin architecture)
│       ├── base.py           # BaseTool abstract class
│       ├── formatting.py     # Shared formatting utilities
│       ├── jina_*.py         # Jina AI tools (reader, search, deepsearch)
│       ├── tavily_*.py       # Tavily tools (search, extract, research)
│       ├── exa_*.py          # Exa tools (search, contents, answer)
│       ├── firecrawl_*.py    # Firecrawl tools (scrape, search, map, crawl)
│       └── bocha_*.py        # Bocha tools (web_search, ai_search)
└── tests/                    # Unit tests (123 tests, 91% coverage)

Acknowledgments

This project integrates with the following search and extraction services:

• Jina AI — Web reader and search
• Tavily — Search, extract, and research
• Exa — Semantic search for LLMs
• Firecrawl — Web scraping and crawling
• Bocha AI — Chinese-optimized search

Users need their own API keys for these services. This project does not provide or distribute any API keys.

License

MIT License — see LICENSE for details.

Contributing

See CONTRIBUTING.md for development setup and guidelines.