Web Tools

Multi-engine web search and content extraction MCP server for AI agents. Drop-in replacement for native web_search / web_fetch tools, compatible with any LLM connected to Grok Build, Claude Code, Cursor, or other MCP hosts.

Features

• 10 classified data sources — Bing, 360 Search, Wikipedia, arXiv, Semantic Scholar, StackOverflow, Hacker News, Brave (optional), comprehensive hybrid, code search
• 4-way parallel multi-angle search — each query auto-expands into 4 complementary search angles, executed concurrently via asyncio.gather
• Content-first extraction — Jina Reader returns clean Markdown from any URL; BeautifulSoup fallback for html pages
• Smart TTL caching — search results cached for 5 min, fetched pages for 30 min; ~3000x speedup on cache hits
• Zero API keys required — all sources work out of the box; optional Brave Search API key for independent search index
• Drop-in compatible — same tool names and parameter schema as native Grok web_search/web_fetch

Quick Start

Prerequisites

• Python 3.11+
• Grok Build / Claude Code / Cursor / any MCP-compatible host

Install

git clone https://github.com/hellowind777/web-tools-mcp.git
cd web-tools-mcp
pip install -r requirements.txt

Configure

Add to ~/.grok/config.toml (or equivalent MCP host config):

[mcp_servers.web-tools]
command = "python"
args = ["/path/to/web-tools-mcp/mcp_server/server.py"]
enabled = true
startup_timeout_sec = 30
tool_timeout_sec = 30

Restart your MCP host. The tools web-tools__web_search and web-tools__web_fetch become available to your model.

Optional: Brave Search API

For higher-quality results from an independent search index:

export BRAVE_API_KEY="your-key-here"

Get a free key at brave.com/search/api (2000 queries/month free).

Tools

web_search

Classified search with 10 source options. Automatically runs 4-angle parallel queries for web/comprehensive/news sources.

Parameter	Type	Default	Description
query	string	required	Search query
source	string	"web"	Data source (see below)
num_results	integer	10	Results to return

Available sources:

Source	Backend	Best for
web	Bing (or Brave if key set)	General search
comprehensive	Bing + Wikipedia + arXiv	Research topics
wikipedia	Wikipedia API	Definitions, facts
scholar	Semantic Scholar → arXiv fallback	CS papers, citations
academic	arXiv API	Preprints, papers
code	GitHub via Bing	Repos, issues
stackoverflow	StackExchange API	Q&A with scores
hackernews	HN Algolia API	Tech news, discussions
chinese	360 Search (so.com)	Chinese-language content
news	Bing (semantic alias)	News queries

web_fetch

Fetch and extract readable content from any URL.

Parameter	Type	Default	Description
url	string	required	URL to fetch
max_length	integer	8000	Max characters returned

Uses Jina Reader as primary backend (clean Markdown, handles JS pages), falls back to BeautifulSoup HTML extraction.

Architecture

web_search(query, source)         web_fetch(url)
         │                              │
    ┌────▼────┐              ┌──────────▼──────────┐
    │  Cache   │              │   Jina Reader (1st)  │
    │ TTL 300s │              │  BeautifulSoup (2nd) │
    └────┬────┘              └──────────┬──────────┘
         │                              │
    ┌────▼──────────────────────────────▼──┐
    │         Data Source Layer             │
    │                                       │
    │  Bing ─── 360 ─── Wikipedia ─── arXiv │
    │  Scholar ─── StackOverflow ─── HN     │
    │  Brave (optional) ─── Comprehensive   │
    └───────────────────────────────────────┘

For web/comprehensive/news sources, each query is expanded into 4 parallel searches:

1. Original query
2. Official documentation
3. Community sources (GitHub/CSDN/Zhihu)
4. Latest (year-aware freshness)

Comparison

	Web Tools MCP	Tavily MCP	Brave MCP	Exa MCP
Free tier	✅ Unlimited	1000/mo	2000/mo	Trial only
Sources	10	1	1	1
Parallel multi-angle	✅ 4-way	search_depth	Single query	Single query
Content extraction	✅ Jina+BS4	extract	None	None
API key required	❌	✅	✅	✅
Chinese support	✅ 360 search	❌	❌	❌
Academic	✅ arXiv+Scholar	❌	❌	❌

Development

# Install dev dependencies
pip install -r requirements.txt

# Run tests (manual — MCP protocol)
python mcp_server/server.py

License

Apache-2.0