WebTool Server

Browser & info access helper for local LLMs via the Model Context Protocol (MCP). Exposes a single HTTP JSON-RPC endpoint LM Studio (and other MCP clients) can call. Optimized for iterative, low‑token browsing: outline first → selective drill‑down → optional link follow.

Features

Tools currently exposed:

Tool	Purpose
fetch_url	Fetch & parse a webpage. Outline-only mode, per‑section retrieval, single‑hop link follow (link_id), or focused chunk view.
web_search	Multi-engine search (duckduckgo, bing, google_cse, multi aggregate).
search_wikipedia	Concise summary of a topic from Wikipedia REST API.
latvian_news	Latest Latvian headlines (Google News RSS) or topic search.
search_duckduckgo	Legacy single DuckDuckGo lookup (prefer web_search).
ai_company_news	Recent headlines per AI/tech company (OpenAI, Google, Anthropic, Microsoft, Nvidia).
get_system_prompt	Returns the internal system prompt with usage guidance.

All tools are discoverable through the MCP tools/list (or tools.list) JSON-RPC method.

Repo

GitHub: https://github.com/SashaYerashoff/webtool-mcp

Quick Start (Ubuntu / Debian / WSL)

sudo apt update && sudo apt install -y python3 python3-venv git
git clone https://github.com/SashaYerashoff/webtool-mcp.git
cd webtool-mcp
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
python app.py  # serves on http://0.0.0.0:5000 (http://localhost:5000)

Keep the process running (e.g. with tmux, screen, or a systemd service) if you want persistent availability.

Quick Start (Windows 10/11 PowerShell)

# Ensure Python 3.11+ from Microsoft Store or python.org is installed
git clone https://github.com/SashaYerashoff/webtool-mcp.git
cd webtool-mcp
python -m venv .venv
. .venv/Scripts/Activate.ps1
pip install --upgrade pip
pip install -r requirements.txt
python app.py  # http://localhost:5000

If Windows Firewall prompts, allow local network access (loopback is enough for LM Studio).

Install as a dependency (optional)

You can also just install straight from Git:

pip install git+https://github.com/SashaYerashoff/webtool-mcp.git

Then run (clone not strictly required, but the above is simplest for development):

python -m webtool_mcp  # (future packaging plan) – for now use app.py directly

Running Behind a Different Port

Change the app.run(... port=5000) line or export PORT and modify code to read it (not yet implemented). If you change the port you must update LM Studio config accordingly.

LM Studio Integration

1. Start this server locally: python app.py → http://localhost:5000/mcp
2. In LM Studio (0.3.17+ with MCP support):
   • Open: Program → Install → (scroll) Edit MCP Configuration (or locate mcp.json).
3. Add / merge the entry:

{
  "mcpServers": {
    "webtool-mcp": {
      "url": "http://localhost:5000/mcp"  // or your LAN IP
    }
  }
}

4. Save and click Reload MCPs (or restart LM Studio).
5. Open a chat with your local model. The tools should appear in the UI or be callable automatically.

Verifying from LM Studio

Ask the model: "List the tools you have." It should respond (or you can request a tools/list internally) with the tools defined above.

System Prompt

See sysprompt.md for the fully maintained prompt (ranking heuristics, fallbacks, efficiency rules). Minimal inline guidance:

Broad topic → web_search (multi) → choose URL → fetch_url(mode='outline') → pick chunk_id OR link_id → summarize with cited sources before deeper retrieval.

Manual Testing (curl examples)

Fetch outline only (cheap): Web search (multi-engine aggregate):

curl -s -X POST http://localhost:5000/mcp \
  -H 'Content-Type: application/json' \
  -d '{"name":"web_search","arguments":{"query":"open source vector databases","engine":"multi","engines":["duckduckgo","bing"],"max_results":5}}'

curl -s -X POST http://localhost:5000/mcp \
  -H 'Content-Type: application/json' \
  -d '{"name":"fetch_url","arguments":{"url":"https://example.com","mode":"outline"}}' | jq -r '.result.content[0].text' | head

Fetch a specific section after outline (example sec-2):

curl -s -X POST http://localhost:5000/mcp \
  -H 'Content-Type: application/json' \
  -d '{"name":"fetch_url","arguments":{"url":"https://example.com","chunk_id":"sec-2"}}'

Follow a link from outline (L5):

curl -s -X POST http://localhost:5000/mcp \
  -H 'Content-Type: application/json' \
  -d '{"name":"fetch_url","arguments":{"url":"https://example.com","link_id":"L5"}}'

Wikipedia summary:

curl -s -X POST http://localhost:5000/mcp \
  -H 'Content-Type: application/json' \
  -d '{"name":"search_wikipedia","arguments":{"query":"Python (programming language)"}}'

Latvian news:

Google Custom Search (Optional)

To enable the google_cse engine inside web_search, export environment variables prior to launch:

export GOOGLE_API_KEY="your_api_key"
export GOOGLE_CSE_ID="your_cse_id"   # Programmable Search Engine ID
python app.py

Then call (example):

{"name":"web_search","arguments":{"query":"vector db benchmarks","engine":"google_cse","max_results":5}}

Search Strategy & Fallbacks

• Ambiguous / exploratory: web_search with engine="multi" and engines=["duckduckgo","bing"].
• Weak results: refine query (add distinguishing noun, remove stopwords) or switch engine.
• After outline: rank links (authority > freshness > relevance) and follow only one link_id per step.
• Avoid re-fetching the same outline unless stale.
• Parsing issue: retry once with mode='outline' then choose alternate source.

JSON-RPC Tool Call Examples

Payloads MCP client sends (wrapping examples):

{"jsonrpc":"2.0","id":1,"method":"tools/list"}
{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"fetch_url","arguments":{"url":"https://example.com","mode":"outline"}}}
{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"fetch_url","arguments":{"url":"https://example.com","chunk_id":"sec-2"}}}
{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"fetch_url","arguments":{"url":"https://example.com","link_id":"L5"}}}
{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"web_search","arguments":{"query":"open source vector database","engine":"multi","engines":["duckduckgo","bing"],"max_results":5}}}
{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"web_search","arguments":{"query":"vector db benchmarks","engine":"google_cse","max_results":5}}}
{"jsonrpc":"2.0","id":7,"method":"tools/call","params":{"name":"latvian_news","arguments":{}}}
{"jsonrpc":"2.0","id":8,"method":"tools/call","params":{"name":"search_wikipedia","arguments":{"query":"Milvus"}}}
{"jsonrpc":"2.0","id":9,"method":"tools/call","params":{"name":"stock_quotes","arguments":{"symbols":"AAPL MSFT"}}}

curl -s -X POST http://localhost:5000/mcp \
  -H 'Content-Type: application/json' \
  -d '{"name":"latvian_news"}'

JSON-RPC Notes

LM Studio now uses JSON-RPC 2.0 methods like initialize, tools/list, and tools/call. This server supports:

• POST /mcp body: { "jsonrpc":"2.0","id":1,"method":"tools/list" }
• Tool call shape: { "jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"fetch_url","arguments":{"url":"https://example.com","mode":"outline"}} }

Legacy (non JSON-RPC) payloads with {"name": "fetch_url", "arguments": {...}} are still handled for quick manual curl tests.

Production & Security Considerations

This is a demo / local helper:

• No auth, rate limiting, or HTTPS.
• User-provided URLs are fetched server-side; avoid exposing it publicly without safeguards.
• Respect target site robots.txt / Terms of Service.
• Consider caching, backoff and user-agent tuning for high volume usage.
• Add an allowlist if you embed this in an automated system.

Roadmap / Ideas

• Package as an installable module with console entry point.
• Add configurable max tokens / chunk merging.
• Optional vector store for revisiting context across sessions.
• Better error normalization & retry policy.

License

Licensed under the MIT License – see LICENSE.

Dependency license compatibility (all permissive / MIT‑compatible):

• Flask (BSD-3-Clause)
• Requests (Apache-2.0)
• BeautifulSoup4 / bs4 (MIT)
• duckduckgo-search (MIT)

No copyleft or restrictive GPL dependencies are included, so MIT distribution is appropriate.

---

Happy browsing with your local models! 🧭