AgentWebSearch

Local WebSearch without API keys + MCP Server

Need WebSearch for your local LLM (Ollama, LM Studio) or Claude Code? No API keys needed - just Chrome.

How It Works

This tool launches REAL Chrome browsers - not headless, not simulated.

It opens actual Chrome windows and controls them via CDP (Chrome DevTools Protocol). This is why it can bypass bot detection and CAPTCHA - because it IS a real browser.

┌─────────────────────────────────────────────────────────┐
│  Your Desktop                                           │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐       │
│  │ Chrome:9222 │ │ Chrome:9223 │ │ Chrome:9224 │       │
│  │   Naver     │ │   Google    │ │   Brave     │       │
│  │  (search)   │ │  (search)   │ │  (search)   │       │
│  └──────┬──────┘ └──────┬──────┘ └──────┬──────┘       │
│         │               │               │               │
│         └───────────────┼───────────────┘               │
│                         │ CDP (WebSocket)               │
│                    ┌────┴────┐                          │
│                    │   MCP   │                          │
│                    │ Server  │                          │
│                    └────┬────┘                          │
│                         │                               │
│                    Claude Code                          │
└─────────────────────────────────────────────────────────┘

Why real browsers?

• No API keys needed (you're just browsing)
• No rate limits (normal browser behavior)
• CAPTCHA resistant (real browser fingerprint)
• Login sessions persist (cookies saved)
• Korean portals work (Naver requires real browser)

Extensible to other sites

• Currently configured for search portals (Naver, Google, Brave)
• Can be extended to other sites by adding portal config
• Each site needs its own Chrome instance and session

Key Benefits

Feature	Tavily/Brave API	SearXNG (OpenClaw)	AgentWebSearch-MCP
API Key	Required	Not needed	Not needed
Cost	Paid/Limited	Free	Free
Setup	Get API key	Host SearXNG server	Just install Chrome
Bot Detection	N/A	Easily blocked	Bypassed (real browser)
Korean Portals	Limited	No Naver	Naver supported
MCP Support	❌	❌	✅ Built-in

Features

• No API keys - Direct Chrome CDP (DevTools Protocol) control
• MCP Server - Use as Claude Code / Cursor / LM Studio tool
• AgentCPM-Explore - Search-optimized 4B model from OpenBMB/THUNLP (optional)
• Parallel search - 3 Chrome instances for Naver/Google/Brave simultaneously
• Bot detection bypass - Session persistence + stealth flags
• CAPTCHA resistant - Real browser sessions avoid most CAPTCHA challenges

Installation

1. Install dependencies

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Start Chrome (debugging mode)

python chrome_launcher.py start   # Start 3 Chrome instances
python chrome_launcher.py status  # Check status
python chrome_launcher.py stop    # Stop all

3. Login & Sync (optional but recommended)

By default, Chrome instances use temporary profiles (/tmp/). For persistent login:

1. Open CDP Chrome (after chrome_launcher.py start)
2. Sign in to Google in the Chrome window
3. Enable sync (Settings → Sync)

Benefits:

• OAuth login ("Sign in with Google") works automatically
• Saved passwords auto-fill
• Better search results (personalized)
• Sessions persist across restarts

Note: To make profiles permanent, change profile paths in chrome_launcher.py from /tmp/chrome-*-profile to ~/.config/chrome-cdp/*

MCP Server

Use web search as an MCP tool in Claude Code, Cursor, LM Studio, or OpenClaw/Moltbot.

Quick Start

# Start Chrome first
python chrome_launcher.py start

# Run MCP server (stdio mode)
python mcp_server.py

Claude Code Registration

claude mcp add agentwebsearch -s user -- python /path/to/mcp_server.py

Claude Code settings.json

{
  "mcpServers": {
    "agentwebsearch": {
      "command": "python",
      "args": ["/path/to/AgentWebSearch-MCP/mcp_server.py"],
      "env": {}
    }
  }
}

OpenClaw/Moltbot Registration

# Add MCP to OpenClaw
openclaw mcp add agentwebsearch -- python /path/to/mcp_server.py

# Or add to ~/.openclaw/config.json

SSE Mode (HTTP Server)

python mcp_server.py --sse --port 8902

Available MCP Tools

Tool	Description	LLM Required
web_search	Search Naver/Google/Brave in parallel	No
fetch_urls	Fetch webpage content from URLs	No
smart_search	Search + auto-fetch with depth control	No
get_search_status	Check search progress and get partial results	No
cancel_search	Cancel ongoing search and get partial results	No
agentcpm	Agentic search with AgentCPM-Explore (SGLang)	Yes

Tool Parameters

smart_search

Parameter	Values	Description
query	string	Search query (required)
depth	simple	Snippets only (fast)
	medium	Fetch top 5 URLs (default)
	deep	Fetch top 15 URLs (slow)
portal	all/naver/google/brave	Search portal

agentcpm

Uses AgentCPM-Explore model (4B, OpenBMB/THUNLP) via SGLang to plan search queries, execute searches, and generate answers.

Parameter	Values	Description
query	string	Search query (required)
depth	simple/medium/deep	Search depth (default: medium)
confirm	boolean	Confirm to proceed if SGLang not running

Requirements:

• SGLang server running on port 30001
• AgentCPM-Explore model loaded

First time setup: Model loading takes ~30-45 seconds. Use smart_search if you don't have SGLang set up.

get_search_status / cancel_search

Partial results support: When search takes too long, you can:

1. Call get_search_status to check progress and see results collected so far
2. Call cancel_search to stop the search and return partial results

Tool	Description
get_search_status	Returns: status, progress %, elapsed time, partial search results, partial fetched contents
cancel_search	Cancels ongoing search and returns all partial results collected

Why AgentCPM-Explore? Trained specifically for search agent tasks by OpenBMB/THUNLP. Generates diverse queries and handles tool calling better than general-purpose models.

Example Usage

"Search latest AI news" → smart_search tool
"Deep search about GPT-5" → smart_search(depth="deep")
"Use AgentCPM for AI news" → agentcpm(query="AI news")

Standalone Agent (CLI)

Run search agent directly from command line.

# Default: SGLang backend
python search_agent.py "search query"

# Search depth
python search_agent.py "query" --depth simple   # snippets only (fast)
python search_agent.py "query" --depth medium   # fetch top 5 URLs (default)
python search_agent.py "query" --depth deep     # fetch all URLs (slow)

# Interactive mode
python search_agent.py -i

# CLI supports multiple backends (--llm ollama/lmstudio/openai)
python search_agent.py --list-backends

AgentCPM-Explore Setup (for agentcpm tool)

AgentCPM-Explore is a 4B parameter model from OpenBMB/THUNLP specifically trained for search agent tasks:

• Automatically generates diverse search queries (Korean/English, multiple perspectives)
• Optimized for tool calling (search, fetch_url)
• Based on Qwen3-4B-Thinking

# 1. Install SGLang (CUDA required)
pip install sglang[all]

# 2. Download AgentCPM-Explore model (~8GB)
# https://huggingface.co/openbmb/AgentCPM-Explore

# 3. Start server
MODEL_PATH=/path/to/AgentCPM-Explore ./start_sglang.sh

Note: The agentcpm MCP tool exclusively uses SGLang + AgentCPM-Explore. For other LLM backends, use smart_search or the CLI (search_agent.py --llm ollama).

Architecture

User Query
    |
MCP Server (mcp_server.py)
+-- web_search      → CDP Search only
+-- smart_search    → CDP Search + URL Fetch
+-- agentcpm        → SGLang + AgentCPM-Explore + CDP Search
    |
CDP Search (parallel)
+-- Chrome:9222 → Naver
+-- Chrome:9223 → Google
+-- Chrome:9224 → Brave
    |
Final Answer + Sources

File Structure

AgentWebSearch-MCP/
├── mcp_server.py         # MCP server (4 tools)
├── search_agent.py       # Standalone CLI agent
├── cdp_search.py         # CDP parallel search
├── chrome_launcher.py    # Chrome instance manager
├── llm_adapters/         # LLM adapters (CLI use)
│   ├── base.py           # Common interface
│   ├── sglang_adapter.py
│   ├── ollama_adapter.py
│   ├── lmstudio_adapter.py
│   └── openai_adapter.py
└── start_sglang.sh       # SGLang server startup

Performance

Mode	Time	Tokens
simple	~35s	~3K
medium	~50s	~17K
deep	~170s	~77K

Requirements

• Python 3.10+
• Chrome/Chromium
• For agentcpm tool: SGLang + AgentCPM-Explore model

License

MIT License

Contributing

Portal addition PRs welcome! See PORTAL_CONFIG in cdp_search.py.

---

Co-Authored-By: inchul [email protected] Co-Authored-By: Claude [email protected]