Local Web Search Service

Self-hosted web search + page fetch for LoreWeave's glossary deep-research feature:

• Search — SearXNG primary (OSS metasearch — no API key, no per-query cost), with an optional Tavily fallback so a research turn never comes back empty. See Search reliability.
• Fetch — backed by a Scrapling sidecar (browser rendering + anti-bot), with an in-process HTTP fallback.

Capabilities are exposed over two surfaces, both calling the same orchestrators:

Capability	HTTP	MCP tool
Search	POST /search (Tavily-compatible)	web_search
Fetch one page	POST /fetch	fetch_page
Fetch many pages	POST /fetch/bulk	fetch_pages

The HTTP POST /search is the LoreWeave contract; the MCP surface is a bonus for direct agent (Cursor/Claude) use. LoreWeave never uses MCP — it calls POST /search. (A web_fetch LoreWeave consumer is not built yet — fetch is used directly via HTTP/MCP for now.)

Contracts implemented: §3/§8 (search) and §10 (fetch) of lore-weave-security/docs/04_integration/2026-06-21-web-search-service-integration.md.

---

Architecture

LoreWeave ──BYOK──▶ provider-registry ──POST /search──▶┐
                                                        ├─▶ web-search shim (this repo)
Cursor/agent ── MCP /mcp · HTTP /search /fetch ────────▶┘     │
                                                              ├─▶ SearXNG       (search; always)
                                                              ├─▶ trafilatura   (search advanced-extract)
                                                              └─▶ Scrapling MCP (fetch: browser/stealth)
                                                                    └ http fallback when sidecar absent

• app/service.py — search orchestrator: clamp → SearXNG → map → (advanced) enrich → answer.
• app/searxng_client.py — SearXNG JSON client + mapping to the §3 shape (http-only URLs, deduped).
• app/fetch_service.py — fetch orchestrator: mode dispatch (http/browser/stealth/auto) + auto-escalation.
• app/scrapling_client.py — MCP client to the Scrapling sidecar.
• app/api.py — POST /search, POST /fetch, POST /fetch/bulk, GET /health, GET /ready.
• app/mcp_server.py — FastMCP tools: web_search, fetch_page, fetch_pages (Streamable HTTP).
• app/main.py — combines everything into one ASGI app (HTTP routes + mounted /mcp).

---

Run

Docker (recommended — ships SearXNG too)

copy .env.example .env      # optional: set WEB_SEARCH_SECRET, etc.
docker compose up -d --build

• Shim: http://localhost:15487 (/search, /health, /ready, /mcp)
• SearXNG: internal to the compose network (publish port 8080 only for debugging).

Local (Python) — SearXNG elsewhere

pip install -r requirements.txt
$env:SEARXNG_URL = "http://localhost:8080"   # a SearXNG with JSON format enabled
uvicorn app.main:app --host 0.0.0.0 --port 15487

---

Try it

.\scripts\smoke.ps1                              # health + a sample /search

curl -X POST http://localhost:15487/search \
  -H "Content-Type: application/json" \
  -d '{"query":"Nezha 哪吒 deity","max_results":5,"include_answer":true}'

Response (exact keys LoreWeave parses):

{
  "query": "Nezha 哪吒 deity",
  "answer": "Nezha is a protection deity ...",
  "results": [
    { "title": "Nezha — Wikipedia", "url": "https://en.wikipedia.org/wiki/Nezha",
      "content": "Nezha is a protection deity ...", "score": 0.95 }
  ]
}

Fetch a specific page

# auto = fast HTTP first, escalate to stealth (Cloudflare/anti-bot) if blocked
curl -X POST http://localhost:15487/fetch \
  -H "Content-Type: application/json" \
  -d '{"url":"https://en.wikipedia.org/wiki/Nezha","mode":"auto","format":"markdown","max_chars":8000}'

{
  "url": "https://en.wikipedia.org/wiki/Nezha",
  "final_url": "https://en.wikipedia.org/wiki/Nezha",
  "status": 200, "title": "Nezha", "content": "# Nezha\n\n...",
  "content_format": "markdown", "length": 5234, "engine": "http", "error": null
}

• mode: http (fast) · browser (JS render) · stealth (anti-bot) · auto (default).
• format: markdown (default) · text · html.
• Bulk: POST /fetch/bulk with {"urls":[...]} → {"results":[...]} (per-URL error isolation, capped at 10).

MCP client config

{
  "mcpServers": {
    "local-web-search": { "type": "http", "url": "http://localhost:15487/mcp" }
  }
}

Tools:

• web_search(query, max_results=5, search_depth="basic", include_answer=false)
• fetch_page(url, mode="auto", format="markdown", max_chars=8000, css_selector=None)
• fetch_pages(urls, mode="auto", format="markdown", max_chars=8000, css_selector=None)

---

Configuration (env)

Var	Default	Meaning
WEB_SEARCH_SECRET	(empty)	Bearer required on /search. Empty ⇒ keyless (Authorization ignored).
SEARXNG_URL	http://localhost:8080	SearXNG base URL (compose: http://searxng:8080).
SEARXNG_ENGINES	(empty)	Comma-separated engine subset; empty ⇒ SearXNG defaults.
SEARXNG_TIMEOUT_S	15	SearXNG request timeout.
TAVILY_API_KEY	(empty)	Enables the Tavily fallback. Empty ⇒ pure-SearXNG mode.
SEARCH_FALLBACK_ENABLED	true	Master toggle for the fallback (needs a key too).
TAVILY_TIMEOUT_S	20	Tavily request timeout.
ENABLE_EXTRACT	true	Full-page extract on search_depth="advanced".
EXTRACT_TOP_N	3	How many top results to extract.
EXTRACT_MAX_CHARS	4000	Cap on extracted content length.
ENABLE_FETCH	true	Enable the /fetch + /fetch/bulk API and fetch_* MCP tools.
SCRAPLING_MCP_URL	http://scrapling:8000/mcp	Scrapling sidecar MCP URL. Empty ⇒ disable Scrapling (http/auto via in-process fallback; browser/stealth → 502).
FETCH_DEFAULT_MODE	auto	Default fetch mode.
FETCH_MAX_CHARS	8000	Default cap on fetched content.
FETCH_BULK_MAX_URLS	10	Max URLs per /fetch/bulk.
PORT	15487	Host-published port (container-internal stays 8090 under compose).

SearXNG must have formats: [html, json] (set in searxng/settings.yml) — the shim calls /search?format=json. Without it you get an upstream error with a clear hint.

---

Register in LoreWeave (BYOK web_search)

1. Provider credential — provider_kind = web_search, endpoint_base_url = http://<host>:15487, secret = WEB_SEARCH_SECRET (empty for keyless).
2. User model — provider_model_name = searxng-default, capability_flags = {"web_search": true} (strict), pricing = {"input_per_mtok":0,"output_per_mtok":0}.
3. is_active = true (+ is_favorite to make it the preferred web_search model).

---

Test

pip install -r requirements.txt pytest
pytest -q          # contract tests, no live SearXNG needed (SearXNG is mocked)

---

Search reliability: two-tier backend

SearXNG is a metasearch scraper, not its own index. It proxies Google/ Bing/Brave/etc., and those engines actively CAPTCHA and rate-limit self-hosted IPs. It is self-hosted but not self-sufficient — that is inherent, not a bug in this service.

Two things make it reliable here:

1. A wide engine pool (searxng/settings.yml). SearXNG's stock config enables only four real web engines (brave, duckduckgo, google, startpage) — and all four block self-hosted instances. When they die at once you get zero results. We enable ~10 engines instead, including mojeek and mwmbl (independent crawlers that don't block servers) plus bing, yahoo, qwant, yep, presearch, duckduckgo web. Measured on the same 8-query benchmark: 2/8 → 8/8, and 10/10 under a rapid burst.

2. A Tavily fallback (optional). Even a wide pool is best-effort. If SearXNG returns zero results or errors, the shim calls Tavily so the turn still succeeds. The response's provider field says which backend served it.

SearXNG has results  →  return them          (Tavily never called ⇒ $0)
SearXNG 0 results    →  try Tavily; still nothing ⇒ empty 200 (valid "not found")
SearXNG errors       →  try Tavily; if it also fails ⇒ surface SearXNG's 502/429

Set TAVILY_API_KEY in .env to enable (free tier: 1,000 credits/month, basic search = 1 credit). Leave it empty for pure-SearXNG mode. Because the fallback only fires when SearXNG comes up empty, normal operation costs nothing.

Note: Brave's Search API removed its free tier in Feb 2026 (now ~$5/1k queries), which is why Tavily is the fallback of choice here.

---

Troubleshooting: ERROR lines in the searxng container logs

Seeing things like SearxEngineCaptchaException, google ... IndexError, Too many request (suspended_time=180), or engine timeout? That's normal and not a fault in this service. They come from SearXNG's per-engine scrapers, not the shim (the web-search container logs stay clean).

• Public engines (Google/DuckDuckGo/Brave/…) routinely block, CAPTCHA, or rate-limit a single server IP. SearXNG logs each failed engine at ERROR.
• SearXNG queries many engines and merges whatever succeeds, so a few engines failing still returns a full result set — that's the resilience.
• If you get zero results, the cause is almost always too few enabled engines, not "SearXNG is broken." The stock config enables only four real web engines and all four block self-hosted IPs. See Search reliability — searxng/settings.yml widens the pool. Removing engines to quiet the logs makes this worse.
• Tor engines (ahmia, torch) are removed outright: they need a Tor proxy we don't run, so they only fail to load and never return anything.
• Bursty testing trips rate limits (engines auto-suspend ~180s then recover). At normal deep-research volume you'll rarely see them.

Notes

• Returned text is untrusted. LoreWeave neutralizes it (INV-6) regardless; this service returns clean text but is never trusted.
• Stateless — no GPU, no model lifecycle (unlike rerank/STT/TTS).
• Graceful — backend down/slow ⇒ 502 {"error":"upstream"} / 429 {"error":"rate_limited"}; LoreWeave degrades the research turn to "search unavailable" and never blocks the chat.