Crawl4ai2vectordb

MCP server for crawling web documentation and storing it in a Supabase vector database. Credits: craw4ai and Cole Medin's mcp server.

Responsibility: ingestion only — crawl, chunk, embed, store. No search, no reranking, no Neo4j.

Tools

Tool	Description
crawl_single_page	Crawl one URL and store its content
smart_crawl_url	Auto-detect URL type (sitemap / txt / webpage) and crawl accordingly
get_available_sources	List all sources stored in the database
delete_source	Delete a source and all its content from the database

Setup

cp .env.example .env
# fill in OPENAI_API_KEY, MODEL_CHOICE, SUPABASE_URL, SUPABASE_SERVICE_KEY
uv sync

Running

stdio (recommended — Claude Code manages the process):

Add to your MCP config:

{
  "mcpServers": {
    "crawl4ai2vectordb": {
      "type": "stdio",
      "command": "uv",
      "args": ["run", "python", "-u", "src/server.py"],
      "cwd": "/path/to/mcp_crawl4ai2vectordb"
    }
  }
}

or

{
  "mcpServers": {
    "crawl4ai2vectordb": {
      "type": "sse",
      "url": "http://localhost:8051/sse"
    }
  }
}

SSE (manual startup, for multi-client use):

TRANSPORT=sse uv run python -u src/server.py

Optional features

Env var	Default	Effect
USE_CONTEXTUAL_EMBEDDINGS	false	Prepend LLM-generated context to each chunk before embedding (improves retrieval accuracy, costs more)
USE_AGENTIC_RAG	false	Extract code blocks and store them separately in code_examples table