RAG Notes Search Server

A semantic search system over personal study notes, exposed as an MCP (Model Context Protocol) tool that Claude Desktop can call mid-conversation.

What it does

Ask Claude Desktop anything about your notes and it automatically:

1. Decides whether to search your notes based on the question
2. Calls the search_notes tool with a semantic query
3. Retrieves the most relevant note content with similarity scores
4. Synthesizes an answer grounded in your actual notes

Example queries that work:

• "What have I written about machine learning pipelines?"
• "Search my notes for how vector databases work"
• "What do my notes say about GCP Cloud Run?"

Architecture

Claude Desktop (MCP Client) ↓ stdio MCP Server (mcp_server.py) ↓ search_notes() tool ↓ Chroma Vector DB (local) ↓ fastembed (all-MiniLM-L6-v2, ONNX)

Tech stack

• MCP — Anthropic's Model Context Protocol (v1.28.1) for tool exposure
• Chroma — local vector database storing note embeddings
• fastembed — lightweight ONNX-based embedding model (no PyTorch dependency)
• sentence-transformers/all-MiniLM-L6-v2 — embedding model, 384 dimensions
• Docker — containerized for reproducibility, built for linux/amd64
• GCP — Container Registry hosts the image, Cloud Run deployment planned

Key technical decisions

Why fastembed over sentence-transformers? sentence-transformers pulls in PyTorch (~2GB). fastembed uses ONNX runtime (~200MB), making Docker builds 10x faster and the image significantly smaller.

Why a similarity threshold? Without a threshold, vector search always returns something even when nothing is relevant — this is how RAG systems silently hallucinate. A threshold of 0.25 means the system returns "no relevant notes found" rather than a low-confidence garbage result.

Why absolute paths in the MCP server? Claude Desktop spawns the MCP server as a subprocess with an unpredictable working directory. Relative paths like ./chroma_db break silently. Absolute paths are required for reliable subprocess execution.

Why stderr for all logging? MCP uses stdout as a JSON wire protocol. Any print() to stdout corrupts the MCP message stream. All logging goes to stderr which Claude Desktop reads separately via the log file.

Setup

Prerequisites

• Python 3.11
• Claude Desktop
• conda or venv

Install

conda create -n rag_demo python=3.11
conda activate rag_demo
pip install chromadb==1.5.9 sentence-transformers mcp fastembed numpy==1.26.4

Index your notes

Add .txt files to the notes/ folder, then run the indexing notebook:

jupyter notebook demo.ipynb

Run all cells — this embeds your notes into Chroma.

Connect to Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "notes-search": {
      "command": "/opt/anaconda3/envs/rag_demo/bin/python",
      "args": ["/absolute/path/to/rag_demo/mcp_server.py"],
      "cwd": "/absolute/path/to/rag_demo"
    }
  }
}

Restart Claude Desktop. Look for the tools icon in the chat input.

Docker

# Build for linux/amd64
docker buildx build --platform linux/amd64 -t rag-demo .

# Run with local chroma_db mounted
docker run --rm \
  -v /absolute/path/to/chroma_db:/app/chroma_db \
  rag-demo

Bugs fixed during development

Bug	Cause	Fix
Read-only file system	Relative ./chroma_db path breaks in subprocess	Use absolute path
Unexpected token 'L' is not valid JSON	print() polluting MCP stdout wire	Route all logs to stderr
cached_download ImportError	sentence-transformers version conflict	Switched to fastembed
np.float_ AttributeError	NumPy 2.0 removed deprecated types	Pinned numpy==1.26.4
no such column: collections.topic	Chroma version mismatch between dev and Docker	Matched versions exactly

Planned improvements

• SSE/HTTP transport for Cloud Run deployment
• Second MCP tool: search_web() using a free news API
• Daily ingestion job via Cloud Scheduler
• Confidence score displayed in Claude's response