Enterprise Knowledge Server

Unstructured Data Pipeline & Remote MCP Server — a production-ready enterprise document knowledge base. Ingests DOCX / PDF / PPTX, parses with Docling, cleans and chunks with metadata, indexes into a hybrid search store, and exposes a Remote MCP Server for Claude Desktop and other agents.

Status

Built incrementally, one step at a time (see CLAUDE.md for the full plan).

• Step 1 — Project Bootstrap: FastAPI + FastMCP + Docker + Pytest
• Step 2 — Document Upload: POST/GET /documents + persistent catalogue (DOCX/PDF/PPTX); upload now auto-runs the full pipeline and indexes immediately (no restart)
• Step 3 — Docling Parser: Docling -> structured ParsedDocument (headings/text/tables/figures, page & slide provenance)
• Step 4 — Cleaning Pipeline: strip repeated headers/footers, page numbers, empty/symbol-only noise (structure preserved)
• Step 5 — Metadata-aware Chunking: semantic chunks (section/table/figure) with full metadata; no fixed-width cuts
• Step 6 — Chroma Indexing: BGE dense embeddings into embedded persistent Chroma (index/search/get/delete)
• Step 7 — Hybrid Retrieval: dense (BGE) + sparse (BM25) fused with RRF; mixed CN/EN tokenizer
• Step 8 — MCP Tools: search_documents / list_documents / get_document / get_chunk on FastMCP
• Step 9 — MCP Resources: documents://all and documents://{document_id}
• Step 10 — Integration Test: end-to-end MCP protocol test (Client -> server -> tool/resource) + runnable client demo

Architecture (target)

DOCX / PDF / PPTX
    -> Docling Parser
    -> Cleaning Pipeline
    -> Metadata-aware Chunking
    -> Hybrid Search Index (BGE dense + BM25 sparse, Chroma)
    -> Remote MCP Server (FastMCP)
    -> Claude Desktop

Tech Stack

Area	Choice
Language	Python 3.11
API	FastAPI
Parsing	Docling
Search	Hybrid Retrieval
Dense Retrieval	BGE Embedding
Sparse Retrieval	BM25
Vector DB	Chroma (embedded)
MCP Framework	FastMCP
Deployment	Docker
Testing	Pytest

Quick Start (local)

This repo ships a pre-created virtual environment (kb_mcp_env/, Windows).

# Install dependencies (incl. dev/test extras)
kb_mcp_env\Scripts\python.exe -m pip install -e ".[dev]"

# Run the tests
kb_mcp_env\Scripts\python.exe -m pytest -q

# Run the server
kb_mcp_env\Scripts\python.exe -m uvicorn app.main:app --reload

• Health check: http://localhost:8000/health
• Remote MCP endpoint: http://localhost:8000/mcp

Run with Docker

cp .env.example .env   # optional
docker compose up --build

Brings up the app on port 8000. Chroma runs embedded in-process (no separate service); its data persists in the chroma_storage Docker volume.

Deploy to Zeabur (public URL)

Zeabur auto-detects the Dockerfile. This is a heavy ML service (BGE-M3 ~2.2GB embedding model + Docling/Torch + Chroma), so it needs a plan with enough RAM (recommended ≥ 4 GB) and a persistent Volume; the first request is slow while models download.

1. Push this repo to GitHub.
2. Zeabur → Create Project → Add Service → Deploy from GitHub → pick this repo (it builds from the Dockerfile).
3. Volumes → add a volume mounted at /data (keeps the index, uploaded files, and downloaded models across redeploys).
4. Variables → set:

   UPLOAD_DIR=/data/uploads
   CHROMA_PERSIST_DIR=/data/chroma_storage
   HF_HOME=/data/hf_cache
   

   (The platform injects $PORT; the Dockerfile already honours it.)
5. Networking → Generate Domain → https://<your>.zeabur.app.

After deploy:

• Health check: https://<your>.zeabur.app/health
• Remote MCP endpoint: https://<your>.zeabur.app/mcp
• API docs (Swagger UI): https://<your>.zeabur.app/docs

Point Claude Desktop / Claude Code at the /mcp URL to use the deployed server.

Parsing & OCR

提醒：OCR 是在解析時對每張圖跑，26 張圖會增加數十秒解析時間。若某類文件不需要，可在 .env 設 OCR_IMAGES=false 關閉。

提醒：預設 embedding 模型為 BAAI/bge-m3（多語，適合中英混雜，1024 維、約 2.2GB，首次會下載）。若只需英文且要更輕量，可在 .env 設 EMBEDDING_MODEL=BAAI/bge-small-en-v1.5。切換模型若維度不同，需先清空 chroma_storage 重新索引。

Example Queries (target MCP tools)

What is the yield improvement plan?
Show me the KPI table from Q4 report.
Summarize slide 5.

Add a document (auto-indexed, no restart)

POST /documents saves the file and runs the full pipeline (Docling parse -> clean -> metadata-aware chunk -> BGE embed -> Chroma index) in the same process, then refreshes BM25. Because it shares the MCP server's vector-store/retriever singletons, the document is searchable over MCP immediately — no restart needed.

# server running on :8000
curl.exe -X POST http://127.0.0.1:8000/documents -F "file=@E:\path\to\report.pdf"
# -> 201 {"document_id": "...", "status": "indexed", "num_chunks": 42, ...}

The response carries status (indexed, or failed with HTTP 500 if parsing errors — the file is still recorded) and num_chunks. The call blocks until indexing finishes (Docling/OCR/embedding can take tens of seconds for large or image-heavy files). scripts/ingest_file.py shares the same pipeline for command-line ingestion.

Verify the MCP Server (client demo + server log)

Start the server, then drive it over the real MCP protocol with the bundled client demo:

# Terminal 1 — run the server
kb_mcp_env\Scripts\python.exe -m uvicorn app.main:app --host 127.0.0.1 --port 8000

# Terminal 2 — connect a remote MCP client and run the example queries
kb_mcp_env\Scripts\python.exe tests\mcp_client_demo.py
# ...or pass your own query:
kb_mcp_env\Scripts\python.exe tests\mcp_client_demo.py "yield improvement plan"

The demo connects (Client -> MCP Server -> search_documents -> Result), lists the server's tools/resources, reads documents://all, and prints the retrieved chunks. Meanwhile the server console logs each invocation:

INFO:app.mcp_server:MCP tool invoked: search_documents | query='...' top_k=3
INFO:app.mcp_server:search_documents retrieved 3 chunk(s): [...]

The hermetic equivalent (no running server, isolated temp index) is the pytest integration test:

kb_mcp_env\Scripts\python.exe -m pytest tests\test_mcp_integration.py -q

AI Workflow

This project is developed with an AI-only workflow (Claude Code + MCP). Each development step follows: plan -> implement -> review -> test, with a dedicated commit per step. See CLAUDE.md for the step-by-step record.