Arxiv Personalized

A full-pipeline service that transforms Chinese queries into ranked arXiv paper recommendations, exposed as an MCP (Model Context Protocol) endpoint for AI agent integration.

Chinese Query → EN Translation → Keyword Extraction → arXiv Search → Semantic Reranking → MCP Endpoint

Project Structure

arxiv_mcp/
├── big_model.py         # Talker class: translation + keyword extraction (DeepSeek API)
├── arxiv_server.py      # FastAPI service (/score + /query, port 5216)
├── mcp_wrapper.py       # fastapi-mcp wrapper exposing API as MCP SSE endpoint (port 8000)
├── quick_test.py        # Quick functional test (no GPU / reranker required)
├── requirements.txt
└── README.md

Quick Start

1. Install Dependencies

pip install -r requirements.txt

GPU users: install a CUDA-compatible PyTorch from pytorch.org for faster reranking.

2. Configure API Key

This project uses the DeepSeek API (OpenAI-compatible) for translation and keyword extraction. Set your API key via environment variable:

export DEEPSEEK_API_KEY="sk-xxxxxxxxxxxxxxxx"

Alternatively, you can pass the key directly when creating a Talker instance:

from big_model import Talker
talker = Talker(api_key="sk-xxxxxxxxxxxxxxxx")

3. Quick Functional Test (No GPU)

python quick_test.py

Validates translation and arXiv search without loading the reranker model.

4. Start the API Service

python arxiv_server.py
# or
uvicorn arxiv_server:app --host 127.0.0.1 --port 5216

Swagger docs: http://127.0.0.1:5216/docs

5. Start the MCP Service

python mcp_wrapper.py

MCP SSE endpoint: http://127.0.0.1:8000/mcp

API Reference

POST /score — Relevance Scoring

Computes normalized relevance scores (0–1) for a query against multiple passages using BAAI/bge-reranker-v2-m3.

Request:

{
  "query": "position embedding in transformer",
  "passages": [
    "We propose a novel rotary position embedding...",
    "This paper introduces a new attention mechanism..."
  ]
}

Response:

{
  "scores": [0.9821, 0.6712]
}

POST /query — Personalized Paper Search

End-to-end pipeline: Chinese query → translation → keyword extraction → arXiv search → reranking.

Request:

{
  "query": "注意力机制中的旋转位置编码",
  "max_results": 5
}

Response:

{
  "query_original": "注意力机制中的旋转位置编码",
  "query_english": "Rotary Position Embedding in Attention Mechanisms",
  "query_terms": "Rotary Position Embedding, Attention Mechanism, Transformer",
  "papers_sorted": [
    {
      "paper_id": "2410.12345",
      "title": "RoPE: Rotary Position Embedding for Transformers",
      "summary_english": "We propose a novel method...",
      "summary_chinese": "我们提出了一种新颖的方法...",
      "authors": ["Author A", "Author B"],
      "pdf_url": "https://arxiv.org/pdf/2410.12345",
      "links": ["https://arxiv.org/abs/2410.12345"],
      "relevance_score": 0.9821
    }
  ]
}

Claude Desktop Integration

Add to ~/.claude/claude_desktop_config.json:

{
  "mcpServers": {
    "arxiv-personalized": {
      "url": "http://127.0.0.1:8000/mcp"
    }
  }
}

Restart Claude Desktop to use query_papers and compute_similarity tools directly in conversations.

Architecture

User Chinese Query
    │
    ▼
[big_model.Talker]
    ├─ trans_cn2en()        CN → EN translation
    └─ extract_key_word()   Academic keyword extraction (JSON)
    │
    ▼
[arxiv.Client]              arXiv paper search
    │
    ▼
[big_model.Talker]
    └─ trans_en2cn()        Abstract EN → CN translation
    │
    ▼
[FlagReranker]              bge-reranker-v2-m3 semantic reranking
    │
    ▼
[FastAPI /query]            Returns ranked paper list
    │
    ▼
[fastapi-mcp /mcp]          MCP SSE endpoint for AI agent consumption

Dependencies

Package	Purpose
openai	DeepSeek API client (OpenAI-compatible)
arxiv	arXiv API Python client
FlagEmbedding	BGE reranker model for semantic scoring
fastapi + uvicorn	REST API server
fastapi-mcp	MCP protocol adapter
torch	Deep learning runtime for reranker
numpy	Numerical computation

Notes

• The reranker model BAAI/bge-reranker-v2-m3 (~1.1 GB) is auto-downloaded from HuggingFace on first run. Set HF_TOKEN or configure a mirror for faster downloads.
• Without a GPU, the reranker falls back to CPU (3–10 seconds per scoring batch).
• arXiv API imposes rate limits; keep max_results ≤ 10 for reliable operation.