Paperstack

PyPI version Python versions License

Overview

paperstack is a production-grade Model Context Protocol (MCP) server focused on arXiv research retrieval. It provides:

• arXiv Atom API search by ID/query
• PDF download, validation, and cache
• PDF text extraction (title, abstract, body, references)
• Token-aware context chunking for LLM pipelines
• CLI, API, and autonomous agent integration support

---

Table of Contents

1. Quickstart
2. Installation
3. Usage
4. MCP Server
5. Project structure
6. Configuration
7. Testing
8. Troubleshooting
9. Contributing
10. License

---

Quickstart

1. Clone repository

git clone https://github.com/Aldrin-Joan/paperstack.git
cd paperstack

2. Set up Python environment (recommended)

python -m venv .venv
# macOS/Linux
source .venv/bin/activate
# Windows
.venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

4. Run smoke test

python test_smoke.py

---

Installation

From source:

pip install -e .

From PyPI:

pip install paperstack-mcp

---

Usage

CLI

paperstack --help

Run server locally:

python -m src.mcp_server

Python API

from paperstack_mcp import entrypoint  # import alias for the package
from src.arxiv_client import ArxivClient
from src.pdf_fetcher import PdfFetcher
from src.pdf_parser import PdfParser
from src.context_builder import ContextBuilder

client = ArxivClient()
results = client.search('quantum computing', max_results=3)

pdf_path = PdfFetcher().fetch_paper(results[0].id)
parsed = PdfParser().parse(pdf_path)

context = ContextBuilder().build(parsed)
print(context.summary)

---

Architecture Layers

Layer	Features
Layer 1 — retrieval (both tools have this)	Search · PDF fetch + cache · Text extraction + chunking
Layer 2 — intelligence (your opportunity)	Citation graph · Concept extraction · Cross-paper synthesis
Layer 3 — dev tooling (highly unique)	Code + dataset links · Implementation diff · Reproducibility audit
Layer 4 — research workflows (unique)	Reading lists · Topic tracking + alerts · Agent-ready Q&A

MCP Server

src/mcp_server/__main__.py starts an MCP tool server exposing:

• arxiv_search (query or ID expand)
• arxiv_fetch_pdf (download + cache)
• arxiv_parse_pdf (extract text and metadata)
• arxiv_build_context (chunk to LLM-friendly context)
• arxiv_citation_graph (author/paper citation network)
• arxiv_extract_contributions (structured contribution extractor)
• arxiv_semantic_index (semantic similarity index builder/query)
• arxiv_compare_papers (paper comparison report)
• arxiv_extract_code_links (discover official GitHub/HuggingFace/Kaggle links from a paper)
• arxiv_reproducibility_score (reproducibility heuristic score with evidence details)
• arxiv_diff_implementations (compare paper method claims against a GitHub implementation)
• arxiv_reading_list (persistent reading list CRUD and filters)
• arxiv_watch_topic (watch query topics and detect new papers)
• arxiv_explain_for_audience (audience-specific explanation synthesis)

Use any MCP-capable client (VS Code MCP extension, custom agent SDK) to connect.

VS Code MCP server setup

In VS Code, add an MCP server entry to your workspace settings (e.g., .vscode/settings.json):

{
  "servers": {
    "arxiv-mcp": {
      "command": "D:/Softwares/Anaconda3/python.exe",
      "args": ["-m", "src.mcp_server"],
      "cwd": "${workspaceFolder}",
      "env": {
        "PYTHONPATH": "${workspaceFolder}",
        "ARXIV_DOWNLOAD_DIR": "${workspaceFolder}/downloads",
        "ARXIV_KEEP_PDFS": "true",
        "CHUNK_SIZE_TOKENS": "800",
        "CHUNK_OVERLAP_TOKENS": "100",
        "ARXIV_RATE_LIMIT_DELAY": "3.0",
        "MAX_RETRIES": "3",
        "HTTP_TIMEOUT": "60"
      }
    }
  }
}

MCP JSON entry for paperstack-mcp

If you installed from PyPI (pip install paperstack-mcp), the MCP server command can be the package executable instead of a direct Python module path. In .vscode/mcp.json or your .code-workspace settings, use an entry like:

{
  "servers": {
    "paperstack-mcp": {
      "command": "paperstack-mcp",
      "args": [],
      "cwd": "C:\\path\\to\\your\\project",
      "env": {
        "PYTHONPATH": "C:\\path\\to\\your\\project",
        "ARXIV_DOWNLOAD_DIR": "C:\\path\\to\\your\\project\\downloads",
        "ARXIV_KEEP_PDFS": "false",
        "CHUNK_SIZE_TOKENS": "800",
        "CHUNK_OVERLAP_TOKENS": "100",
        "ARXIV_RATE_LIMIT_DELAY": "3.0",
        "MAX_RETRIES": "3",
        "HTTP_TIMEOUT": "60"
      }
    }
  }
}

Adjust values for your local path, rate limit, and retry/timeouts.

• Run pip install paperstack-mcp first.
• Ensure workspace cwd and PYTHONPATH point to the project root.
• Customize ARXIV_DOWNLOAD_DIR for your downloaded PDF cache location.

Adjust values for your local path, rate limit, and retry/timeouts.

• Run pip install paperstack-mcp first.

• Ensure workspace cwd and PYTHONPATH point to the project root.

• Customize ARXIV_DOWNLOAD_DIR for your downloaded PDF cache location.

• ARXIV_DOWNLOAD_DIR: local storage for downloaded PDFs.

• ARXIV_KEEP_PDFS: keep cached PDFs after parse.

• CHUNK_SIZE_TOKENS / CHUNK_OVERLAP_TOKENS: controls text-chunking in context builder.

• ARXIV_RATE_LIMIT_DELAY: delay between arXiv API calls.

• MAX_RETRIES, HTTP_TIMEOUT: network robustness.

You can apply this configuration also in other compatible MCP clients using their server configuration schema.

---

Project structure

• src/ - package source
  • arxiv_client/ - arXiv Atom API logic
  • pdf_fetcher/ - download/cache PDF
  • pdf_parser/ - extract/clean PDF text
  • context_builder/ - tokenization + chunking
  • mcp_server/ - MCP protocol/adapters
• tests/ - pytest suite
• requirements.txt - dependencies
• pyproject.toml - package metadata

---

Configuration

Environment variables:

• ARXIV_CACHE_DIR (default: ./downloads)
• ARXIV_CACHE_TTL (default: 604800 seconds / 7 days)
• ARXIV_DB_PATH (default: ${ARXIV_DOWNLOAD_DIR}/arxiv_mcp.db) path to the SQLite workflow database
• ARXIV_RATE_LIMIT (default: 1 request/sec)
• S2_API_KEY (optional; Semantic Scholar API key for higher rate limits)
• OLLAMA_BASE_URL (default: http://localhost:11434)
• OLLAMA_MODEL (default: mistral)
• SEMANTIC_INDEX_DIR (default: ${ARXIV_DOWNLOAD_DIR}/semantic_index)
• CITATION_CACHE_TTL (default: 86400 seconds / 24 hours)
• CONTRIBUTION_CACHE_TTL (default: 604800 seconds / 7 days)
• EMBEDDING_MODEL (default: sentence-transformers/all-MiniLM-L6-v2)
• GITHUB_TOKEN (optional; for GitHub API auth, improves 60 -> 5000 req/hour)
• LINK_CACHE_TTL (default: 172800 seconds / 48 hours)
• REPRO_CACHE_TTL (default: 604800 seconds / 7 days)
• DIFF_CACHE_TTL (default: 86400 seconds / 24 hours)
• GITHUB_MAX_FILES (default: 20)
• GITHUB_MAX_FILE_SIZE_KB (default: 50)

Set in shell or via .env before running.

---

Testing

Run full tests:

pytest -q

Smoke test:

python test_smoke.py

---

Troubleshooting

• arxiv-mcp command not found: ensure virtualenv is active and package installed
• PDF download failure: check network access to https://arxiv.org/pdf/
• Rate-limit errors: lower request frequency or adjust ARXIV_RATE_LIMIT
• Topic duplicates observed after repeated tests: use DatabaseClient.reset() on workflow DB and/or topic_watcher.add now enforces dedupe by (query, label).
• Reading list duplicate notes: ReadingListManager.add now avoids re-appending identical note blocks.
• Ollama not available fallback: _passthrough now uses arXiv metadata.abstract for all explanation fields (what_it_is/problem_solved/how_it_works/why_it_matters/key_result).
• Dependency pin check: pip install -r requirements.txt includes protobuf==3.20.3 and urllib3>=2.0.0,<3 to avoid known warning/conflict cases (TensorFlow + ChromaDB MessageFactory and Requests RequestsDependencyWarning).
• Smoke harness summary: scripts/run_all_tools.py prints final status with count of run/passed/failed tools.

---

Contributing

1. Fork repo
2. Create feature branch
3. Add tests and update README
4. Open PR

Follow style checks (Black, formatting and lint).

---

License

Apache-2.0