shuji-bonji/xcomet-mcp-server

npm version CI MCP License: MIT

日本語版 README はこちら

⚠️ This is an unofficial community project, not affiliated with Unbabel.

Translation quality evaluation MCP Server powered by xCOMET (eXplainable COMET).

🎯 Overview

xCOMET MCP Server provides AI agents with the ability to evaluate machine translation quality. It integrates with the xCOMET model from Unbabel to provide:

• Quality Scoring: Scores between 0-1 indicating translation quality
• Error Detection: Identifies error spans with severity levels (minor/major/critical)
• Batch Processing: Evaluate multiple translation pairs efficiently (optimized single model load)
• GPU Support: Optional GPU acceleration for faster inference

graph LR
    A[AI Agent] --> B[Node.js MCP Server]
    B -- stdio JSON-RPC --> C[Python Worker]
    C --> D[xCOMET Model<br/>Persistent in Memory]
    D --> C
    C --> B
    B --> A

    style D fill:#9f9

🔧 Prerequisites

Python Environment

• Python 3.9 - 3.12 recommended (3.13+ is not yet supported by xCOMET dependencies)

xCOMET requires Python with several packages. We recommend using a virtual environment:

# If using uv (recommended - auto-downloads the correct Python version)
uv venv ~/.xcomet-venv --python 3.12
source ~/.xcomet-venv/bin/activate
uv pip install "unbabel-comet>=2.2.7,<3.0"

# Or using standard venv (requires Python 3.9-3.12 already installed)
python3 -m venv ~/.xcomet-venv
source ~/.xcomet-venv/bin/activate  # Windows: ~/.xcomet-venv\Scripts\activate
pip install "unbabel-comet>=2.2.7,<3.0"

Why Python 3.9-3.12? unbabel-comet declares numpy = "^1.20.0", so it resolves numpy 1.x. The last numpy 1.x release, 1.26.4, ships wheels for cp39-cp312 only. On Python 3.13 or later, pip has to build numpy from source.

Note (v0.5.0+): The Python worker now talks to Node.js over stdin/stdout (line-delimited JSON-RPC). FastAPI, uvicorn, and pydantic are no longer required — only unbabel-comet is.

Note: When using with Claude Desktop or other MCP hosts, set XCOMET_PYTHON_PATH to point to the venv Python (see Configuration).

Model Download

Important: XCOMET-XL and XCOMET-XXL are gated models on HuggingFace. You must:

1. Create a HuggingFace account

2. Visit Unbabel/XCOMET-XL and request access

3. Authenticate, either via the CLI:

   source ~/.xcomet-venv/bin/activate
   hf auth login
   

   (huggingface-cli login still works but prints a deprecation warning since huggingface_hub 0.34; hf is the current command.)

   Or by setting HF_TOKEN in the MCP host's env block, which is the option when the host launches the server in an environment where no CLI login has been performed:

   "env": {
     "XCOMET_PYTHON_PATH": "~/.xcomet-venv/bin/python3",
     "HF_TOKEN": "hf_..."
   }
   

   huggingface_hub reads HF_TOKEN first and falls back to the token file written by hf auth login.

Unbabel/wmt22-comet-da does not require authentication (but requires reference translations).

After authentication, download the model (~14GB for XL, ~42GB for XXL):

source ~/.xcomet-venv/bin/activate
python -c "from comet import download_model; download_model('Unbabel/XCOMET-XL')"

Where the model is stored

Not in the virtualenv. The venv holds the Python packages; the model weights go to the huggingface_hub cache, which is a separate directory shared by every tool on the machine that pulls from the Hub.

~/.xcomet-venv/                              ← Python packages only
└── lib/python3.x/site-packages/
    ├── comet/                               unbabel-comet itself
    └── torch/  transformers/  ...           its dependencies

~/.cache/huggingface/                        ← the model weights live here
└── hub/
    └── models--Unbabel--XCOMET-XL/
        ├── blobs/                           the actual ~14GB checkpoint
        └── snapshots/<revision>/
            ├── checkpoints/model.ckpt       what download_model() returns
            └── hparams.yaml

download_model() passes cache_dir=None to snapshot_download(), so huggingface_hub picks the location: HF_HUB_CACHE, which defaults to HF_HOME/hub, where HF_HOME defaults to $XDG_CACHE_HOME/huggingface (~/.cache/huggingface when XDG_CACHE_HOME is unset).

Three consequences worth knowing:

• Rebuilding or deleting the venv does not re-download the model.
• Several venvs, and other Hub-based tools, share the same copy.
• The size of the venv directory does not account for the 14GB. Use hf cache scan to see what is actually on disk, and hf cache delete to remove a revision.

To put the checkpoint somewhere else — a larger volume, a shared drive — set XCOMET_SAVING_DIRECTORY (v0.7.0+) or the standard HF_HOME. Both are read at download time, so a model already downloaded to the default location is not moved; it is downloaded again into the new one.

Node.js

• Node.js >= 22.0.0 (matches engines.node in package.json; CI runs on 22 and 24)
• npm or yarn

📦 Installation

Note: If you just want to use xCOMET MCP Server, you do not need to clone this repository. Install the Python environment and model (see Prerequisites), then use npx (see Usage). The section below is for contributors and local development only.

Local Development

For contributors and local development:

# Clone the repository
git clone https://github.com/shuji-bonji/xcomet-mcp-server.git
cd xcomet-mcp-server

# Set up Python virtual environment and install dependencies
uv venv .venv --python 3.12    # or: python3 -m venv .venv
source .venv/bin/activate
pip install -r python/requirements.txt

# Install Node.js dependencies and build
npm install
npm run build

🚀 Usage

With Claude Desktop (npx)

Add to your Claude Desktop configuration (claude_desktop_config.json):

{
  "mcpServers": {
    "xcomet": {
      "command": "npx",
      "args": ["-y", "xcomet-mcp-server@latest"],
      "env": {
        "XCOMET_PYTHON_PATH": "~/.xcomet-venv/bin/python3"
      }
    }
  }
}

Tip: If you installed Python packages system-wide or use pyenv, XCOMET_PYTHON_PATH may be omitted (auto-detection will find it). See Python Path Auto-Detection for details.

With Claude Code

claude mcp add xcomet --env XCOMET_PYTHON_PATH=~/.xcomet-venv/bin/python3 -- npx -y xcomet-mcp-server@latest

Global Installation

If you prefer installing globally:

npm install -g xcomet-mcp-server

Then configure:

{
  "mcpServers": {
    "xcomet": {
      "command": "xcomet-mcp-server",
      "env": {
        "XCOMET_PYTHON_PATH": "~/.xcomet-venv/bin/python3"
      }
    }
  }
}

Local Development Build

If you cloned and built the repository locally (see Installation):

{
  "mcpServers": {
    "xcomet": {
      "command": "node",
      "args": ["/path/to/xcomet-mcp-server/dist/index.js"],
      "env": {
        "XCOMET_PYTHON_PATH": "~/.xcomet-venv/bin/python3"
      }
    }
  }
}

🛠️ Available Tools

xcomet_evaluate

Evaluate translation quality for a single source-translation pair.

Parameters:

Name	Type	Required	Description
source	string	✅	Original source text
translation	string	✅	Translated text to evaluate
reference	string	❌	Reference translation
source_lang	string	❌	Source language code (ISO 639-1)
target_lang	string	❌	Target language code (ISO 639-1)
response_format	"json" | "markdown"	❌	Output format (default: "json")
use_gpu	boolean	❌	Use GPU for inference (default: false)

Example:

{
  "source": "The quick brown fox jumps over the lazy dog.",
  "translation": "素早い茶色のキツネが怠惰な犬を飛び越える。",
  "source_lang": "en",
  "target_lang": "ja",
  "use_gpu": true
}

Response:

{
  "score": 0.847,
  "errors": [],
  "summary": "Good quality (score: 0.847) with 0 error(s) detected."
}

xcomet_detect_errors

Focus on detecting and categorizing translation errors.

Parameters:

Name	Type	Required	Description
source	string	✅	Original source text
translation	string	✅	Translated text to analyze
reference	string	❌	Reference translation
min_severity	"minor" | "major" | "critical"	❌	Minimum severity (default: "minor")
response_format	"json" | "markdown"	❌	Output format
use_gpu	boolean	❌	Use GPU for inference (default: false)

xcomet_batch_evaluate

Evaluate multiple translation pairs in a single request.

Performance Note: With the persistent server architecture (v0.3.0+), the model stays loaded in memory. Batch evaluation processes all pairs efficiently without reloading the model.

Parameters:

Name	Type	Required	Description
pairs	array	✅	Array of {source, translation, reference?} (max 500)
source_lang	string	❌	Source language code
target_lang	string	❌	Target language code
response_format	"json" | "markdown"	❌	Output format
use_gpu	boolean	❌	Use GPU for inference (default: false)
batch_size	number	❌	Batch size 1-64 (default: 8). Larger = faster but uses more memory

Example:

{
  "pairs": [
    {"source": "Hello", "translation": "こんにちは"},
    {"source": "Goodbye", "translation": "さようなら"}
  ],
  "use_gpu": true,
  "batch_size": 16
}

🔗 Integration with Other MCP Servers

xCOMET MCP Server is designed to work alongside other MCP servers for complete translation workflows:

sequenceDiagram
    participant Agent as AI Agent
    participant DeepL as DeepL MCP Server
    participant xCOMET as xCOMET MCP Server
    
    Agent->>DeepL: Translate text
    DeepL-->>Agent: Translation result
    Agent->>xCOMET: Evaluate quality
    xCOMET-->>Agent: Score + Errors
    Agent->>Agent: Decide: Accept or retry?

Recommended Workflow

1. Translate using DeepL MCP Server (official)
2. Evaluate using xCOMET MCP Server
3. Iterate if quality is below threshold

Example: DeepL + xCOMET Integration

Configure both servers in Claude Desktop:

{
  "mcpServers": {
    "deepl": {
      "command": "npx",
      "args": ["-y", "@anthropic/deepl-mcp-server"],
      "env": {
        "DEEPL_API_KEY": "your-api-key"
      }
    },
    "xcomet": {
      "command": "npx",
      "args": ["-y", "xcomet-mcp-server@latest"],
      "env": {
        "XCOMET_PYTHON_PATH": "~/.xcomet-venv/bin/python3"
      }
    }
  }
}

Then ask Claude:

"Translate this text to Japanese using DeepL, then evaluate the translation quality with xCOMET. If the score is below 0.8, suggest improvements."

⚙️ Configuration

Environment Variables

Variable	Default	Description
XCOMET_MODEL	Unbabel/XCOMET-XL	xCOMET model to use
XCOMET_PYTHON_PATH	(auto-detect)	Python executable path (see below)
XCOMET_PRELOAD	false	Pre-load model at startup (v0.3.1+)
XCOMET_DEBUG	false	Enable verbose debug logging (v0.3.1+)
XCOMET_NUM_WORKERS	1	DataLoader workers for model.predict() (v0.6.0+). Increase to better utilize idle CPU cores when running large batches, especially on GPU. Invalid values silently fall back to 1.
XCOMET_SAVING_DIRECTORY	(HuggingFace cache)	Directory to download the checkpoint into (v0.7.0+). Unset, the model goes to the huggingface_hub cache (HF_HOME, default ~/.cache/huggingface). Set this to put a 14GB (XL) or 43GB (XXL) checkpoint on another volume.
XCOMET_LOCAL_FILES_ONLY	false	Resolve the checkpoint from the local cache only (v0.7.0+). Set to true to start with no network access; the model must already be downloaded.
HF_TOKEN	(unset)	HuggingFace access token, read by huggingface_hub. An alternative to hf auth login for the gated models (XCOMET-XL, XCOMET-XXL, the CometKiwi models).

Model Selection

Choose the model based on your quality/performance needs:

Model	Parameters	Size	Memory	Reference	HF Auth	Quality	Use Case
Unbabel/XCOMET-XL	3.5B	~14GB	~8-10GB	Optional	✅ Required	⭐⭐⭐⭐	Recommended for most use cases
Unbabel/XCOMET-XXL	10.7B	~42GB	~20GB	Optional	✅ Required	⭐⭐⭐⭐⭐	Highest quality, requires more resources
Unbabel/wmt22-comet-da	580M	~2GB	~3GB	Required	Not required	⭐⭐⭐	Lightweight, faster loading

Important: XCOMET-XL and XCOMET-XXL are gated models on HuggingFace. Each model requires separate access approval. See Model Download for authentication setup.

Important: wmt22-comet-da requires a reference translation for evaluation. XCOMET models support referenceless evaluation.

Tip: If you experience memory issues or slow model loading, try Unbabel/wmt22-comet-da for faster performance with slightly lower accuracy (but remember to provide reference translations).

To use a different model, set the XCOMET_MODEL environment variable:

{
  "mcpServers": {
    "xcomet": {
      "command": "npx",
      "args": ["-y", "xcomet-mcp-server@latest"],
      "env": {
        "XCOMET_MODEL": "Unbabel/XCOMET-XXL"
      }
    }
  }
}

Python Path Auto-Detection

The server automatically detects a Python environment with unbabel-comet installed:

1. XCOMET_PYTHON_PATH environment variable (if set)
2. pyenv versions (~/.pyenv/versions/*/bin/python3) - checks for comet module
3. Homebrew Python (/opt/homebrew/bin/python3, /usr/local/bin/python3)
4. Fallback: python3 command

This ensures the server works correctly even when the MCP host (e.g., Claude Desktop) uses a different Python than your terminal.

Example: Explicit Python path configuration

{
  "mcpServers": {
    "xcomet": {
      "command": "npx",
      "args": ["-y", "xcomet-mcp-server@latest"],
      "env": {
        "XCOMET_PYTHON_PATH": "/Users/you/.pyenv/versions/3.11.0/bin/python3"
      }
    }
  }
}

⚡ Performance

Persistent Worker Architecture (v0.3.0+, stdio since v0.5.0)

The server uses a persistent Python worker process that keeps the xCOMET model loaded in memory. The Node.js MCP server talks to the worker over stdin/stdout using a line-delimited JSON-RPC protocol — no local HTTP listener, no port binding, no FastAPI.

Request	Time	Notes
First request	~25-90s	Model loading (varies by model size)
Subsequent requests	~500ms	Model already loaded

This provides a 177x speedup for consecutive evaluations compared to reloading the model each time.

Eager Loading (v0.3.1+)

Enable XCOMET_PRELOAD=true to pre-load the model at server startup:

{
  "mcpServers": {
    "xcomet": {
      "command": "npx",
      "args": ["-y", "xcomet-mcp-server@latest"],
      "env": {
        "XCOMET_PRELOAD": "true"
      }
    }
  }
}

With preload enabled, all requests are fast (~500ms), including the first one.

graph LR
    A[MCP Request] --> B[Node.js Server]
    B -- stdio JSON-RPC --> C[Python Worker]
    C --> D[xCOMET Model<br/>in Memory]
    D --> C
    C --> B
    B --> A

    style D fill:#9f9

Batch Processing Optimization

The xcomet_batch_evaluate tool processes all pairs with a single model load:

Pairs	Estimated Time
10	~30-40 sec
50	~1-1.5 min
100	~2 min

GPU vs CPU Performance

Mode	100 Pairs (Estimated)
CPU (batch_size=8)	~2 min
GPU (batch_size=16)	~20-30 sec

Note: GPU requires CUDA-compatible hardware and PyTorch with CUDA support. If GPU is not available, set use_gpu: false (default).

Best Practices

1. Let the persistent server do its job

With v0.3.0+, the model stays in memory. Multiple xcomet_evaluate calls are now efficient:

✅ Fast: First call loads model, subsequent calls reuse it
   xcomet_evaluate(pair1)  # ~90s (model loads)
   xcomet_evaluate(pair2)  # ~500ms (model cached)
   xcomet_evaluate(pair3)  # ~500ms (model cached)

2. For many pairs, use batch evaluation

✅ Even faster: Batch all pairs in one call
   xcomet_batch_evaluate(allPairs)  # Optimal throughput

3. Memory considerations

• XCOMET-XL requires ~8-10GB RAM
• For large batches (500 pairs), ensure sufficient memory
• If memory is limited, split into smaller batches (100-200 pairs)

Auto-Restart (v0.3.1+)

The server automatically recovers from failures:

• Monitors health every 30 seconds
• Restarts after 3 consecutive health check failures
• Up to 3 restart attempts before giving up

📊 Quality Score Interpretation

Score Range	Quality	Recommendation
0.9 - 1.0	Excellent	Ready for use
0.7 - 0.9	Good	Minor review recommended
0.5 - 0.7	Fair	Post-editing needed
0.0 - 0.5	Poor	Re-translation recommended

What the score does and does not tell you

The score answers "does this read like a translation of that source", and it is good at it. It does not answer "are the facts in this translation correct". Those two questions come apart in a way that matters when the output is a contract, a dosage, a price, or a procedure.

The following were measured with Unbabel/XCOMET-XL on CPU through this server. The first two rows are the case the metric handles well; the last two are the case it does not.

Source	Translation	Score
ファイルを保存せずに終了しますか？	Do you want to quit without saving the file?	0.956
ファイルを保存せずに終了しますか？	The mountain sings in violet every third Thursday.	0.212
保証期間は購入日から一年間です。	The warranty period is ten years from the date of purchase.	1.000
電源を切ってから、カバーを取り外してください。	Remove the cover, then turn off the power.	1.000

A translation that is unrelated to the source collapses to ~0.2, which is what you want. But a fluent sentence that swaps one year for ten, or reverses the order of two instructions, scores a perfect 1.000. Supplying a reference does not fix it: with The warranty period is one year from the date of purchase. as the reference, the "ten years" translation still scores 0.983.

This is not a defect in this server or in xCOMET specifically. It is a known property of neural MT metrics: they "struggle with detecting certain phenomena that can be considered as critical errors, such as deviations in entities and numbers" (Rei et al., 2023).

Using it accordingly

Good fits

• Ranking or triaging a set of translations — which segments to review first, which of two MT systems is better on your data.
• Catching adequacy collapse — truncated output, the wrong segment pasted in, a model that lost the thread, an untranslated passthrough.
• Tracking quality over time on a fixed test set, where the comparison is between runs rather than against an absolute bar.
• A first-pass filter ahead of human review, to decide where the human time goes.

Poor fits

• A sole release gate for content where a single wrong number or name is the failure — medical, legal, financial, safety instructions. Check numbers, dates, units, currencies, and named entities separately, with a rule that actually compares them; the score will not do it for you.
• An absolute quality claim. 0.95 is not "95% correct", and the value is not comparable across models, language pairs, or segment lengths.
• Very short segments (a UI label, a single word), where the score saturates and stops discriminating.

Use xcomet_detect_errors alongside the score. The error spans mark where the model believes something went wrong, with an MQM severity. A high score with a critical span is a more useful signal than either number alone.

🔍 Troubleshooting

Common Issues

"No module named 'comet'"

Cause: Python environment without unbabel-comet installed.

Solution:

# Check which Python is being used
python3 -c "import sys; print(sys.executable)"

# If using a virtual environment, make sure it's activated
source .venv/bin/activate
pip install -r python/requirements.txt

# For MCP hosts (e.g., Claude Desktop), specify the venv Python path
export XCOMET_PYTHON_PATH=~/.xcomet-venv/bin/python3

The venv stopped working after a Homebrew upgrade

Symptom: zsh: no such file or directory: .venv/bin/python3, or python3 inside an activated venv resolving to a different interpreter than the venv's, or No module named 'comet' in a venv that worked yesterday.

Cause: A venv does not contain an interpreter — it stores an absolute symlink to the one it was created from, recorded in pyvenv.cfg:

home = /opt/homebrew/opt/[email protected]/bin
version = 3.14.3

When Homebrew upgrades or removes that formula, the link dangles and the venv is dead. lib/python3.x/site-packages/ is still there, but nothing can run it.

Check:

ls -l .venv/bin/python3 && .venv/bin/python3 -V
cat .venv/pyvenv.cfg

Solution: recreate it. uv venv --python 3.12 is the more durable form, because uv fetches and pins that interpreter itself instead of borrowing Homebrew's current one.

rm -rf .venv
uv venv .venv --python 3.12
source .venv/bin/activate
uv pip install "unbabel-comet>=2.2.7,<3.0"

Reinstalling the packages is a few hundred MB, but the model is not re-downloaded: the checkpoint lives in the huggingface_hub cache, not in the venv (see Where the model is stored).

An old version starts after an upgrade

Symptom: a fix you know is published is missing from the running server. The startup banner in the log names an older version.

grep "running on stdio" ~/Library/Logs/Claude/mcp-server-xcomet.log | tail -1
xcomet-mcp-server v0.6.3 running on stdio

(On macOS, Claude Desktop writes each MCP server's stderr to ~/Library/Logs/Claude/mcp-server-<name>.log.)

Cause: npx resolves latest from npm's cached registry metadata, and runs the copy it already installed under ~/.npm/_npx. For a while after a release, that copy is the previous version. It catches up on its own, but not at a time you choose. @latest does not change this — to npm it means the same as writing no version at all.

Solution: clear the npx cache, then restart the MCP host.

rm -rf ~/.npm/_npx

ps shows which copy is running, and which cache directory it came from.

ps -axo pid,command | grep xcomet-mcp-server | grep -v grep

To stay on a known build, pin an exact version: [email protected].

Model download fails or times out

Cause: Large model files (~14GB for XL) require stable internet connection. XCOMET models also require HuggingFace authentication (see Model Download).

Solution:

# Authenticate with HuggingFace (required for XCOMET-XL/XXL)
hf auth login          # or: export HF_TOKEN=hf_...

# Pre-download the model manually
python -c "from comet import download_model; download_model('Unbabel/XCOMET-XL')"

If the download was interrupted, the cache keeps a snapshot directory with no checkpoints/model.ckpt in it. The server reports that path and asks you to delete the directory; hf cache scan lists where it is.

GPU not detected

Cause: PyTorch not installed with CUDA support.

Solution:

# Check CUDA availability
python -c "import torch; print(torch.cuda.is_available())"

# If False, reinstall PyTorch with CUDA
pip install torch --index-url https://download.pytorch.org/whl/cu118

Slow performance on Mac (MPS)

Cause: Mac MPS (Metal Performance Shaders) has compatibility issues with some operations.

Solution: The server automatically uses num_workers=1 for Mac MPS compatibility. For best performance on Mac, use CPU mode (use_gpu: false).

High memory usage or crashes

Cause: XCOMET-XL requires ~8-10GB RAM.

Solutions:

1. Use the persistent server (v0.3.0+): Model loads once and stays in memory, avoiding repeated memory spikes
2. Use a lighter model: Set XCOMET_MODEL=Unbabel/wmt22-comet-da for lower memory usage (~3GB)
3. Reduce batch size: For large batches, process in smaller chunks (100-200 pairs)
4. Close other applications: Free up RAM before running large evaluations

# Check available memory
free -h  # Linux
vm_stat | head -5  # macOS

VS Code or IDE crashes during evaluation

Cause: High memory usage from the xCOMET model (~8-10GB for XL).

Solution:

• With v0.3.0+, the model loads once and stays in memory (no repeated loading)
• If memory is still an issue, use a lighter model: XCOMET_MODEL=Unbabel/wmt22-comet-da
• Close other memory-intensive applications before evaluation

Getting Help

If you encounter issues:

1. Check the GitHub Issues
2. Enable debug logging (check Claude Desktop's Developer Mode logs, or set XCOMET_DEBUG=true)
3. Open a new issue with:
   • Your OS and Python version
   • The error message
   • Your configuration (without sensitive data)

🧪 Development

# Install dependencies
npm install

# Build TypeScript
npm run build

# Watch mode
npm run dev

# Run tests (Vitest)
npm test

# Run the Python-side tests (pytest only — no comet, no model)
npm run test:python

# Test with MCP Inspector
npm run inspect

npm run test:python calls python3 -m pytest, so pytest has to be importable from whichever python3 is on your PATH. Activating the xCOMET venv does not help unless pytest is installed in it. Any of these work:

uvx pytest tests/test_server.py -q                    # nothing to install
~/.xcomet-venv/bin/python -m pip install pytest       # add it to the xCOMET venv
python3 -m venv .venv-dev && .venv-dev/bin/pip install pytest

tests/README.md documents every suite, what it covers, and why.

📋 Changelog

See CHANGELOG.md for version history and updates.

📝 License

MIT License - see LICENSE for details.

🙏 Acknowledgments

• Unbabel for the xCOMET model
• Anthropic for the MCP protocol
• Model Context Protocol community

📚 References

• xCOMET Paper
• COMET Framework
• BLEU Meets COMET (Rei et al., 2023) — on neural metrics missing entity and number errors; the basis for What the score does and does not tell you
• Hugging Face Hub cache layout
• MCP Specification