Model Safety

Malicious or unsafe ML artifacts are a real supply-chain problem. There have already been real cases of malicious or suspicious models being discovered in the wild, including models hosted on public platforms and larger batches of unsafe AI/ML artifacts. See RL identifies malware ML model hosted on Hugging Face and Over 100 Malicious AI/ML Models Found on Hugging Face.

One of the biggest reasons this matters is deserialization risk. Many model formats, especially pickle-based and framework-specific formats, can execute code or invoke unsafe logic while being loaded or reconstructed. That means a model file can become an execution vector, not just a passive blob of weights.

model-safety is an MCP server for inspecting machine learning model artifacts before you load, ship, or trust them.

It is designed for practical triage:

• scan a local model file
• scan a downloadable model URL
• triage a whole directory of artifacts
• combine heuristic checks with dedicated scanners
• return normalized findings and concrete next actions

What It Can Do

The server currently exposes these tools:

• available_scanners Shows which scanner backends are installed and ready.
• artifact_safety_report Runs the broadest scan available on one model artifact and returns per-scanner results, normalized findings, highest severity, and recommended actions.
• modelscan_scan Runs ModelScan directly against a local file or URL.
• picklescan_scan Runs PickleScan directly against a local file or URL.
• deep_model_inspect Runs lightweight structural checks for risky extensions, embedded pickle members, and suspicious packaging patterns.
• scan_directory Runs artifact_safety_report across every file in a directory and aggregates the risky files.

Quick Start

1. Add the MCP server to Cursor or Claude Code using the local Python runtime in this repo.
2. Start with artifact_safety_report on a local file or direct URL.
3. Review normalized_findings, highest_severity, and recommended_actions.
4. Use scan_directory when you need to triage a whole model drop.

Minimal Cursor config:

{
  "mcpServers": {
    "model-safety": {
      "type": "stdio",
      "command": "C:/Users/Lenovo/Documents/ModelSafetyMCP/python312/python.exe",
      "args": ["C:/Users/Lenovo/Documents/ModelSafetyMCP/run_server.py"]
    }
  }
}

Current Detection Strengths

This MCP is strongest when scanning:

• PyTorch checkpoints such as .pt and .pth
• pickle-like artifacts such as .pkl, .pickle, and .joblib
• Keras and TensorFlow HDF5 models such as .h5
• model bundles that are actually ZIP or TAR containers

It currently combines:

• modelscan Best general-purpose backend, especially for model-specific unsafe patterns like Keras Lambda.
• picklescan Best supporting backend for pickle-oriented artifacts.
• deep_model_inspect Fast heuristic fallback that catches risky packaging even when specialized scanners are quiet.

Supported Inputs

You can scan either:

• a local filesystem path
• a direct http or https artifact URL

For Hugging Face specifically, use the direct file URL:

• Good: https://huggingface.co/<repo>/resolve/main/model.h5
• Not ideal: https://huggingface.co/<repo>/blob/main/model.h5

blob URLs usually return an HTML page, while resolve URLs return the real artifact bytes.

Runtime

This repo includes a local Python 3.12 runtime in python312. That is the supported runtime because it works with:

• mcp
• modelscan
• picklescan
• h5py

If you ever need to rebuild that runtime manually:

python312\python.exe -m pip install mcp picklescan modelscan h5py

Start The Server

Run the MCP server with:

python312\python.exe run_server.py

The launcher is repo-local and uses the runtime already bundled in this project.

Cursor Setup

Add the server in Cursor MCP settings with:

• Name: model-safety
• Type: stdio
• Command: C:/Users/Lenovo/Documents/ModelSafetyMCP/python312/python.exe
• Args: C:/Users/Lenovo/Documents/ModelSafetyMCP/run_server.py

Equivalent JSON:

{
  "mcpServers": {
    "model-safety": {
      "type": "stdio",
      "command": "C:/Users/Lenovo/Documents/ModelSafetyMCP/python312/python.exe",
      "args": ["C:/Users/Lenovo/Documents/ModelSafetyMCP/run_server.py"]
    }
  }
}

A copy-paste example also lives in cursor.mcp.example.json.

Claude Code Setup

Add the server with:

claude mcp add model-safety -- C:/Users/Lenovo/Documents/ModelSafetyMCP/python312/python.exe C:/Users/Lenovo/Documents/ModelSafetyMCP/run_server.py

Workflow

1. Confirm the server is connected

In your MCP client, ask:

Use available_scanners from the model-safety MCP server

You should see the installed backends, including modelscan and picklescan.

2. Start with the broad report

For a local file:

Use artifact_safety_report with path="C:/path/to/model.pth"

For a direct URL:

Use artifact_safety_report with url="https://example.com/model.h5"

This is the default entrypoint for single-artifact scans.

3. Read the normalized findings

artifact_safety_report returns:

• raw per-scanner outputs
• normalized_findings
• highest_severity
• finding_count
• recommended_actions

That means the tool is useful both for detailed investigation and for quick decision-making.

4. Drill down only when needed

Use:

• modelscan_scan when you want the dedicated ModelScan output
• picklescan_scan when you want pickle-specific detail
• deep_model_inspect when you want lightweight packaging and archive heuristics

5. Triage folders in bulk

If you have a whole drop of models:

Use scan_directory with path="C:/path/to/model-folder"

This returns:

• aggregated normalized findings
• a directory-level highest severity
• risky_files for quick prioritization
• per-file nested reports

scan_directory works best on folders that mostly contain model artifacts rather than general source code.

Examples

Example: suspicious PyTorch checkpoint

Use artifact_safety_report with path="C:/Users/Lenovo/Documents/ModelSafetyMCP/credit-risk-pytorch-v1.1.pth"

Typical result shape:

• risky extension .pth
• embedded pickle member inside the archive
• recommended action to avoid trusted loading until reviewed

Example: suspicious Keras model from Hugging Face

Use artifact_safety_report with url="https://huggingface.co/MrKrauzer/FacenetRetweeted/resolve/main/facenet-retrained.h5"

Typical result shape:

• risky extension .h5
• ModelScan H5LambdaDetectScan finding
• recommendation to inspect the Keras Lambda layer before deserializing

Output Philosophy

The server treats the scanners as complementary:

• modelscan is the primary security backend
• picklescan is a specialized supporting backend
• deep_model_inspect is a fast structural fallback

Instead of forcing users to interpret each tool separately, the server also produces normalized findings with:

• scanner
• severity
• category
• evidence
• source
• recommended_action

Limitations

This tool helps triage risk. It does not prove a model is safe.

Important limits:

• some formats are better covered than others
• a clean scan does not guarantee harmless behavior
• remote URL scanning depends on the URL pointing to the real artifact bytes
• directory scans are literal and can be noisy on non-model folders
• model behavior risks are different from serialization and packaging risks

Development

Quick verification:

python312\python.exe -m compileall src run_server.py