Local LLM Tool

Python License MCP

A local MCP (Model Context Protocol) server that runs Llama models entirely on your machine. No API keys, no cloud costs, 100% private and offline-capable.

✨ Features

• 🚀 100% Local - All inference runs on your CPU/GPU, no data leaves your machine
• 🔒 Private - Your conversations stay on your device
• 💰 Free - No API costs or usage limits
• 🛠️ Multiple Tools - generate_text, chat, complete, read_file, analyze_file, and session management via MCP
• 💬 Conversation History & Sessions - Persistent session management with automatic history trimming to minimize storage
• 📡 Streaming Support - Optional incremental token streaming for faster response display
• 🪟 Windows Optimized - Pre-built wheels and installation scripts included
• 🔌 Cursor Compatible - Works seamlessly with Cursor IDE
• 🌌 Antigravity Compatible - Native integration with Google's Antigravity AI assistant

🆕 Recent Additions

• Conversation History & Sessions: Create persistent conversation sessions with automatic history management. Sessions store messages in history/ folder with configurable limits to minimize storage usage.
• Streaming Responses: Enable incremental token streaming for faster perceived response times. Configure chunk size and enable/disable via environment variables.

📋 Requirements

• Python 3.10 or higher
• Windows 10/11 (Linux/Mac support coming soon)
• A Llama model in GGUF format (can be downloaded automatically)

🚀 Quick Start

1. Clone the repository

git clone https://github.com/Marcel-MSC/local-llm-mcp-tool.git
cd local-llm-mcp-tool

2. Install dependencies

pip install -r requirements.txt

3. Install llama-cpp-python

For Windows, use pre-built wheels (recommended):

Option A: CPU only

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

Option B: NVIDIA GPU (better performance)

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121

Or use the automated installer:

PowerShell:

.\scripts\install_llama.ps1

CMD/Batch:

scripts\install_llama.bat

⚠️ Note: If you get compilation errors, see WINDOWS_INSTALLATION.md for troubleshooting. You can also install Visual Studio Build Tools to compile from source.

4. Configure the model

copy .env.example .env

Edit .env and set your model path:

MODEL_PATH=C:\path\to\your\model.gguf

5. Download a model (if needed)

python scripts/download_model.py

Or download manually from Hugging Face and update MODEL_PATH in .env.

6. Test the setup

python scripts/test_server.py

7. Run the server

python server.py

Or use the FastMCP version (simpler):

python server_fastmcp.py

🔧 Configuration

Environment Variables (.env)

Variable	Description	Default
MODEL_PATH	Path to your GGUF model file	Required
CONTEXT_SIZE	Maximum context window size	2048
N_THREADS	Number of CPU threads	4
N_GPU_LAYERS	GPU layers (use -1 for all, 0 for CPU only)	0
SESSION_HISTORY_DIR	Directory for storing conversation history	history
SESSION_MAX_MESSAGES	Maximum messages per session (older messages trimmed)	40
SESSION_MAX_FILE_BYTES	Maximum size per session file (bytes)	2097152 (~2MB)
SESSION_AUTO_TRIM	Automatically trim history when limits exceeded	true
STREAMING_ENABLED	Enable streaming responses (tokens sent incrementally)	false
STREAMING_CHUNK_SIZE	Approximate chunk size for streaming (characters)	50

Using with Cursor IDE

1. Open Cursor Settings (Ctrl+,)
2. Search for "MCP" or edit settings.json directly
3. Add the configuration:

{
  "mcpServers": {
    "local-llm": {
      "command": "python",
      "args": [
        "C:\\path\\to\\local-llm-mcp-tool\\server.py"
      ],
      "env": {
        "PYTHONUNBUFFERED": "1"
      }
    }
  }
}

Or use project-specific config: Create .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "local-llm": {
      "command": "python",
      "args": [
        "C:\\path\\to\\local-llm-mcp-tool\\server.py"
      ],
      "env": {
        "PYTHONUNBUFFERED": "1"
      }
    }
  }
}

4. Restart Cursor
5. The server will appear in Tools & MCP → Installed MCP Servers

Using with Antigravity

Antigravity natively supports the Model Context Protocol. You can connect your local models easily using the provided configuration example antigravity_mcp.json.example.

For detailed instructions, see the Antigravity Integration Guide.

🛠️ Available Tools

The server exposes several MCP tools:

1. generate_text

Generate text using the local Llama model.

Parameters:

• prompt (required): The text prompt
• max_tokens (optional, default: 256): Maximum tokens to generate
• temperature (optional, default: 0.7): Sampling temperature (0.0-2.0)
• top_p (optional, default: 0.9): Top-p sampling (0.0-1.0)

2. chat

Chat with the model using a message-based format.

Parameters:

• messages (required): Array of messages [{"role": "user", "content": "..."}]
• max_tokens (optional, default: 256): Maximum tokens to generate
• temperature (optional, default: 0.7): Sampling temperature

3. complete

Complete a text prompt.

Parameters:

• text (required): The text to complete
• max_tokens (optional, default: 128): Maximum tokens to generate
• temperature (optional, default: 0.7): Sampling temperature

4. read_file

Read a local text file from the MCP server's project directory. Relative paths are resolved from the directory containing server.py. Access is restricted to that directory tree (no ../.. traversal).

Parameters:

• path (required): File path (relative to the server root directory)
• max_bytes (optional, default: 200000): Maximum bytes to read (prevents huge reads)
• encoding (optional, default: "utf-8"): Text encoding used to decode file bytes

5. analyze_file

Read a local text file and ask the local Llama model to analyze it (purpose, structure, issues, improvements).

Parameters:

• path (required): File path (relative to the server root directory)
• instruction (optional): Custom analysis instruction (e.g. "Focus on security", "Summarize in 3 bullets")
• max_bytes (optional, default: 200000): Maximum bytes to read from the file
• encoding (optional, default: "utf-8"): File encoding
• max_tokens (optional, default: 512): Max tokens for the analysis response
• temperature (optional, default: 0.3): Sampling temperature for analysis

6. start_session

Start a new conversation session and get back a session_id. This groups multiple turns together while keeping CPU and disk usage bounded.

Parameters:

• metadata (optional): JSON object with metadata like purpose, label, etc.

7. continue_session

Continue an existing session by adding a new user message. The server loads only the most recent messages for context (limited by environment variables) to avoid heavy CPU and storage usage.

Parameters:

• session_id (required): The ID returned by start_session.
• message (required): The new user message.
• max_tokens (optional, default: 256): Maximum tokens to generate.
• temperature (optional, default: 0.7): Sampling temperature.
• top_p (optional, default: 0.9): Top-p sampling.

8. end_session

Mark a session as ended and optionally delete its history from disk.

Parameters:

• session_id (required): The ID of the session to end.
• delete (optional, default: false): Whether to delete the stored history.

📚 Usage Examples

In Cursor Chat

Basic text generation:

Use the generate_text tool from local-llm with prompt: Write a short sentence about programming.

Chat with messages:

Use the chat tool from local-llm with messages: [{"role": "user", "content": "What is Python?"}]

Read a file from disk (returns the file text):

Use the read_file tool from local-llm with:
path: README.md

Analyze a file from disk (server reads the file and the LLM analyzes it):

Use the analyze_file tool from local-llm with:
path: server.py
instruction: Summarize the main components and list 3 improvements.

“Use generate_text and read README.md + server.py” (2-step workflow):

1. Read each file (one tool call per file):

Use the read_file tool from local-llm with:
path: README.md

Use the read_file tool from local-llm with:
path: server.py
max_bytes: 200000

2. Then call generate_text using the file contents shown above in chat context:

Use the generate_text tool from local-llm with prompt: Compare the README and server.py content above. Are the documented tools accurate? List any mismatches and propose README fixes.

Using conversation sessions:

1. Start a session:

Use the start_session tool from local-llm with metadata: {"label": "coding-help"}

2. Continue the conversation (use the session_id from step 1):

Use the continue_session tool from local-llm with:
session_id: abc123...
message: How do I create a Python function?

3. Continue with more messages using the same session_id to maintain context.

4. End the session when done:

Use the end_session tool from local-llm with:
session_id: abc123...
delete: false

Programmatic Usage

See scripts/example_usage.py for Python examples.

🎯 Recommended Models

Any Llama-compatible model in GGUF format works. Recommended:

• Llama 3.2 1B - Lightweight, fast, good for CPU
• Llama 3.1 8B - Balanced performance/quality
• Mistral 7B - Alternative option

Download from: Hugging Face GGUF Models

💬 Conversation History & Sessions

The server supports persistent conversation sessions that maintain context across multiple interactions while minimizing storage and CPU usage.

How Sessions Work

1. Start a session using start_session to get a unique session_id
2. Continue conversations using continue_session with the same session_id to maintain context
3. History is stored in the history/ folder (one .jsonl file per session)
4. Automatic trimming keeps only the most recent messages (configurable limits)
5. End sessions with end_session when done (optionally delete history)

Storage Management

• History files are stored in history/<session_id>.jsonl (line-delimited JSON)
• Session metadata is tracked in history/sessions_index.json
• Automatic trimming prevents unbounded growth:
  • Maximum messages per session (default: 40)
  • Maximum file size per session (default: ~2MB)
• The history/ folder is gitignored by default

Configuration

See the Environment Variables table above for session-related settings:

• SESSION_HISTORY_DIR: Where to store history files
• SESSION_MAX_MESSAGES: How many messages to keep per session
• SESSION_MAX_FILE_BYTES: Maximum file size before trimming
• SESSION_AUTO_TRIM: Enable/disable automatic trimming

Example Session Flow

# 1. Start session
session_id = start_session(metadata={"label": "coding-help"})

# 2. Continue conversation (maintains context)
response1 = continue_session(session_id, "What is Python?")
response2 = continue_session(session_id, "How do I create a function?")  # Remembers previous context

# 3. End session
end_session(session_id, delete=False)  # Keep history, or delete=True to remove

📡 Streaming Responses

The server supports optional streaming for faster response display. When enabled, tokens are sent incrementally as they're generated, rather than waiting for the complete response.

Enabling Streaming

Set STREAMING_ENABLED=true in your .env file:

STREAMING_ENABLED=true
STREAMING_CHUNK_SIZE=50

• STREAMING_ENABLED: Enable/disable streaming (default: false)
• STREAMING_CHUNK_SIZE: Approximate characters per chunk (default: 50). Smaller values = more frequent updates but slightly more overhead.

How It Works

When streaming is enabled:

• generate_text, chat, complete, and continue_session tools return multiple TextContent chunks
• Each chunk contains a portion of the generated text
• The client (Cursor) can display text incrementally as it arrives
• For continue_session, the full accumulated text is still persisted to session history after streaming completes

Performance Notes

• Streaming adds minimal CPU overhead (just chunking logic)
• Response quality is unchanged - streaming only affects delivery timing
• On slower machines, consider using smaller models (1B-3B) with streaming enabled for best experience
• Streaming works with both CPU and GPU inference

Disabling Streaming

Set STREAMING_ENABLED=false (or omit it) to return complete responses in a single chunk, matching the original behavior.

🐛 Troubleshooting

Error: "Model not found"

• Verify MODEL_PATH in .env is correct
• Use absolute paths on Windows
• Ensure the .gguf file exists

Error: "llama-cpp-python not installed"

• Install with: pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
• For GPU: pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121

Server doesn't appear in Cursor

• Check the path in your MCP configuration
• Use absolute paths with double backslashes (\\) or forward slashes
• Restart Cursor after adding configuration
• Check Cursor's Output panel (MCP logs) for errors

Slow performance

• Use smaller models (1B-3B) for CPU-only setups
• Set N_GPU_LAYERS=-1 in .env if you have NVIDIA GPU
• Adjust N_THREADS to match your CPU cores
• Reduce CONTEXT_SIZE if you don't need long context

Compilation errors on Windows

• See INSTALL_COMPILER_WINDOWS.md for installing Visual Studio Build Tools
• Or use pre-built wheels (recommended)

📁 Project Structure

local-llm-mcp-tool/
├── server.py              # Main MCP server (standard API)
├── server_fastmcp.py      # Alternative server (FastMCP, simpler)
├── scripts/               # Helper and setup scripts
│   ├── download_model.py      # Model download helper
│   ├── example_usage.py       # Usage examples
│   ├── install_llama.bat      # Batch installer
│   ├── install_llama.ps1      # PowerShell installer
│   ├── suggest_model.py       # Script to suggest a model based on hardware
│   └── test_server.py         # Setup test script
├── requirements.txt       # Python dependencies
├── .env.example          # Configuration template
├── .gitignore           # Git ignore rules
└── README.md            # This file

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• llama.cpp - The core inference engine
• llama-cpp-python - Python bindings
• Model Context Protocol - The MCP specification
• Cursor - The IDE that makes this useful

🔮 Future Ideas

See FUTURE_IDEAS.md for planned features:

• ✅ Conversation history/sessions - Implemented!
• ✅ Streaming responses - Implemented!
• RAG (document Q&A)
• Multiple model support
• Session summarization for long conversations
• And more...

---

Made with ❤️ for privacy-conscious developers who want local AI without the cloud.