Hybrid Vision

Privacy-First Visual Memory System for MCP Agents

Project Mission

Hybrid Vision is a local "Visual Memory" server for AI Agents (Claude Desktop, Cursor, etc.). It acts as a Semantic Recorder that efficiently captures, indexes, and lets agents query your screen history without sending pixels to the cloud. Think of it as "Open Recall" for your AI assistant—running entirely on your local hardware.

Architecture

• Core Logic: Rust (Async/Tokio)
• MCP SDK: Official rmcp crate v0.9
• Memory System: In-memory Ring Buffer (metadata only) + SQLite with vector embeddings
• OCR Engine: PaddleOCR via ONNX Runtime (DBNet + SVTR)
• Embeddings: all-MiniLM-L6-v2 via ONNX Runtime (384-dim vectors)
• Protocol: Model Context Protocol (Stdio/JSON-RPC)
• Parallelization: tokio::spawn_blocking + channels for non-blocking OCR
• Hardware Target: Consumer CPU (optimized for multi-core)

Available Tools

The server exposes the following MCP tools:

Tool Name	Description	Parameters
read_screen	Returns last captured frame with timestamp & OCR text	None
get_screen_history	Retrieves screen history from last N minutes	minutes (default: 5), limit (default: 10)
search_screen_history	Full-text or semantic search across history	query, limit, semantic (bool)

---

Quick Start

See SETUP.md for detailed installation and configuration instructions.

Prerequisites

• Rust toolchain (1.75+)
• Linux: build-essential libssl-dev pkg-config libx11-dev libxcb1-dev libdbus-1-dev
• Windows: Visual Studio Build Tools with C++ support
• macOS: Xcode Command Line Tools

Build

cargo build --release

Configure Claude Desktop

Edit claude_desktop_config.json:

Linux/macOS:

{
  "mcpServers": {
    "hybrid-vision": {
      "command": "/absolute/path/to/hybrid-vision/target/release/hybrid-vision-server",
      "args": []
    }
  }
}

Windows:

{
  "mcpServers": {
    "hybrid-vision": {
      "command": "C:\\Users\\<username>\\code\\hybrid-vision\\target\\release\\hybrid-vision-server.exe",
      "args": []
    }
  }
}

Example Usage:

"I had a compile error 5 minutes ago, what was it?"

Expected response: (Agent searches history and returns the error message)

---

Technical Design

Why Stdio Transport?

Standard input/output is the fastest way for local agents (Claude Desktop, Cursor) to communicate with tools. No HTTP overhead, no network stack, just pure IPC.

Why SQLite?

• Serverless, runs in-process (no separate DB server to manage).
• Native ORDER BY support - no workarounds needed.
• Lightweight (~500KB bundled) and battle-tested.
• Easy debugging with any SQLite viewer (DB Browser, DBeaver, etc.).
• Vector embeddings stored as BLOB with cosine similarity in Rust.

Why Ring Buffer?

• Keeps RAM usage predictable (~1GB limit).
• Avoids filling up the hard drive with video files.

---

Project Structure

hybrid-vision/
├── src/
│   ├── main.rs              # Entry point, rmcp stdio transport
│   ├── server.rs            # MCP server with tool definitions
│   ├── recorder.rs          # Memory-efficient ring buffer
│   ├── screen_recorder.rs   # Parallel capture with spawn_blocking OCR
│   ├── memory.rs            # SQLite with embedding support
│   ├── embedding.rs         # all-MiniLM-L6-v2 text embeddings
│   ├── config.rs            # Unified YAML configuration
│   ├── vision.rs            # Screen capture abstraction
│   └── ocr.rs               # PaddleOCR text extraction
├── config.yml               # User configuration
├── data/
│   ├── memory.db            # SQLite database (gitignored)
│   └── hybrid-vision.log    # Log file (gitignored)
├── Cargo.toml               # Rust dependencies
├── README.md                # This file
└── SETUP.md                 # Installation & configuration guide

---

Testing

Manual Test

# Run the server (Linux/macOS)
./target/release/hybrid-vision-server
# Windows: .\target\release\hybrid-vision-server.exe

# Send a JSON-RPC initialize message (paste and press Enter)
{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}

With Claude Desktop

1. Configure as shown in SETUP.md
2. Restart Claude Desktop completely
3. Ask: "What tools do you have available?"
4. Should see read_screen, get_screen_history, and search_screen_history listed
5. Ask: "What was I doing 5 minutes ago?"

---

License

MIT License. Copyright (c) 2025 Yashasvi Asthana.