Computer Use Engine

License Python 3.11+ MCP

English | 한국어

The missing safety layer between AI agents and your desktop.

CUE is safety middleware for AI computer use agents. It monitors, enforces policies, and provides guardrails so that AI agents can interact with your desktop — safely and compliantly.

---

Problem

AI computer use is accelerating fast, but safety infrastructure hasn't kept up:

• Rapid adoption — 40% of enterprises plan to deploy AI computer use agents by end of 2026
• No guardrails — only 50% of organizations have any safety controls for autonomous agents
• Regulation is coming — the EU AI Act (Aug 2026) mandates human oversight for high-risk AI systems

There is no open-source framework that sits between AI agents and the desktop to enforce safety policies. CUE fills that gap.

What CUE Does

┌─────────────────────────────────┐
│  Any AI Agent                   │  Claude, GPT, Agent-S, ...
│  (natural language in/out)      │
└──────────┬──────────────────────┘
           │ MCP protocol (stdio)
┌──────────▼──────────────────────┐
│  CUE — Computer Use Enforcer   │  ← this project
│  ┌────────────────────────────┐ │
│  │ Policy Engine              │ │  risk classification, action filtering
│  │ Guardrails                 │ │  app blocklist, key blocking, rate limits
│  │ Audit Logger               │ │  JSONL compliance trail
│  │ Monitor (coming soon)      │ │  real-time dashboard & event streaming
│  └────────────────────────────┘ │
└──────────┬──────────────────────┘
           │ pyautogui / pygetwindow / pywin32
┌──────────▼──────────────────────┐
│  Desktop OS                     │
└─────────────────────────────────┘

Core Value Proposition

Capability	Status	Description
Action Guardrails	Available	App blocklist, key blocking, per-session action limits
Audit Logging	Available	Every action logged to JSONL for compliance review
FAILSAFE	Available	Mouse to (0,0) aborts immediately
Policy Engine	Phase 1	Risk classification, rule-based action filtering
Real-time Monitor	Phase 2	Live dashboard with event streaming
Human-in-the-Loop	Phase 3	Approval workflows for high-risk actions
Agent Adapters	Phase 4	Agent-agnostic backends (Claude, GPT, open-source)
Compliance Reports	Phase 5	Automated audit reports for EU AI Act, SOC 2

Quick Start

1. Install

git clone https://github.com/yonghwan1106/computer-use-engine.git
cd computer-use-engine
pip install -e .

2. Register with Claude

python scripts/register.py

This automatically adds CUE to both Claude Desktop and Claude Code configurations.

3. Restart Claude and go

Restart Claude Desktop or Claude Code. Then just ask:

"Take a screenshot of my screen"

"Open Notepad and type 'Hello, CUE!'"

"Show me all open windows"

Current Features

MCP Tools (12)

Screenshot & Screen

Tool	Description	Parameters
cue_screenshot	Capture full screen or a region as JPEG	region_x, region_y, region_width, region_height (all optional)
cue_screen_size	Get screen resolution	—
cue_cursor_position	Get current cursor coordinates	—

Mouse

Tool	Description	Parameters
cue_click	Click at coordinates	x, y, button (left/right/middle), clicks (1-3)
cue_scroll	Scroll at position	x, y, clicks (positive=up, negative=down)
cue_move	Move cursor	x, y
cue_drag	Drag from point A to B	start_x, start_y, end_x, end_y, button, duration

Keyboard

Tool	Description	Parameters
cue_type	Type text (auto clipboard fallback for non-ASCII like Korean/CJK)	text
cue_key	Press key or combo	key (e.g. "enter", "ctrl+c", "alt+tab")

Window Management

Tool	Description	Parameters
cue_list_windows	List all visible windows with geometry	—
cue_focus_window	Focus a window by partial title match	title
cue_window_info	Get active window info	—

Safety Features

Feature	Description	Default
Action limit	Max actions per session before requiring reset	100
App blocklist	Prevents interaction with sensitive apps	Registry Editor, Windows Security
Key blocklist	Blocks dangerous key combos	win+r, ctrl+alt+del
Audit log	Every action logged to JSONL file	cue_audit.jsonl
FAILSAFE	Move mouse to (0, 0) to abort immediately	Enabled
Action delay	Pause between actions for safety	50ms

Safety Policy

All safety settings are configurable in config/default.yaml:

safety:
  max_actions_per_session: 100
  action_delay: 0.05
  failsafe: true
  allowed_apps: []
  blocked_apps:
    - "Windows Security"
    - "Registry Editor"
    - "Task Manager"
  blocked_keys:
    - "win+r"
    - "alt+f4"
    - "ctrl+alt+del"

Manual Registration

If you prefer to configure manually instead of using register.py:

Claude Desktop — edit %APPDATA%/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "cue": {
      "command": "python",
      "args": ["-m", "cue"]
    }
  }
}

Claude Code — edit ~/.claude/settings.json:

{
  "mcpServers": {
    "cue": {
      "command": "python",
      "args": ["-m", "cue"]
    }
  }
}

Project Structure

computer-use-engine/
├── cue/
│   ├── __init__.py            # Package version
│   ├── __main__.py            # python -m cue entry point
│   ├── server.py              # FastMCP server initialization
│   ├── tools/
│   │   ├── screenshot.py      # Screen capture tools (3)
│   │   ├── mouse.py           # Mouse control tools (4)
│   │   ├── keyboard.py        # Keyboard input tools (2)
│   │   └── window.py          # Window management tools (3)
│   ├── safety/
│   │   ├── guardrails.py      # Action limits, app/key blocking
│   │   └── logger.py          # JSONL audit logger
│   ├── core/                  # Policy engine, risk scoring (Phase 1)
│   ├── monitor/               # Real-time dashboard (Phase 2)
│   ├── adapters/              # Agent-agnostic backends (Phase 4)
│   └── utils/
│       ├── screen.py          # DPI awareness, image processing
│       └── keymap.py          # xdotool → pyautogui key mapping
├── config/
│   └── default.yaml           # Safety configuration
├── scripts/
│   └── register.py            # Auto-registration for Claude
├── tests/                     # Unit tests
├── pyproject.toml
├── LICENSE                    # Apache 2.0
└── README.md

Development

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Run the server directly (stdio mode)
python -m cue

Roadmap

Phase	Focus	Status
MVP	MCP server, 12 tools, basic guardrails	Done
Phase 1	Policy engine, risk classification, session management	Next
Phase 2	Real-time monitoring dashboard, event streaming	Planned
Phase 3	Human-in-the-loop approval workflows	Planned
Phase 4	Agent-agnostic adapters (Claude, GPT, Agent-S)	Planned
Phase 5	Compliance reports (EU AI Act, SOC 2)	Planned

Requirements

• Python 3.11+
• Windows 10/11
• Claude Desktop or Claude Code with MCP support

Contributing

Contributions are welcome! Please see LICENSE for details.

License

Apache 2.0