Macos Computer Use Kit

screenshot → guess coordinates → click → hope      ✗  slow, fragile, moves your mouse
snapshot   → act on #ref       → verify the diff   ✓  what this kit does

• One command per client. Works with Claude Code, Claude Desktop, Codex, Cursor, Gemini CLI, opencode, pi and DeepSeek Harness. It is a stdio MCP server with no runtime dependencies beyond pyobjc and Pillow.
• Accessibility first. Each element comes back with a stable #ref, its role, its label and exact screen geometry. Re-observing returns only the diff. Electron and Chromium apps get their full tree turned on automatically.
• Background input. Clicks, keys, scrolls, drags and Unicode typing are posted to the target process, so the user can keep working. CJK, emoji and accents arrive intact, and the clipboard is left alone.
• Verified, never optimistic. Every result separates action_sent from verified. Failures say whether it is safe to retry, whether to reobserve, or whether to never retry.
• Safe by default. There is a built-in deny list for password managers and system auth. The kit refuses to type into a password field that holds Secure Event Input, redacts secure fields, and blocks lock, log-out and force-quit chords. It also offers a dry-run mode, a size-only audit log and app allow/deny lists.
• Vision only when needed. Set-of-mark screenshots draw numbered marks linked to refs, and on-device Apple Vision OCR returns screen coordinates. Both are fallbacks for canvas, game and custom-drawn UI.

Contents

• Quick start
• Tools
• How an agent should use it
• Safety and privacy
• CLI reference
• Other agent integrations
• Jev semantic guards (optional)
• Related projects
• Known limitations
• Repository layout

Quick start

Requirements

• macOS 12 or later, Python 3.10 or later.
• Grant two permissions to the app that runs your agent (your terminal, Claude Desktop, Cursor, and so on). macos-cu doctor tells you exactly which app is missing which permission.
  • Accessibility: needed for reading the tree, AX actions and posted input.
  • Screen Recording: needed for screenshots and OCR (without it every frame is black).

Claude Code

Pick one of these:

# A. As a plugin: MCP server, skill and /macos-doctor command
/plugin marketplace add Sur-Cai/macos-computer-use-kit
/plugin install macos-computer-use@macos-computer-use-kit

# B. As a plain MCP server
claude mcp add --scope user macos-computer-use -- uvx macos-computer-use-kit mcp

The plugin finds macos-cu on your PATH and otherwise falls back to uvx or pipx run. Nothing has to be pre-installed except uv or pipx.

Every other client, with one command

pipx install macos-computer-use-kit          # or: pip install / uv tool install
macos-cu setup claude-code                   # also: claude-desktop, codex, cursor, gemini, opencode
macos-cu setup skill                         # copy the agent skill to Claude, Codex and opencode
macos-cu doctor                              # permissions, displays, OCR, policy

setup is idempotent and supports --dry-run. JSON configs are merged, so your other servers are preserved, and a .bak copy is written before the first change. --read-only registers only the observation tools.

{
  "mcpServers": {
    "macos-computer-use": {
      "command": "uvx",
      "args": ["macos-computer-use-kit", "mcp"]
    }
  }
}

[mcp_servers.macos-computer-use]
command = "uvx"
args = ["macos-computer-use-kit", "mcp"]

Tools

The MCP server exposes 18 tools. Observation tools carry readOnlyHint, so clients can auto-approve them. To trim the list, use mcp --read-only, --tools a,b or --exclude-tools c.

How an agent should use it

macos_doctor                                   once: permissions + display layout
macos_snapshot app="Notes"                     → #a1b2c3d4 AXButton 28x28 @screen[812,64] New Note …
macos_act ref="a1b2c3d4"                       → {"ok":true,"action_sent":true,"verified":true}
macos_type app="Notes" text="周会纪要 ✅"        → Unicode events, clipboard untouched
macos_snapshot app="Notes" diff_against=<path> → only what changed

The rules behind the loop, which the bundled skill teaches:

1. Observe semantically. Snapshot before you screenshot.
2. Act on the element, not the pixel. Prefer menu paths and AX actions, then a click at center_screen, then keyboard shortcuts.
3. Verify, don't sleep. Use a snapshot diff or macos_wait.
4. Never retry blindly. If action_sent is true, re-observe before repeating anything that can't safely run twice. A second "Send" is a second message.
5. stale_ref means re-observe. It never means "pick the nearest element".

Coordinate spaces

Displays placed left of or above the primary have negative coordinates. That is normal, and macos_doctor prints the layout.

Safety and privacy

Everything runs locally. The kit makes no network calls, except the optional Jev guard, which only runs when you configure a key and call it.

Local files stay private to your account:

• Snapshot caches and the audit log live in ~/.cache/macos-computer-use/, or wherever MACOS_CU_CACHE_DIR points.
• They are created with 0700/0600 permissions.
• Snapshots are pruned to the newest 20 (MACOS_CU_SNAPSHOT_KEEP).
• Screenshots and OCR captures are written only where you ask (out=). Temporary OCR captures are deleted.

CLI reference

Every command prints JSON (or compact text for ax tree/find/snapshot), so an agent can drive the kit from any shell. Exit codes are stable: 0 ok, 2 usage or permission, 3 not found / stale ref, 4 capture failed, 5 target changed, 6 refused by policy, 7 timeout.

macos-cu ax snapshot --app Finder --interactive
macos-cu ax press --app Finder --ref 9d2261d7
macos-cu menu select --app Safari --path "File > New Private Window"
macos-cu input type --app Notes --text "你好, world 👋"
macos-cu input click --window-id 12345 --x 171 --y 28 --expect "<pid:wid:x:y:w:h>"
macos-cu shot annotate --app "System Settings" --out /tmp/marks.png
macos-cu ocr --app Preview --text "Total"

Other agent integrations

The MCP server covers most clients. There are native bridges for two harnesses that prefer their own tool format. Both are thin wrappers over the same CLI and call it with argv arrays (shell: false).

Install the CLI first (pipx install macos-computer-use-kit). If an app launched from the Dock can't find it, set MACOS_CU_BIN=/abs/path/to/macos-cu. All artifacts share one version number and are released together. See packages/pi and packages/dsh for details.

Jev semantic guards (optional)

This is the only feature that needs a key. Right before an irreversible step, a small model gives calibrated judgments: is this still the intended recipient? does the field hold the intended text? what blocks the action? Code decides whether to proceed, and the model may only suggest the two recoveries that cannot send anything.

echo '{"task":"send the report to Alice",
       "expected":{"recipient":"Alice","message":"Q3 numbers"},
       "observed":{"chat_title":"Bob","input_text":"Q3 numbers"}}' | macos-cu jev guard
# {"answers":{"right_target":0.02,"input_ok":0.98,"blocker":"wrong_target"},"decision":"switch_target"}

The key is read from TYPESAFE_API_KEY or ~/.config/typesafe/api_key (https://console.typesafe.ai/keys). Question design is covered in skill/reference/jev-best-practices.md.

Related projects

Computer use is a busy space. These are the projects worth knowing, with star counts as of September 2026. Pick the one that fits.

Where this kit fits. It is pure Python on pyobjc, so there's no binary to notarize: uvx runs it anywhere. Its focus is agent-loop reliability. That means verified results with retry semantics, stale-ref detection, snapshot diffs, and a deterministic safety policy. The optional Jev guard adds a calibrated check before irreversible steps. The design follows the patterns mature computer-use agents converged on, reimplemented from scratch.

Known limitations

• macOS only. AX, CGEvent, ScreenCaptureKit and Vision are macOS APIs.
• Custom-drawn UIs (games, canvases, some chat apps) expose little or no accessibility data. Use annotate, ocr and coordinate clicks there, and verify visually.
• Most apps accept background input, but a few only accept typing or pasting while frontmost. Use macos_app action=activate first.
• The user may be working at the same time. One extra verification before an irreversible step is cheap.

Repository layout

src/macos_computer_use/   the CLI + MCP server (single source of truth)
skill/                    the agent skill (canonical; synced by scripts/sync-skill.sh)
plugins/claude-code/      Claude Code plugin (MCP launcher, skill, /macos-doctor)
.claude-plugin/           marketplace manifest for `/plugin marketplace add`
packages/pi/              pi package (skill + native tools)
packages/dsh/             DeepSeek Harness bundle
tests/                    unit tests: policy, chords, diffs, OCR geometry, MCP protocol
tools/*.py                legacy script shims over the same modules

See CONTRIBUTING.md for conventions, PUBLISHING.md for release steps and CHANGELOG.md for the changelog.

License

MIT