Desktop GUI Automation

Every other tool needs a browser or a vision model. This one reads what the OS already knows.

A cross-platform MCP server for desktop GUI automation — driven by accessibility APIs, not screenshots.

Works with Claude, Codex, Gemini, Ollama, or any OpenAI-compatible client that supports MCP.

bad-ass-mcp recording itself interacting with MarkTheCrab

Why

Most AI desktop automation tools work by taking screenshots and asking a vision model "what do you see?" That's slow, expensive, and fragile.

Every major OS exposes a structured accessibility tree — the same one used by screen readers — that describes every button, text field, combo box, and menu in every running application. bad-ass-mcp speaks that language directly.

• macOS: AXUIElement via PyObjC
• Linux: AT-SPI2 via gi.repository.Atspi
• Windows: UI Automation via comtypes

Actions fire on control objects directly, so the target window doesn't need focus. The user can keep working while automation runs in the background.

Tools

Tool	Description
list_windows	List all visible application windows
get_tree	Full accessibility tree for a window as nested JSON
find_elements	Find elements by role and/or name
click	Click / invoke an element (foreground-independent)
click_at	Click at absolute screen coordinates — fallback for webview/canvas UI
type_text	Type text into a field (SetValue → EditableText → key injection)
select_option	Select an option in a combo box or list
get_value	Get the current value or text of an element
wait_for_window	Wait until a window matching a pattern appears
wait_for_element	Wait until an element exists and optionally has a given state
screenshot	Capture a PNG (write to output_path or return base64)
start_recording	Start recording the screen (or a specific window)
stop_recording	Stop recording and export as a GIF
learn_layout	Resolve semantic names → live handle IDs (one call per session)
run_sequence	Execute a list of actions server-side — no per-action round-trips

Webviews: waking lazy accessibility trees

Chromium-based apps (Electron, CEF, and every Chromium browser) have a full accessibility tree — they just don't build it until an assistive technology announces itself. Until then the window looks canvas-only: get_tree returns nothing and screenshot-clicking is the only option.

bad-ass-mcp announces itself. When a window's a11y tree comes back empty and the process looks like a webview, the backend performs the platform's "a screen reader is here" handshake, waits a beat, and re-probes before giving up:

• Linux: sets org.a11y.Status.ScreenReaderEnabled on the session D-Bus at server start so Chromium/Electron apps launched afterwards enable accessibility from birth. Windows that stay hollow (an application+frame whose phantom child fetches as None) are detected via the AT-SPI toolkit name and reported accessible: false: the app either launched before the flag went up or opts out on its own. Restarting the app with the server running should pick the flag up per Chromium's documented behavior, but note that is not yet verified here: the only Linux app live-tested so far (flatpak Vivaldi) ignores the flag entirely, its own first-run assistive-technology toggle included, and only produces a tree when launched with --force-renderer-accessibility.
• macOS: sets AXManualAccessibility on the app (Electron's documented switch), falling back to AXEnhancedUserInterface (what VoiceOver sets, watched by plain Chrome/CEF). The attribute set only succeeds on apps that support it, so it's a free no-op elsewhere.
• Windows: sets the system SPI_SETSCREENREADERRUNNING flag, then queries the Chrome_RenderWidgetHostHWND child window — Chromium hangs its DOM tree off that renderer child, not the top-level HWND (which only shows empty panes). get_tree / find_elements are routed through the renderer child automatically.

One wake attempt per process per server run. If list_windows still reports accessible: false after this, the window is genuinely canvas-only (custom OpenGL/Vulkan surface, immediate-mode toolkit without AccessKit) — fall back to screenshot + click_at.

The tradeoff: a woken app spends some memory/CPU maintaining its tree, which is exactly why Chromium lazy-loads it. The wake targets the app you're automating, not the whole desktop (the Linux flag is technically session-global, but only a11y-aware apps respond, and only on demand).

Installation

Requirements: Python 3.11+, ffmpeg + gifsicle for recording (optional)

# macOS
pip install 'bad-ass-mcp[macos]'

# Linux — install system PyGObject + AT-SPI bindings first, then:
pip install bad-ass-mcp

# Windows
pip install 'bad-ass-mcp[windows]'

Linux: PyGObject and AT-SPI are not on PyPI — install them from your distro:

# Debian/Ubuntu
sudo apt install python3-gi gir1.2-atspi-2.0 at-spi2-core

# Arch
sudo pacman -S python-gobject at-spi2-core

# Fedora
sudo dnf install python3-gobject at-spi2-core

Then make sure AT-SPI is enabled (most desktop environments — GNOME, KDE, XFCE — have it on by default):

# Check
gsettings get org.gnome.desktop.interface toolkit-accessibility

# Enable
gsettings set org.gnome.desktop.interface toolkit-accessibility true

If you're using a venv, create it with --system-site-packages so it can see the distro-installed gi:

python -m venv --system-site-packages venv

Windows: Elevated (admin) applications can only be automated from an elevated Python process. No special permissions are needed for normal apps.

Register with Claude Code

claude mcp add bad-ass-mcp --scope user -- bad-ass-mcp

Or manually in ~/.claude.json:

{
  "mcpServers": {
    "bad-ass-mcp": {
      "type": "stdio",
      "command": "bad-ass-mcp"
    }
  }
}

Usage

list the windows on screen
→ [{ "id": "12345", "name": "Firefox", ... }]

find the search box in window 12345 and type "hello"
→ find_elements(window_id="12345", role="entry")
→ type_text(handle_id="...", text="hello")

Electron/CEF apps get their lazy accessibility tree woken automatically (see Webviews above). For apps that still don't expose a clean tree, type_text falls back to platform-native key injection automatically (AT-SPI on Linux, Quartz events on macOS, PostMessage on Windows).

Architecture

bad_ass_mcp/
├── server.py          # FastMCP tool definitions
├── types.py           # WindowInfo, ElementHandle, ActionResult, StaleHandleError
└── backend/
    ├── base.py        # Abstract DesktopBackend interface
    ├── linux.py       # AT-SPI2 + xdotool
    ├── macos.py       # AXUIElement + Quartz
    └── windows.py     # UI Automation + ctypes Win32

The abstract backend interface means adding a new platform is just implementing one class. The full contract-test suite runs against every backend via FakeBackend.

Releasing

Tag a version and the release workflow builds and attaches the wheel automatically:

git tag v0.2.0 && git push --tags

License

MIT