Windeep Computer Control

Production-Grade Windows Desktop Automation Engine
生产级 Windows 桌面自动化引擎

Intelligent routing, fault self-healing, UIA COM, SendInput, virtual display — fully autonomous Windows control. 智能路由、故障自愈、UIA COM 直调、SendInput 真实击键、虚拟显示器——完全自主的 Windows 控制。

License Python Windows Version

---

🌟 Features | 特性

Core Architecture | 核心架构

Feature	Description
Route Integrity Guard	3-function bytecode SHA256 fingerprint — SSDT-style protection against runtime hooking
Fault Lockdown	Auto-lock after 3 consecutive route failures, energy-linked auto-recovery
Energy Model	3-zone ATP metabolism (HIGH/WARNING/COMA) with automatic enter_comath() recovery
Async Verification	_verifier_loop background thread — settle → capture → element assertion → auto lockdown
NT Kernel Mapping	IRQL → input routing priority, SSDT → route guard, Object Manager → layout knowledge

Input Layer | 输入层

Feature	Description
SendInput Real Keystrokes	send_click(x, y), send_type_text(text), send_hotkey([ctrl], 's') — kernel-level events
UIA COM Direct	_uia_com_invoke() — InvokePattern / ExpandCollapsePattern / ValuePattern via Python comtypes
PostMessage Fallback	Background-safe WM_CHAR/WM_CLICK for Win32 controls
XAML Click Adapter	4-tier strategy: comtypes UIA COM → dispatch=auto → foreground SendInput → PostMessage

Capture Layer | 截图层

Feature	Description
DXGI Desktop Duplication	DirectX-based full-screen capture, works on Electron/DirectX windows
PrintWindow API	Occluded/minimized Win32 window capture
BitBlt + MSS	Multi-monitor visible content capture
PIL ImageGrab	Universal fallback

System Infrastructure | 基础设施

Feature	Description
Virtual Display	VirtualDrivers signed driver (v25.7.23) — 2 screens, no test mode required
Clipboard Guard	Multi-format backup/restore (CF_UNICODETEXT + CF_HDROP + CF_DIB)
Resilient Element Matching	Weighted multi-attribute: AutoId(0.50) > Name(0.35) > Class(0.08) > Type(0.05) > Parent(0.02)
Cross-Session Layout Knowledge	LRU disk persistence with fuzzy title matching
UiTreeCache	2s TTL with resilient fallback via element fingerprint
One-Click Deployment	scripts/install_headless.ps1 — 8-step automated setup
Cognitive Unity	cognitive_unity.py — Hermes↔Codex++ unified cognitive loop with ARD Federation

Browser Control | 浏览器控制

Feature	Description
Playwright + Chromium	Direct browser automation — navigate, fill, click, screenshot
CDP Bridge	Chrome DevTools Protocol for multi-tab orchestration

---

📊 Comparison vs Computer Use Plugin | 对比

Dimension	WindEep v3.2	Computer Use Plugin
Route Auto-Degradation	✅ 3-tier SSDT guard	❌ Manual primitive selection
Clipboard Protection	✅ Multi-format guard	❌ None
Async Verify + Self-Heal	✅ _verifier_loop + lockdown	❌ Manual re-snapshot
Virtual Display	✅ 2 screens (signed driver)	❌ Physical desktop only
UIPI Elevation	✅ _get_window_integrity + _uia_route	❌ Not addressed
Cross-Session Memory	✅ LayoutKnowledge LRU	❌ Stateless
SendInput Real Keystrokes	✅ scripts/sendinput.py	✅ Native
WGC Occlusion Capture	✅ DXGI + PrintWindow + BitBlt	✅ Windows.Graphics.Capture
Browser Control	✅ Playwright + Chromium	✅ CDP multi-tab
macOS / iOS	❌ Windows only	✅

Overall: Hermes wins 9 : 4 for Windows desktop control (Codex++ evaluation, 2026-06-29).

---

🚀 Quick Start | 快速开始

One-Click Headless Deployment | 一键部署

# Run as Administrator | 以管理员身份运行
PowerShell -ExecutionPolicy Bypass -File scripts\install_headless.ps1

This script:

1. ✅ Checks admin privileges
2. ✅ Verifies Python availability
3. ✅ Downloads & installs VirtualDrivers signed display driver
4. ✅ Validates the windeep project
5. ✅ Runs Python syntax checks on all modules
6. ✅ Checks route integrity
7. ✅ Verifies DXGI screenshot capability
8. ✅ Creates optional startup scheduled task

Manual Setup | 手动安装

# 1. Install dependencies | 安装依赖
pip install comtypes Pillow mss playwright
python -m playwright install chromium

# 2. Verify virtual display | 验证虚拟显示器
python -c "import ctypes; u=ctypes.windll.user32; print(f'Screens: {u.GetSystemMetrics(80)}')"
# Expected: Screens: 2 (with virtual display) | 预期: 2屏(含虚拟显示器)

# 3. Test route integrity | 测试路由完整性
python computer_control_enhanced.py route-status
# Expected: integrity ✅ OK, locked routes ✅ none

# 4. Start MCP server | 启动MCP服务器
python winctl_mcp_server.py --port 59322

# 5. Register with Hermes | 注册到Hermes
hermes mcp add winctl --url http://127.0.0.1:59322

CLI Usage | 命令行使用

# Route health | 路由健康检查
python computer_control_enhanced.py route-status --json

# Click with UIA COM (XAML-safe) | UIA COM点击(XAML安全)
python computer_control_enhanced.py click --pid 1234 --element 6

# Type text | 输入文本
python computer_control_enhanced.py type-text --pid 1234 "hello world"

# Screenshot | 截图
python computer_control_enhanced.py screenshot --pid 1234

# Cache info | 缓存信息
python computer_control_enhanced.py cache-info

Python API

from computer_control_enhanced import (
    capture, click, type_text, paste_text,
    type_keys, scroll, drag, list_apps,
    _check_routing_integrity, _get_route_status,
    _uia_com_invoke, _xaml_click,
)

# Check system health | 检查系统健康
assert _check_routing_integrity()["ok"]
status = _get_route_status()

# XAML-safe click via UIA COM | UIA COM点击(XAML安全)
result = _uia_com_invoke(
    pid=1234,
    element_name="文件(F)",   # File menu in Chinese Notepad
    hwnd=0x12345678
)

# SendInput real click | SendInput真实点击
from scripts.sendinput import send_click, send_type_text, send_hotkey
send_click(x=500, y=300)
send_type_text("Hello from Hermes!")
send_hotkey(['ctrl'], 's')

# Capture with occlusion support | 遮挡窗口截图
result = _capture_window_wgc(pid=1234, window_id=0x12345678)

# Async verification | 异步验证
_enqueue_verify(pid=1234, window_id=12345678, settle_ms=200)
# Returns immediately, verifier runs in background thread

---

🏗 System Architecture | 系统架构

windeep/                          ← Project root | 项目根目录
│
├── computer_control_enhanced.py  ← Main orchestration | 主编排层 (1425+ lines)
│                                      Route guard / Energy model / Async verify
│                                      UIA COM / XAML click / WGC capture
│
├── winctl_mcp_server.py          ← MCP Server (22 tools, HTTP :59322)
│
├── compress_image.py             ← Progressive compression pipeline | 渐进式压缩管线
│
├── scripts/
│   ├── sendinput.py              ← SendInput real keystrokes | 真实击键 (NEW v3.1)
│   ├── dxgi_capture.py           ← DXGI + fallback capture | DXGI截图管线 (NEW v3.2)
│   ├── clipboard_guard.py        ← Multi-format clipboard guard | 剪贴板守卫
│   ├── resilient_matcher.py      ← Weighted element matching | 弹性元素匹配
│   ├── layout_knowledge.py       ← Cross-session layout persistence | 布局知识
│   ├── ui_tree_cache.py          ← UI tree cache | UI树缓存
│   ├── element_fingerprint.py    ← SHA256 element fingerprints | 元素指纹
│   ├── ocr_finder.py             ← WinRT OCR engine | 原生OCR
│   ├── smart_matcher.py          ← 5-strategy smart matching | 智能匹配
│   ├── assertion_verifier.py     ← 4-type assertion system | 断言系统
│   ├── shared_ui_state.py        ← Cross-agent shared state | 共享状态
│   ├── action_queue.py           ← Deferred action queue | 延迟队列
│   └── install_headless.ps1      ← One-click deploy | 一键部署

Input Routing Pipeline | 输入路由管线

UIA COM (comtypes, 10ms)
  ├─ ✅ XAML menus, buttons, text fields
  ▼ (fails)
SendInput (real keystrokes, 50ms)
  ├─ ✅ Electron/SPA/UWP apps
  ▼ (fails)
PostMessage (background-safe, 1ms)
  ├─ ✅ Win32 native controls
  ▼ (fails)
ctypes BM_CLICK (last resort)

Capture Pipeline | 截图管线

DXGI Desktop Duplication
  ├─ ✅ Full desktop, occluded windows, Electron/DirectX
  ▼ (fails)
PrintWindow API
  ├─ ✅ Occluded Win32 windows
  ▼ (fails)
BitBlt + MSS
  ├─ ✅ Multi-monitor visible content
  ▼ (fails)
PIL ImageGrab
  ├─ ✅ Universal full-desktop fallback

---

⚡ Performance | 性能

Operation	Latency	Notes
Route integrity check	<0.1ms	Bytecode SHA256
Route status query	<0.05ms	Lock/energy/stats
UIA COM invoke	10-350ms	First call includes comtypes init
SendInput click	~1ms	Kernel-level event
Clipboard guard	~2ms	Backup → set → restore
Layout knowledge	~3-6ms	Learn + lookup
get_window_state (MCP)	10-100ms	DXGI + UIA tree
type_text (10 chars)	~300ms	30ms/char
capture (DXGI)	10-50ms	Full screen

---

📋 Requirements | 依赖

Mandatory | 必须

• OS: Windows 10+ (x64)
• Python: 3.10+
• comtypes: pip install comtypes (for UIA COM)

Recommended | 推荐

• Virtual Display Driver: VirtualDrivers/Virtual-Display-Driver v25.7.23 (signed, no test mode)
• Pillow: pip install Pillow (screenshot compression)
• mss: pip install mss (multi-monitor capture)
• Playwright: pip install playwright && python -m playwright install chromium (browser control)
• Hermes Agent: Hermes Agent (for MCP integration)

Zero Dependencies | 零依赖

Core functions (window management, input control, UIA COM) work via ctypes + comtypes only. No npm, no Node.js, no .NET runtime required.

---

🧪 Verification | 验证

# Run full system audit | 全系统审计
python -c "
import computer_control_enhanced as cce
# Route integrity | 路由完整性
assert cce._check_routing_integrity()['ok']
# Energy | 能量
assert cce._get_route_status()['energy_ok']
# No locked routes | 无锁定
assert not cce._get_route_status()['locked_routes']
# UIA COM | UIA COM可用
assert hasattr(cce, '_uia_com_invoke')
# XAML click | XAML点击可用
assert hasattr(cce, '_xaml_click')
# WGC capture | 截图可用
assert hasattr(cce, '_capture_window_wgc')
print('ALL CHECKS PASSED')
"

---

📜 License | 许可证

Apache 2.0

---

🤝 Architecture Credits | 架构贡献

• Hermes Agent (Nous Research) — Agent runtime & MCP framework
• cua-driver — Windows desktop control backend (38 MCP tools)
• Codex++ (bobliang1979) — Cognitive bridge, UIA COM research, architecture audit
• VirtualDrivers — Signed virtual display driver
• Computer Use Plugin (OpenAI) — Reference architecture for comparison

---

© 2026 BOBLIANG. All rights reserved. Built by Hermes Agent × Codex++