feat: add C# and Rust Windows Computer Use feasibility spike

[sunheyi6]

This draft PR adds a language-blind Windows Computer Use feasibility spike for C# System.Windows.Automation and a direct-COM Rust helper, aligned with apache/maka#4318.

Scope

The spike implements a bounded, fail-closed private JSON-RPC protocol for supervised Windows helpers. It covers HWND/PID/process-start-time/window-generation identity, one-shot snapshot and authorization tokens, UIA observation, semantic set_value/click/select/toggle/scroll actions, typed readback, target-window WGC CreateForWindow(HWND) capture, cancellation, hung-provider recovery, parent-death cleanup, and C# SendInput ABI diagnostics. The action tier does not fall back to coordinates, rectangles, PostMessage, foreground forcing, or arbitrary keyboard input. The explicitly named compatibility tier permits only one-shot authorized Unicode/Enter SendInput after exact foreground/focus/identity checks.

Latest cross-machine validation

Validation was run on Windows 11 Pro 10.0.26200, x64, on 2026-09-02. All real-application tests used isolated temporary fixtures, Chrome profiles, user-data directories, or documents. No existing user Chrome profile or user document was opened or modified.

Environment:

• Node v22.16.0
• .NET SDK 10.0.400
• Rust rustc 1.89.0, cargo 1.89.0
• LibreOffice 26.8.0.3 at D:\soft\program\soffice.exe
• Chrome 152.0.7977.x, temporary profiles only
• Repository Electron runtime available
• Microsoft Calculator, Paint, and Notepad AppX packages available

Build and protocol checks:

• C# helper, HangWindow fixture, and WPF fixture build with .NET 10.0.400 with zero warnings and zero errors.
• cargo fmt --check passes.
• cargo clippy --all-targets -- -D warnings passes.
• Rust tests pass: 8 unit tests plus 3 protocol tests.
• C# and Rust each pass lifecycle 34/34 and protocol 3/3; the combined comparison result is 74/74.
• The ASF source-header audit passes for the PR checkout. The header policy now covers the C#/XAML/project formats used by this experiment and explicitly classifies the generated Rust lockfile.

Application matrix:

Matrix	Execution	Contract	Notes
WPF fixture	12 pass, 2 blocked, 2 unknown	16/16 pass	Each helper is 6/8 execution. Typed Enter is intentionally blocked; compatibility Enter remains unknown when helper-side readback is unavailable.
Chromium semantic tasks	18/18 pass	18/18 pass	Three repetitions per helper in force-renderer-accessibility-complete mode.
Chromium navigation	12 pass, 6 unknown	12 pass, 6 not tested	The independent page oracle confirms 18/18 navigations. Helper-side Enter remains unknown and is not upgraded.
LibreOffice	Observation/capture available	Write not tested	Isolated temporary Writer document; no safe document ValuePattern was exposed.
Calculator	Observe and semantic action pass	N/A	Fresh HWND/PID/start-time/window-generation identity was checked.
Notepad	Observe and isolated text action pass	N/A	Temporary document only; Document/ValuePattern was verified.
Paint	Observation smoke pass	Mutation not tested	No mutation was attempted without a safe semantic target.
Electron fixture	Observation smoke pass	Mutation not tested	Local fixture and isolated user-data directory only.

The unknown, blocked, and not tested outcomes are preserved as evidence. A page-oracle success is not rewritten as helper verification, mutation is not retried merely to change an outcome, and identity checks are not weakened.

Published artifacts and performance

The published package is self-contained win-x64, single-file, and trimming-disabled. It is intentionally unsigned and records distributionReady=false; it is experimental evidence, not a production release certification.

• C# maka-cu-windows.exe: 199,642,135 bytes, SHA256 A705BCF778CD15187AD7FECF00AF0017E06279B8393588ECEED557947EEDE576
• Rust maka-cu-windows-rust.exe: 533,504 bytes, SHA256 E297321C6D74F8B34FC66261B21E06FDDE894A7E28CF5DA115C06B10D3190ADD
• HangWindow fixture: 142,690,490 bytes, SHA256 D753288D843A8D6331D6ED33CA792C76B9B2DD8EC7FEFCF2019D0CB716CF61BF
• Three cold handshakes: C# 69–80 ms, Rust 39–41 ms
• Process working set: C# approximately 22.7 MiB, Rust approximately 7.2 MiB
• First fixture WGC frame: C# 62 ms, Rust 239 ms

Changes made after the initial PR validation

• Moved the helper and fixtures from .NET 8 targeting to .NET 10 targeting and corrected runtime metadata.
• Fixed the .NET 10 WgcCapture interop warning.
• Added the installed LibreOffice path and version discovery and recorded isolated application evidence.
• Increased the bounded cold UIA observation budget and changed production actionable discovery to one descendants query with local filtering, while retaining deterministic debug-limit traversal.
• Kept the root observation node lightweight so slow providers cannot consume the tree budget on repeated root property calls.
• Probed ControlType.Document for ValuePattern/InvokePattern so Notepad text actions are represented safely.
• Added current cross-machine result artifacts and the full report at experiments/maka-cu-windows/CROSS-MACHINE-TEST-REPORT-2026-09-02.md.
• Added ASF header coverage and provenance declarations required by the repository CI policy.

Evidence files

• experiments/maka-cu-windows/CROSS-MACHINE-TEST-REPORT-2026-09-02.md
• experiments/maka-cu-windows/comparison-results-cross-machine.json
• experiments/maka-cu-windows/app-task-results-cross-machine.json
• experiments/maka-cu-windows/browser-results-semantic-cross-machine-latest.json
• experiments/maka-cu-windows/browser-results-navigation-cross-machine-latest.json
• experiments/maka-cu-windows/observe-tree-results-cross-machine.json
• experiments/maka-cu-windows/real-app-probe-cross-machine.json

Final boundary

The fixture/protocol baseline and the tested semantic Computer Use paths are reproduced. No additional source defect is indicated by this matrix. This is suitable as clean-machine runtime/package evidence for the experiment, but it is not signed production distribution, arbitrary-Windows-app certification, or a production backend admission. Remaining limitations are the intentionally unknown helper-side Enter result, safe semantic document write coverage where providers expose no reliable ValuePattern, production signing, and independent OS-level parent supervision.

中文翻译

这是一个针对 apache/maka#4318 的语言无关 Windows Computer Use 可行性实验 PR，包含 C# System.Windows.Automation helper 和直接 COM 的 Rust helper。

范围

该实验实现了一个受监督 Windows helper 使用的、有界且失败关闭的私有 JSON-RPC 协议，覆盖 HWND/PID/进程启动时间/windowGeneration 身份、一次性 snapshot 和授权 token、UIA 观察、语义 set_value/点击/选择/切换/滚动动作、typed readback、目标窗口 WGC CreateForWindow(HWND) 截图、取消、卡住 provider 恢复以及父进程退出清理，并包含 C# SendInput ABI 诊断。动作层不会回退到坐标、矩形、PostMessage、强制前台或任意键盘输入。明确命名的 compatibility tier 只有在严格的前台/焦点/身份检查通过后，才允许一次性授权的 Unicode/Enter SendInput。

最新跨机器验证

验证于 2026-09-02 在 Windows 11 Pro 10.0.26200、x64 上完成。所有真实应用测试都使用隔离的临时 fixture、Chrome profile、user-data 目录或文档。没有打开或修改现有用户 Chrome profile 或用户文档。

环境：

• Node v22.16.0
• .NET SDK 10.0.400
• Rust rustc 1.89.0、cargo 1.89.0
• LibreOffice 26.8.0.3，路径为 D:\soft\program\soffice.exe
• Chrome 152.0.7977.x，仅使用临时 profile
• 仓库内 Electron runtime 可用
• Microsoft Calculator、Paint 和 Notepad AppX package 可用

构建和协议检查：

• C# helper、HangWindow fixture 和 WPF fixture 使用 .NET 10.0.400 构建，warning 为 0、error 为 0。
• cargo fmt --check 通过。
• cargo clippy --all-targets -- -D warnings 通过。
• Rust 测试通过：8 个 unit test 加 3 个 protocol test。
• C# 和 Rust 分别通过 lifecycle 34/34 和 protocol 3/3；合计 comparison 结果为 74/74。
• PR checkout 的 ASF source-header audit 通过。header policy 现在覆盖本实验使用的 C#/XAML/project 格式，并明确分类生成的 Rust lockfile。

应用矩阵：

矩阵	Execution	Contract	说明
WPF fixture	12 pass、2 blocked、2 unknown	16/16 pass	每个 helper 为 6/8 execution。typed Enter 按设计为 blocked；helper 侧 readback 不可用时，compatibility Enter 保持 unknown。
Chromium 语义任务	18/18 pass	18/18 pass	每个 helper 在 force-renderer-accessibility-complete 模式下重复 3 次。
Chromium 导航	12 pass、6 unknown	12 pass、6 not tested	独立 page oracle 确认 18/18 导航。helper 侧 Enter 保持 unknown，不会升级。
LibreOffice	观察/截图可用	写入未测试	隔离的临时 Writer 文档；没有暴露安全的文档 ValuePattern。
Calculator	观察和语义动作通过	N/A	已检查新的 HWND/PID/启动时间/windowGeneration 身份。
Notepad	观察和隔离文本动作通过	N/A	仅使用临时文档；已验证 Document/ValuePattern。
Paint	观察 smoke 通过	修改未测试	没有安全语义目标时没有尝试修改。
Electron fixture	观察 smoke 通过	修改未测试	仅使用本地 fixture 和隔离 user-data 目录。

unknown、blocked 和 not tested 结果均作为证据保留。page oracle 成功不会被改写成 helper verified，不会为了改变结果而重试变更动作，也不会放宽身份检查。

发布产物和性能

发布包是 self-contained win-x64、single-file 且禁用 trimming。它有意保持 unsigned，并记录 distributionReady=false；这是实验性证据，不是生产发行版认证。

• C# maka-cu-windows.exe：199,642,135 bytes，SHA256 A705BCF778CD15187AD7FECF00AF0017E06279B8393588ECEED557947EEDE576
• Rust maka-cu-windows-rust.exe：533,504 bytes，SHA256 E297321C6D74F8B34FC66261B21E06FDDE894A7E28CF5DA115C06B10D3190ADD
• HangWindow fixture：142,690,490 bytes，SHA256 D753288D843A8D6331D6ED33CA792C76B9B2DD8EC7FEFCF2019D0CB716CF61BF
• 三次冷启动握手：C# 69–80 ms，Rust 39–41 ms
• 进程工作集：C# 约 22.7 MiB，Rust 约 7.2 MiB
• fixture 首帧 WGC 截图：C# 62 ms，Rust 239 ms

初始 PR 验证后的改动

• 将 helper 和 fixtures 从 .NET 8 target 迁移到 .NET 10 target，并修正 runtime metadata。
• 修复 .NET 10 下的 WgcCapture interop warning。
• 增加已安装 LibreOffice 的路径和版本发现，并记录隔离应用证据。
• 增加有界的冷启动 UIA 观察预算，并将生产 actionable discovery 改为一次 descendants query 加本地过滤，同时保留确定性的 debug-limit traversal。
• 保持 root observation node 轻量化，避免慢 provider 在重复 root property 调用上耗尽 tree budget。
• 对 ControlType.Document 探测 ValuePattern/InvokePattern，使 Notepad 文本动作可以安全呈现。
• 增加当前跨机器结果 artifacts，并新增完整报告 experiments/maka-cu-windows/CROSS-MACHINE-TEST-REPORT-2026-09-02.md。
• 增加仓库 CI policy 要求的 ASF header 覆盖和 provenance 声明。

证据文件

• experiments/maka-cu-windows/CROSS-MACHINE-TEST-REPORT-2026-09-02.md
• experiments/maka-cu-windows/comparison-results-cross-machine.json
• experiments/maka-cu-windows/app-task-results-cross-machine.json
• experiments/maka-cu-windows/browser-results-semantic-cross-machine-latest.json
• experiments/maka-cu-windows/browser-results-navigation-cross-machine-latest.json
• experiments/maka-cu-windows/observe-tree-results-cross-machine.json
• experiments/maka-cu-windows/real-app-probe-cross-machine.json

最终边界

fixture/protocol 基线以及已测试的语义 Computer Use 路径均已复现。该矩阵没有显示出仍需修复的源代码缺陷。它可以作为本实验的 clean-machine runtime/package 证据，但不是签名生产发行版、任意 Windows 应用认证或生产 backend 准入证据。剩余限制包括有意保留的 helper 侧 Enter 结果 unknown、provider 没有可靠 ValuePattern 时的安全语义文档写入覆盖、生产签名以及独立的 OS 级父进程监督。