feat(browser): add autonomous native previews - #666
Merged
Merged
Conversation
pascalandr
force-pushed
the
feat/browser-automation
branch
from
August 29, 2026 08:25
59b106f to
36e39fb
Compare
pascalandr
changed the title
pascalandr
force-pushed
the
feat/browser-automation
branch
from
August 29, 2026 08:35
36e39fb to
8b6a471
Compare
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/33243527867 Artifacts expire in 7 days.
|
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/33243527867 Artifacts expire in 7 days.
|
pascalandr
force-pushed
the
feat/browser-automation
branch
from
September 4, 2026 12:14
8b6a471 to
adfc03f
Compare
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/33871760485 Artifacts expire in 7 days.
|
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/33878676695 Artifacts expire in 7 days.
|
1 similar comment
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/33878676695 Artifacts expire in 7 days.
|
Expose session-scoped browser open, navigate, snapshot, click, type, and screenshot actions through the existing authenticated CodeNomad automation adapter. Reuse the shared native request transport and session ownership fencing from Developer Automation instead of restoring a separate plugin or route. Implement isolated persistent browser previews in Electron and Windows Tauri with hardened popup, permission, download, navigation, deadline, and stale-reference handling. Keep preview registration and open claims scoped to the owning CodeNomad session across windows. Add the session preview UI, native adapters, localized labels, browser-specific architecture documentation, and focused Electron, Tauri, server, and UI coverage. Developer Automation launch, CDP feedback, and lifecycle code remain in DEV-v2 and are not duplicated here.
Apply rustfmt to the new browser command and navigation wiring in main.rs. No behavioral changes.
Create an agent-requested preview before switching the active session so Tauri never registers the previous workspace preview under the new session. Ignore transient registrations that are still loading another URL. Allow failed or closing claim owners to release browser-open ownership and redistribute the request to another local window in both Electron and Tauri. Prevent the failed owner from immediately reclaiming the same request. Fence Electron click dispatch against navigation before every mouse event, matching Tauri. Add focused regressions for claim transfer, stale click refs, and stale Tauri page-load expectations.
Route renderer commands, navigation, events, client-state flushes, and zoom through the primary Tauri Webview instead of requiring a WebviewWindow. Native window operations now derive the parent Window explicitly, and focus/menu targeting ignores browser child webviews so an open preview cannot displace the CodeNomad renderer. Route agent-requested previews through the active instance when restored tabs expose the same OpenCode session more than once. Use Chromium's selectAll editing command and virtual key metadata before replacement typing in both desktop hosts, preserving clear semantics in WebView2. Add Tauri, Electron, and UI regressions, run the browser adapter test in PR CI, and remove a redundant wall-clock assertion from the registration-contention test while retaining its behavior assertions. Validated with repeated 146-test Tauri runs, the 204-test Electron native suite, desktop typechecks, a release Tauri build, and packaged WebView2 open/navigate/snapshot/type/click/screenshot actions.
pascalandr
force-pushed
the
feat/browser-automation
branch
from
September 6, 2026 21:11
6290822 to
47a56f2
Compare
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/34412291757 Artifacts expire in 7 days.
|
Integrate dev@ea37f97b after the global-session, nested-scroll, and selective-pruning fixes while preserving the native browser-preview implementation. The merge is conflict-free and keeps the PR history intact for exact-head CI validation.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/34486377217 Artifacts expire in 7 days.
|
…s updates Merge dev aa51cbb including the official client migration, shared runtime compatibility and native preferences transition protocol. Resolve the Tauri preferences entrypoint by keeping the primary-Webview authorization used to exclude browser children while forwarding dev's new transition state. Document browser previews alongside the independently reviewed pruning RPC; neither exception opens the generic RPC proxy or restores V1 plugin hosting. Validation: server 502 passed/2 skipped; UI stores 441 passed; Electron native 209 passed/4 skipped after installing its runtime; Tauri 167 passed; packaged server/UI build and server/UI/Electron typechecks passed. Interactive preview automation and non-Windows hosts were not rerun. Whole-crate cargo fmt --check reports an inherited cli_manager.rs log-line formatting difference.
Contributor
Author
Refresh against dev aa51cbbPublished head: merge(dev): retain native browser isolation through V2 and preferences updates Merge dev aa51cbb including the official client migration, shared runtime Validation: server 502 passed/2 skipped; UI stores 441 passed; Electron native Validation above is local execution evidence. CI on this published head and the explicitly untested interactive/platform scenarios remain separate. No PR was merged. |
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/35158893213 Artifacts expire in 7 days.
|
Ship one bundled codenomad.automation V2 plugin for browser and desktop tools in ordinary user projects. All tools follow backend presence, while normal Electron and Windows Tauri startup prepares native instrumentation. Remove the Developer Mode activation toggle and marker without changing authenticated session/window targeting or ownership of the shared OpenCode daemon. Resolve plugin discovery roots from the connected daemon's authenticated config.get response rather than future-start environment overrides. Share installation and lease mechanics with pruning, retain older backends' recorded storage, and use scoped preparation claims to unwind stale or interrupted installation without retiring concurrent valid owners. Close native browser findings involving Tauri child capabilities, multi-webview window counting, primary-renderer reload cleanup, fragment completion, snapshot lock ordering and expired CDP dispatch. Fix UI retry/disposal handling, insecure HTTP preview mounting, Electron guest readiness, redirect feedback and explicit return navigation. Repeated independent UI, native and server reviews report no remaining actionable findings. Validation includes 570 server tests, 214 Electron tests, 180 Tauri tests, 14 real browser/guest regressions, isolated automation contracts on beta-19271 and 2.0.3/2.0.5/2.0.7, native pruning concurrency/payload coverage, production builds and both packaged-resource smokes. Extend CI with the automation matrix and Xvfb rendering.
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/35357948272 Artifacts expire in 7 days.
|
Contributor
Author
|
Gatekeeper follow-up: the complete PR Build Validation run for ad7f2c8 is now green: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/35357948272 . All 21 executed jobs passed, covering Windows/macOS/Linux tests and builds, native pruning, compatibility, and automation runtime contracts on beta-19271, 2.0.3, 2.0.5 and 2.0.7. Tauri Linux ARM64 is the sole skipped job. This closes the cross-platform CI validation boundary recorded above; actual WSL execution and a live packaged WebView2 interaction smoke remain unclaimed. Final independent UI, native and server review passes have no remaining actionable findings. No merge performed. |
|
PR builds are available as GitHub Actions artifacts: https://github.com/NeuralNomadsAI/CodeNomad/actions/runs/35357948272 Artifacts expire in 7 days.
|
pascalandr
added a commit
that referenced
this pull request
Sep 19, 2026
…at reloads (#712) ## Summary Restore the saved project/conversation promptly, preserve restored tabs when later hydration fails, and remove a filesystem heartbeat that repeatedly invalidated OpenCode configuration. ## Root cause and reproduction PR #666 placed desktop plugin presence leases beneath the authenticated OpenCode configuration discovery root. Updating their mtimes every two seconds triggers recursive configuration watchers and expensive revalidation across loaded locations. With CodeNomad closed and the same shared OpenCode daemon left running, updating only a temporary file in that directory reproduced the slowdown: - Quiet: 77–92 ms for the same 200-message request. - File mtime updated every two seconds: 5.3–8.8 seconds. - Updates stopped: 71–138 ms. The pre-#666 server/UI comparison was only a partial baseline: native reads recovered after startup, but the older UI also exhibited restoration failures. ## Implementation - Derive a sibling plugin storage namespace from the connected daemon's authoritative discovery root. Move new heartbeats outside its recursively watched configuration tree. - Migrate existing managed inside-root storage while retaining read-only observation of older backends' leases. Preserve existing outside-root storage, independent backend lifetimes, and native WSL path translation. - Seed saved selection identity before HTTP hydration, prioritize the saved project, and hydrate its conversation directly. Preserve bound project tabs across subsequent hydration errors/timeouts and respect user selection changes. - Load model/agent catalogues when the client becomes ready and retry reads superseded by the initial stream connection. - Publish session lists without waiting for runtime-status reads; merge delayed status results without overwriting SSE updates or local admissions. Use lightweight status reconciliation for liveness. - Bound secondary reads across projects, coalesce Shell refreshes, and prevent duplicate catalogue/Shell refreshes through the native transcript projection. - Preserve directory authorization while avoiding Git mutation-identity resolution on reads; reject unrelated Git common directories before native worktree inventory scans. ## Validation - Targeted UI and server suites covering reconnect races, status authority, background queue cancellation, projection isolation, Shell refresh coalescing, and ownership enforcement. - 19 plugin path/lifecycle/migration/presence tests passed. - Six browser restoration scenarios covering Electron/Tauri, saved selection, user selection, and hydration timeout behavior passed. - Isolated OpenCode 2.0.9 integration fixture passed: native provisioning, plugin availability, independent leases, shutdown/reopening, and OS watcher verification that heartbeats do not modify the configuration tree. A negative control using the old storage placement fails that assertion. - Typechecks, complete server/UI/plugin build, and `git diff --check` passed. - Installed usual-profile Tauri validation on the requested conversation: launch-to-display 3.86 and 5.33 seconds; initial 200-message requests 178–239 ms; latest forced reload 218 ms. No intermediate wrong conversation selection observed. Verified all three saved projects and the real model menu. ## Remaining observed edge case A forced reload racing restoration of an old scroll position can still show the existing history retry control. Retrying succeeded. This PR addresses the reproduced latency trigger and startup restoration races; it does not claim that every scroll-restoration edge case is resolved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
codenomad.automationV2 plugin for browser, inspect, act and screenshot. All tools are available by backend presence; ordinary desktop startup prepares the instrumentation. Remove the Developer Mode toggle, marker-based activation and activation restart requirement.config.getroots, including WSL and reconnects. Share installation/lease mechanics with pruning, preserve user-authored entries and older backends' leases, and unwind stale or interrupted preparation.Gatekeeper review — September 18, 2026
Merged current
devat96ef38c6and repeated independent UI, native and server reviews after each finding until all three scopes reported zero remaining actionable findings.Corrections include ordinary-project discovery without Developer Mode; connected-daemon namespace selection and stale lease cleanup; Tauri child-webview capability isolation, native-window counting, primary-renderer reload cleanup, fragment navigation, snapshot lock ordering and deadline-fenced protocol dispatch; and Electron guest readiness, redirects and explicit navigation intent. Shared-frame regressions also cover insecure HTTP and late/rejected native IPC.
Architecture decision
Current OpenCode V2 documents MCP
_meta.sessionID; the historical missing-session-metadata argument no longer applies. A single native V2 plugin remains appropriate for the requested unified automation surface: it reuses the existing authenticated transport and native controllers, supports backend presence, and needs no second protocol server or mode-based permission framework. Plugin loading, tool availability and execution targeting are separate concerns.Validation
--no-bundle); focused Rust formatting andgit diff --check.Validation boundaries
The fixtures never use or stop the shared daemon or a user's database. Actual WSL runtime execution and a live packaged WebView2 interaction smoke were not performed in this pass; WSL resolution and Tauri controller/capability behavior have focused regressions. The currently running desktop artifact was not replaced. Agent interruption remains deadline-based rather than end-to-end native cancellation. Cross-platform CI must validate the pushed head.