Skip to content

feat(cli): restore cursor overlay through page.evaluate - #2869

Merged
shrey150 merged 3 commits into
mainfrom
agent/browse-v4-cli-cursor-overlay
Sep 17, 2026
Merged

shrey150 merged 3 commits into
mainfrom
agent/browse-v4-cli-cursor-overlay

Conversation

@shrey150

@shrey150 shrey150 commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

Restore Browse's visible cursor as a CLI-owned DOM overlay, without adding a cursor API to core Stagehand V4.

  • Keep the overlay implementation in one dedicated cursor-overlay.ts file.
  • Install it idempotently for the current document through page.evaluate(CURSOR_OVERLAY_SCRIPT) and for future navigations through page.addInitScript(...).
  • Retry installation on DOMContentLoaded when the init script runs before the document root exists.
  • Keep injection in the top frame and update the marker from coordinate input, including when input lands inside a child frame.
  • Treat visual position updates as best-effort so they cannot block or invalidate real mouse input.
  • Preserve the V3 browse cursor JSON response: { "cursor": "enabled" }.

Stack (#2872)

  1. chore(cli): import Browse V3 baseline #2833 — exact Browse V3 baseline import
  2. refactor(cli): migrate Browse runtime and commands to Stagehand V4 #2834 — Stagehand V4 runtime and standard command parity
  3. feat(cli): restore cursor overlay through page.evaluate #2869 — CLI-owned cursor overlay
  4. fix(cli): restore V3 network capture through a CDP sidecar #2849 — CLI-private CDP sidecar; V3 network parity
  5. refactor(cli): remove coordinate XPath return flag #2835 — remove --return-xpath; supported V3 parity/release checkpoint
  6. test(evals): exercise the workspace V4 CLI #2838 — eval and packaging integration
  7. fix(cli): persist context names in Browserbase #2839 — managed Context names (fast-follow)
  8. refactor(cli): consume shared Functions core #2701 — shared Functions core consumer (fast-follow)

Why this is separate

The cursor is a self-contained optional visual feature with different review concerns from the combined V4 runtime/command migration: injected DOM/CSS, idempotency, event handling, and screenshot behavior. Keeping it additive on #2834 lets this feature be reviewed or reverted without disturbing browser lifecycle or commands.

E2E Test Matrix

Review-feedback verification compared the exact prior head 6a9d6aa09 with fixed implementation head 1035fbbf5 through the built CLI and real Browserbase browsers. Final head 68f6fcb2c only expands automated coverage and does not change runtime code. Targets were the public example.com and example.org pages.

Command / flow Observed output Confidence / sufficiency
Prior head: enable cursor, alternate 20 cross-origin navigations, inspect #__browse_cursor_overlay__ before any mouse input Overlay count was 0 after 20/20 navigations Reproduces the DOM-readiness bug raised in review
Fixed head: repeat the same 20-navigation flow Overlay count was 1 after 20/20 navigations (0 misses) Proves the DOMContentLoaded retry restores the overlay after navigation in the real browser path
Prior head: replace the page's cursor-position callback with a throwing function, then run browse mouse click 200 200 against an oversized synthetic button Command exited 1; the page's click state remained null Reproduces the visual-update failure blocking real mouse input
Fixed head: repeat the same forced overlay failure and click CLI returned { "clicked": true }; page click state became "yes" Proves overlay rendering is best-effort while real input still executes
Built CLI: browse cursor { "cursor": "enabled" } (prior head returned { "enabled": true }) Confirms V3-compatible output for existing scripts
pnpm --filter browse lint Passed formatting, ESLint, and TypeScript checks Static validation on the final head
pnpm --filter browse test:cli 26 files / 393 tests passed Full Browse suite, including DOM readiness, safe styling, idempotency, top-frame isolation, cursor positioning/clamping, and all four coordinate input commands
browse stop after each live run Completed successfully Covers Browserbase session and daemon cleanup

The already-uploaded screenshot below remains representative visual proof of the same overlay behavior.

Browse V4 CLI cursor overlay surviving navigation and pointing inside an iframe

No LLM path or customer data was involved.

@changeset-bot

changeset-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 20f96c6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Architecture diagram
sequenceDiagram
    participant User as Browse CLI User
    participant CLI as Browse CLI
    participant Runtime as Runtime Command Handlers
    participant Manager as Page Manager
    participant Page as Active Browser Page
    participant DOM as Page DOM Overlay
    participant Browser as Browser Input and Renderer

    Note over CLI,DOM: CLI-owned cursor visualization (no Stagehand core cursor API)

    User->>CLI: browse cursor
    CLI->>Runtime: Dispatch cursor command
    Runtime->>Manager: Resolve activePage()
    Manager-->>Runtime: Active page
    Runtime->>Page: evaluate(CURSOR_OVERLAY_SCRIPT)
    Page->>DOM: Ensure fixed overlay exists

    alt Overlay already installed
        DOM-->>Page: Reuse __browse_cursor_overlay__
    else Overlay not installed
        DOM->>DOM: Create fixed SVG cursor element
        DOM->>DOM: Set aria-hidden, z-index, pointer-events none
        DOM->>DOM: Append overlay to document root
        DOM->>DOM: Install capture-phase mousemove listener
    end

    Page-->>Runtime: Evaluation complete
    Runtime-->>CLI: Return enabled=true
    CLI-->>User: Cursor overlay enabled

    User->>CLI: browse mouse hover x y
    CLI->>Browser: Move mouse to viewport coordinates
    Browser->>DOM: Dispatch mousemove event
    DOM->>DOM: Clamp clientX and clientY to non-negative values
    DOM->>DOM: Update fixed overlay left and top

    User->>CLI: browse screenshot --path cursor.png
    CLI->>Page: Capture current page
    Page->>Browser: Render DOM and overlay
    Browser-->>CLI: Screenshot including visible cursor
    CLI-->>User: Save cursor.png
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/cli/src/lib/driver/cursor-overlay.ts
Comment thread packages/cli/src/lib/driver/commands/runtime.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/cli/src/lib/driver/commands/runtime.ts
Comment thread packages/cli/src/lib/driver/commands/mouse.ts
@shrey150
shrey150 force-pushed the agent/browse-v4-cli-cursor-overlay branch from dc60e5f to 8428e33 Compare September 11, 2026 18:23
@shrey150
shrey150 force-pushed the agent/browse-v4-cli-cursor-overlay branch from 8428e33 to 638c86f Compare September 15, 2026 23:10
shrey150 added a commit that referenced this pull request Sep 15, 2026
## Summary

Import `packages/cli/**` exactly from the published `browse@0.9.6` V3
release, without changing its source or runtime behavior.

This is intentionally a provenance checkpoint, not a line-by-line
feature review. The imported source is kept runnable by a root,
version-scoped pnpm override that resolves its unchanged Stagehand
dependency to `3.7.1`. #2834 removes that override and starts the V4
migration.

## Exact-source provenance

- Annotated tag: `browse@0.9.6`
(`548c56407431db27823a212f53475443c7e8358d`)
- Release commit: `1d49a95c0c230c346f8d50647e10303d6310fcd2`
- Authoritative CLI tree: `b4048badce921cf54f199f96033d9a014ef977ec`
- This PR's `HEAD:packages/cli` tree:
`b4048badce921cf54f199f96033d9a014ef977ec`

The tag's ignored README whitespace is retained too; formatting the
import would invalidate the tree proof.

## Verification

- Current remote head: `a77e1507b85e3c02553f36ead6ebd0237b0cccc6`, based
on current `main`.
- `HEAD:packages/cli` exactly equals the published V3 tree hash above.
- pnpm 11 frozen install and the repository supply-chain release-age
policy pass.
- Browse lint, typecheck, and build pass; the full baseline suite
passes: 25 files / 366 tests.
- A fresh extension build still exactly matches the Go-embedded archive:
SHA-256
`8efc7d171a625cca95c02d02d369b59435fae776cae6c7dd2f6fe72eb19785c0` on
both files. This specifically verifies that adding the V3 dependency
graph does not perturb the current V4 extension artifact.
- This layer intentionally exercises V3 through the scoped Stagehand
3.7.1 override. V4 behavior starts in #2834.

## Stack (#2872)

1. **#2833 — exact Browse V3 baseline import**
2. #2834 — Stagehand V4 runtime and standard command parity
3. #2869 — CLI-owned cursor overlay
4. #2849 — CLI-private CDP sidecar; V3 network parity
5. #2835 — remove `--return-xpath`; supported V3 parity/release
checkpoint
6. #2838 — eval and packaging integration
7. #2839 — managed Context names (fast-follow)
8. #2701 — shared Functions core consumer (fast-follow)

## Review and landing boundary

Review this PR by verifying the tree hashes, dependency pin, root
package wiring, and changeset—not by treating the imported V3 source as
newly authored code. This head deliberately imports V3 code into the V4
repository and is not independently publishable. It lands only as the
base of the complete stack.

The framework network-event schema proposal in #2832 is intentionally
outside this landing stack.
@shrey150
shrey150 force-pushed the agent/browse-v4-cli-cursor-overlay branch from 638c86f to 75d8e04 Compare September 15, 2026 23:31

@AzamAbdul AzamAbdul left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feedback from Astra labeled with my take:

seems like a good idea to change to DOM readiness

  • The overlay can disappear after navigation until the next mouse action.
    The initialization script runs at document creation, when document.documentElement and document.body can still be absent. ensureCursor() then returns without scheduling another attempt. There is no DOMContentLoaded handler or retry, so a subsequent screenshot can have no cursor. V3 explicitly waited for DOM readiness and retried installation.
    Fix: retry installation once the DOM is ready.
    Reference:
    const ensureCursor = () => {

not sure about this one

  • A cursor rendering failure can prevent the actual mouse action.
    Click, hover, scroll, and drag all await the overlay update before performing the action. If that extra page.evaluate() fails—for example, during navigation—the command aborts before sending its mouse input. The branch already treats the final drag overlay update as optional, but does not do so for these initial updates. V3 swallowed cursor-update errors throughout.
    Fix: make visual position updates best-effort while continuing to propagate errors from the actual mouse operation.
    Reference:
    await positionCursorOverlay(manager, page, x, y);

lets definitely do this to support backwards compatibility

  • browse cursor changes its JSON output unnecessarily.
    V3 returned { "cursor": "enabled" }; this branch returns { "enabled": true }. The command prints that result directly, so existing scripts checking .cursor break.
    Fix: preserve the old response unless this is an intentional API change.
    Reference:

Base automatically changed from agent/browse-v4-4-runtime to main September 16, 2026 00:23
shrey150 added a commit that referenced this pull request Sep 16, 2026
…2834)

## Summary

Migrate Browse's browser lifecycle and standard command surface together
from Stagehand V3 to V4.

- Replace the V3 constructor/init lifecycle with V4 browser factories
and `Stagehand.create()`.
- Support managed local, Browserbase remote, and attached CDP connection
targets.
- Preserve owned-versus-attached cleanup, daemon persistence,
Browserbase session identity, and timeout handling.
- Restore navigation, page information, deterministic locator actions,
keyboard/mouse input, viewport/screenshot, snapshot, eval, and tab
commands on V4 APIs.
- Keep click/fill/select deterministic; this does not add a model-free
structured `act()` path.
- Make the remaining cursor, network, and coordinate-XPath gaps fail
explicitly for the stack layers that restore or remove them.

## Stack (#2872)

1. #2833 — exact Browse V3 baseline import
2. **#2834 — Stagehand V4 runtime and standard command parity**
3. #2869 — CLI-owned cursor overlay
4. #2849 — CLI-private CDP sidecar; V3 network parity
5. #2835 — remove `--return-xpath`; supported V3 parity/release
checkpoint
6. #2838 — eval and packaging integration
7. #2839 — managed Context names (fast-follow)
8. #2701 — shared Functions core consumer (fast-follow)

## Review shape

The lifecycle and command migration remain two ordered implementation
commits:

1. `389e2dae6` — V4 browser/session foundation and lifecycle ownership.
2. `b45167462` — standard command translation on that foundation.

They are one PR because both commits rewrite the same nine command/test
files. Reviewing their combined final diff avoids temporary
deletion/stubbing followed by reimplementation, while the commits still
provide useful lifecycle-versus-command checkpoints. Review follow-up
`24178275f` adds narrowly scoped ownership, error-sanitization, and
timeout guards. The resulting PR diff is 23 files, +988/−436.

Cursor DOM injection and private CDP network transport remain separate
because they are independently reviewable mechanisms and cleanly
additive diffs. The legacy coordinate `returnXPath` request is still
accepted here but fails explicitly until #2835 removes the option. This
remains an intentionally non-publishable intermediate head.
## E2E Test Matrix

Fresh post-flatten verification used the actual built CLI at final head
`6f7e9c209`. Every daemon command used an isolated owner-only runtime
directory.

| Command / flow | Observed output | Confidence / sufficiency |
| --- | --- | --- |
| `pnpm install --frozen-lockfile` | Lockfile passed supply-chain
policy, was already up to date, and installation completed | Proves the
flattened stack resolves exactly from the committed lockfile |
| `pnpm exec turbo run build --filter=browse` | Protocol, extension,
Stagehand SDK, and Browse CLI built successfully (4/4 tasks) | Proves
the CLI was tested against this head's protocol/extension/SDK artifacts,
not stale workspace `dist` files |
| Compare the rebuilt extension with
`packages/sdk-go/internal/extensionassets/stagehand-extension.zip` |
Exact byte match; both SHA-256
`8efc7d171a625cca95c02d02d369b59435fae776cae6c7dd2f6fe72eb19785c0`;
archive manifest and package version both `1.0.2` | Confirms the
TypeScript/CLI build and Go-embedded extension are synchronized |
| Built CLI: `browse open <synthetic-data-url> --remote`; `browse
status` | Remote browser connected and initialized; deterministic
fixture loaded | Exercises production Browserbase provisioning plus the
V4 daemon/session lifecycle on the exact final head |
| `browse get text //h1`; `fill`; `select`; `click`; `is checked`; `wait
selector`; `highlight`; `viewport`; `screenshot`; `snapshot --full` |
XPath returned `Ready`; input became `Ada`; select became `b`; click
produced `Clicked`; checkbox was true; PNG was 17,761 bytes; snapshot
contained the fixture | Covers deterministic V4 reads, actions, waits,
state, and rendering without an LLM |
| Set a page marker; `tab new`; `tab list`; `tab close`; read the marker
from a new CLI process | Tab count changed to 2 and the original page
returned marker `yes` | Proves daemon persistence, active-tab handling,
and state reuse across invocations |
| Inspect the isolated runtime directory/PID; `browse stop`; poll the
Browserbase session | Modes were `0700` / `0600`; the owned remote
session reached `COMPLETED` | Proves owner-only daemon files and owned
Browserbase resource cleanup |
| `browse cursor`; `browse network on`; `browse mouse hover ...
--return-xpath` | Each exited 1 with the intended explicit
layer-boundary error | Confirms this intermediate layer fails honestly
until the cursor, network, and flag-removal layers land |
| `pnpm --filter browse test` | 25 files / 385 tests passed | Full
Browse unit/integration suite on the exact final head |

The runner has no Chrome/Chromium installation, so a fresh attached-CDP
ownership smoke was not possible. Attached-browser non-ownership remains
covered by the focused suite and is not claimed as a fresh live result
here.
@shrey150
shrey150 force-pushed the agent/browse-v4-cli-cursor-overlay branch from 75d8e04 to 6a9d6aa Compare September 16, 2026 00:23
@shrey150

Copy link
Copy Markdown
Collaborator Author

Addressed the three points from #2869 (review) in 1035fbbf5:

  • defer overlay installation to DOMContentLoaded when the document root is not available yet;
  • make cursor-position rendering best-effort while preserving failures from the real mouse operation;
  • restore the V3 response shape: { "cursor": "enabled" }.

I also reproduced the review findings on the prior head with the built CLI and a real Browserbase browser. The old code missed the overlay after 20/20 cross-origin navigations and aborted a coordinate click when the visual update was forced to fail. On this commit the overlay survived 20/20 navigations, the same forced visual failure still delivered the click, and the page recorded the click. Full Browse verification is green: lint passed and 26 test files / 390 tests passed. The PR description now contains the complete before/after matrix.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread packages/cli/tests/cursor-overlay.test.ts
Addresses review feedback from AzamAbdul.
Addresses feedback from cubic-dev-ai[bot].
@shrey150
shrey150 force-pushed the agent/browse-v4-cli-cursor-overlay branch from 68f6fcb to 20f96c6 Compare September 16, 2026 21:50
@shrey150
shrey150 merged commit 7664ad7 into main Sep 17, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants