fix(evals): use workspace Browse and repair CLI adapter - #2963
Conversation
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 8 files
Architecture diagram
sequenceDiagram
participant Eval as Eval Tool Adapter
participant Run as BrowseCliRuntime
participant CLI as Browse CLI (workspace)
participant Pkg as Package Manager
participant Repo as Repo Root
Note over Eval,Repo: Workspace Dependency Resolution
Eval->>Eval: Read workspace dependency manifest
Eval->>Pkg: Resolve browse dependency
alt Resolved as workspace version
Pkg-->>Eval: Local package (workspace:*)
Eval->>Repo: Locate CLI entrypoint
Repo-->>Eval: packages/cli/bin/run.js
else Resolved as published version
Pkg-->>Eval: Published package
Eval->>CLI: Use installed binary
end
Note over Eval,CLI: Native Adapter Session Startup
Eval->>Run: Create session with generated name
Run->>Run: Generate session identifier
Note over Run: Include PID, timestamp, random suffix
Run->>CLI: open about:blank --session <id>
alt Local environment
Run->>CLI: Add --local flag
else Remote environment
Run->>CLI: Add --remote flag
end
CLI-->>Run: Connection established
Run-->>Eval: Session ready
Note over Eval,CLI: Tab Management Flow
Eval->>CLI: tab list --session <id>
CLI-->>Eval: Tabs with targetId mappings
Eval->>Eval: Map tabs to page handles
Eval->>CLI: tab new <url> --session <id>
CLI-->>Eval: New tab targetId
Eval->>CLI: tab switch <targetId> --session <id>
Eval->>CLI: tab close <targetId> --session <id>
Note over Eval,CLI: Page Operations
Eval->>CLI: navigate --wait <state> --timeout <ms> --session <id>
Eval->>CLI: screenshot --base64 --full-page --type <fmt> --session <id>
CLI-->>Eval: Base64-encoded image
Eval->>CLI: wait selector <selector> --timeout <ms> --state <state> --session <id>
Eval->>CLI: mouse click <x> <y> --session <id>
Eval->>CLI: mouse hover <x> <y> --session <id>
Eval->>CLI: mouse scroll <x> <y> <dx> <dy> --session <id>
Eval->>CLI: type <text> --session <id>
Eval->>CLI: fill <selector> <text> --session <id>
Eval->>CLI: press <key> --session <id>
Note over Eval: Command output parsed as JSON
opt Startup failure
Run->>CLI: open about:blank --session <id>
CLI-->>Run: Error
Run->>Run: Cleanup session resources
Run-->>Eval: Propagate error
end
alt Tab response missing targetId
CLI-->>Eval: Tab entry without targetId
Eval->>Eval: Reject as ambiguous
else Valid tab response
Eval->>Eval: Address by targetId, preserving identity
end
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Follow the CLI-selected tab after closing, use cryptographic session names, and throw a typed error for missing target IDs. Add tab-selection regressions and format the inherited README to unblock CI.
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
shrey150
changed the title
shrey150
changed the title
|
@cubic-dev-ai review this PR. Please review the full combined diff against |
@shrey150 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 9 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Architecture diagram
sequenceDiagram
participant Eval as Stagehand Evals
participant Workspace as pnpm Workspace
participant CLI as Browse CLI (packages/cli)
participant Adapter as BrowseCliTool Adapter
participant Session as BrowseCliSession
participant CLIProc as CLI Subprocess
participant Tabs as Browser Tabs
Note over Eval,CLI: Dependency Resolution
Eval->>Workspace: CHANGED: Declare "browse": "workspace:*"
Workspace-->>Eval: Resolve to workspace CLI package
Eval->>CLI: CHANGED: Use workspace CLI (was external 0.9.5)
CLI-->>Eval: Expose package.json version/entrypoint
Note over Adapter,CLIProc: Session Startup
Adapter->>Session: Create session with session name
Session->>Session: generate session name (pid + random hex)
Session->>CLIProc: Launch subprocess
CLIProc->>CLIProc: Start with local/remote browser
Note over Session,Tabs: Command Flow (Stable Tab IDs)
Session->>CLIProc: tab list
CLIProc-->>Session: Return tabs with targetIds
Session->>Session: Store handles by targetId
Session->>CLIProc: tab new [url] / tab switch targetId
CLIProc->>Tabs: Create/switch browser tab
Tabs-->>CLIProc: Tab ready
CLIProc-->>Session: Confirm operation
Note over Session,CLIProc: Page Operations (Updated Grammar)
Session->>CLIProc: wait selector --timeout --state
Session->>CLIProc: mouse click/hover/scroll [coords]
Session->>CLIProc: fill selector text / type text
Session->>CLIProc: screenshot --base64 --full-page --type --quality
CLIProc->>Tabs: Execute browser action
Tabs-->>CLIProc: Action result
CLIProc-->>Session: Return result
Note over Session,CLIProc: Tab Close & Selection
Session->>CLIProc: tab close targetId
CLIProc->>Tabs: Close tab
CLIProc-->>Session: Return selectedTargetId (CLI active tab)
alt CLI returns selected tab
Session->>Session: Set activePageId = selectedTargetId
Session->>CLIProc: Refresh tab list
else No selected tab returned
Session->>Session: Clear activePageId
Session->>CLIProc: Refresh tab list (optional)
end
Note over Session,CLIProc: Error Handling
Session->>CLIProc: Send malformed command
CLIProc-->>Session: Return error/empty targetId
Session->>Session: Throw EvalsError (missing targetId)
Note over Adapter,CLIProc: Session Cleanup
Adapter->>Session: close()
Session->>CLIProc: Terminate subprocess
CLIProc-->>Session: Confirm shutdown
Session-->>Adapter: Session closed
Note over Eval,CLI: Metadata Reporting
Eval->>CLI: Read workspace CLI package.json
CLI-->>Eval: version + entrypoint
Eval->>Eval: Include in Braintrust rows / TUI metadata
2 participants
Release versioning rewrites evals' pinned Browse dependency without updating the lockfile, causing the next frozen install to fail (failed release). Use
browse: "workspace:*", update its lockfile importer, and make metadata tests read the workspace CLI version and entrypoint.The native Browse eval adapter also sends obsolete commands and fails at
--jsonwith both published Browse 0.9.5 and the workspace CLI. Update startup, tab/mouse commands, timeout and screenshot flags to the supported grammar. Track tabs by stable target IDs, follow the CLI-selected active tab after closing a page, and share short cryptographically random session names with the external wrapper. Regression tests cover tab identity, active-tab selection, malformed responses, and session-name safety/uniqueness.This PR targets
mainand includes the dependency/build changes from #2965 together with the adapter fixes. #2965 is superseded. Changesets policy and public-package versions are unchanged; broader wrapper restrictions and abort/cleanup-timeout work from #2838 remain outside this PR.E2E Test Matrix
Current head
2ea510e1bhas the same code, dependencies, lockfile, and tests as verified commit09f3feb7c; only the upstream README was updated afterward. Retargeting this PR tomaindid not change its head commit. Both the dependency-only subset and the combined implementation were tested independently with pnpm 11.10.0.CI=true pnpm install --frozen-lockfileCI=true pnpm exec changeset version,CI=true pnpm exec tsx scripts/release/consolidate-changelogs.ts, thenCI=true pnpm install --frozen-lockfilepnpm exec turbo run build --filter=@browserbasehq/stagehand-evalspnpm exec vitest run --config packages/evals/vitest.config.ts --maxWorkers=4The native adapter's obsolete-command failure predates the dependency switch; the build fix itself does not require the adapter changes. Browser smokes used local Chromium with an isolated daemon directory and a test-only
--no-sandboxlauncher for the container. Remote adapter sessions, paid model evals, and package publishing were not performed locally.