Skip to content

fix(evals): use workspace Browse to unblock release builds - #2965

Closed
shrey150 wants to merge 2 commits into
mainfrom
fix/evals-workspace-build
Closed

shrey150 wants to merge 2 commits into
mainfrom
fix/evals-workspace-build

Conversation

@shrey150

@shrey150 shrey150 commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Superseded by #2963, which now targets main and includes every change from this PR together with the adapter fixes. No separate merge of this PR is needed.

Release versioning rewrites evals' pinned Browse dependency without updating the lockfile, causing the next frozen install to fail (failed release). Use browse: "workspace:*" so evals consume the current checkout and their dependency remains stable across release bumps.

This build-fix PR changes only the eval dependency, its lockfile importer, and workspace path/version assertions (four files). Native adapter runtime fixes are in the stacked child #2963 and are not required to merge this release fix. This implements the workspace approach as an alternative to #2961's Changesets flag.

Stack

  1. This PR → main: workspace dependency and build fix.
  2. fix(evals): use workspace Browse and repair CLI adapter #2963 → this branch: repair the pre-existing native Browse adapter and add runtime regression coverage.

Merge this PR first. The child preserves the already-reviewed adapter fixes separately.

E2E Test Matrix

Verified on a7711cac6 with pnpm 11.10.0. The parent was checked out and tested independently of the adapter fixes. Current head 468ae0a70 merges main's README cleanup; all files except README are identical to the verified head, and README exactly matches main and passes formatting.

Command / flow Observed output Confidence / sufficiency
GitHub checks on 468ae0a70 46 passed, 9 skipped; none failed or pending Updated build-only parent is conflict-free and passes independently; no outstanding Cubic findings
CI=true pnpm install --frozen-lockfile Passed; evals link to workspace Browse Confirms the dependency and lockfile agree
CI=true pnpm exec changeset version, CI=true pnpm exec tsx scripts/release/consolidate-changelogs.ts, then CI=true pnpm install --frozen-lockfile Passed; Browse became 0.9.7; eval manifest and lockfile were unchanged Reproduces the release sequence that previously failed; generated release files were restored afterward

| pnpm exec turbo run build --filter=@browserbasehq/stagehand-evals | 14/14 tasks passed | Builds the workspace CLI and eval bundles |
| pnpm exec vitest run --config packages/evals/vitest.config.ts --maxWorkers=4 | 90 files / 761 tests passed | Full eval unit suite, with the inherited devbox OpenAI key unset |
| Actual generated Claude/Codex Browse wrapper against local Chromium | Opened a synthetic form, filled/clicked it, and read the expected title; daemon files cleaned up | Proves the existing external wrapper works with the workspace dependency without child adapter fixes |
| Evals typecheck and repository formatting check | Passed | No type or formatting errors |

The existing native adapter's obsolete-command failure also occurs with published Browse 0.9.5. It is intentionally repaired in #2963. This PR does not claim native runtime compatibility or perform a live package publish.

@changeset-bot

changeset-bot Bot commented Sep 17, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 468ae0a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 5 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant EvalPkg as @browserbasehq/stagehand-evals
    participant Pnpm as pnpm (workspace resolver)
    participant BrowseCli as Browse CLI package (workspace)
    participant Lockfile as pnpm-lock.yaml
    participant Release as Release tooling (changeset version)
    participant Test as Eval test suite

    Note over EvalPkg,BrowseCli: Workspace Dependency Resolution Flow

    EvalPkg->>EvalPkg: package.json declares browse: "workspace:*"
    Pnpm->>Pnpm: Resolve workspace dependency
    Pnpm->>BrowseCli: Link local package (packages/cli)
    BrowseCli-->>Pnpm: Confirmed available
    Pnpm->>Lockfile: Record workspace importer
    Lockfile-->>Pnpm: Lockfile entry synced

    Note over Release,Lockfile: Release Versioning Sequence
    
    Release->>Pnpm: Run changeset version
    Pnpm->>BrowseCli: Bump version (e.g., 0.9.7)
    Pnpm->>Lockfile: Update CLI package version
    alt PR pre-fix (published version pinned)
        Pnpm->>Pnpm: Rewrite eval dependency to new version
        Pnpm->>Lockfile: Mismatch - no lockfile update
        Note over Pnpm,Lockfile: Frozen install fails on next run
    else CHANGED: workspace protocol
        Pnpm->>EvalPkg: Keep workspace:* reference
        Pnpm->>Lockfile: Lockfile remains consistent
    end
    Pnpm-->>Release: Versioning succeeds

    Note over Test,BrowseCli: Runtime Eval Flow (unchanged external wrapper)
    
    Test->>Test: Import workspace Browse CLI
    Test->>BrowseCli: Resolve tool metadata
    BrowseCli-->>Test: Version + entrypoint (packages/cli/bin/run.js)
    Test->>BrowseCli: Execute Browse wrapper (external)
    BrowseCli->>Browser: Drive local Chromium
    Browser-->>BrowseCli: Page interactions complete
    BrowseCli-->>Test: Test results pass

    alt Frozen install verification
        Pnpm->>Lockfile: pnpm install --frozen-lockfile
        alt Lockfile consistent
            Lockfile-->>Pnpm: Success
            Pnpm-->>EvalPkg: Linked workspace dependency
        else Lockfile mismatch
            Lockfile-->>Pnpm: Failure (blocked release)
        end
    end
Loading

Re-trigger cubic

@shrey150
shrey150 added this pull request to stack #2968 September 17, 2026 21:53
@shrey150
shrey150 removed this pull request from stack #2968 September 17, 2026 22:07
@shrey150 shrey150 closed this Sep 17, 2026
shrey150 added a commit that referenced this pull request Sep 17, 2026
Release versioning rewrites evals' pinned Browse dependency without
updating the lockfile, causing the next frozen install to fail ([failed
release](https://github.com/browserbase/stagehand/actions/runs/35200720996/job/105134544106)).
Use `browse: "workspace:*"`, update its lockfile importer, and make
metadata tests read the workspace CLI version and entrypoint.

The native Browse eval adapter also sends obsolete commands and fails at
`--json` with both published Browse 0.9.5 and the workspace CLI. Update
startup, tab/mouse commands, timeout and screenshot flags to the
supported grammar. Track tabs by stable target IDs, follow the
CLI-selected active tab after closing a page, and share short
cryptographically random session names with the external wrapper.
Regression tests cover tab identity, active-tab selection, malformed
responses, and session-name safety/uniqueness.

This PR targets `main` and includes the dependency/build changes from
#2965 together with the adapter fixes. #2965 is superseded. Changesets
policy and public-package versions are unchanged; broader wrapper
restrictions and abort/cleanup-timeout work from #2838 remain outside
this PR.

## E2E Test Matrix

Current head `2ea510e1b` has the same code, dependencies, lockfile, and
tests as verified commit `09f3feb7c`; only the upstream README was
updated afterward. Retargeting this PR to `main` did not change its head
commit. Both the dependency-only subset and the combined implementation
were tested independently with pnpm 11.10.0.

| Command / flow | Observed output | Confidence / sufficiency |
| --- | --- | --- |
| `CI=true pnpm install --frozen-lockfile` | Passed for dependency-only
and combined changes | Confirms workspace dependency resolution and
lockfile consistency |
| `CI=true pnpm exec changeset version`, `CI=true pnpm exec tsx
scripts/release/consolidate-changelogs.ts`, then `CI=true pnpm install
--frozen-lockfile` | Passed for both; Browse became 0.9.7; eval manifest
and lockfile stayed unchanged | Exercises the failing release sequence;
generated release files were restored afterward |
| `pnpm exec turbo run build --filter=@browserbasehq/stagehand-evals` |
14/14 build tasks passed for both | Builds Browse and eval bundles |
| `pnpm exec vitest run --config packages/evals/vitest.config.ts
--maxWorkers=4` | Combined: 92 files / 768 tests passed. Dependency-only
subset: 90 files / 761 tests passed | Full eval unit suites, with the
inherited devbox OpenAI key unset |
| Actual native adapter against local Chromium | Startup, navigation,
waits, fill/hover/coordinate click, press/typing, PNG/JPEG screenshots,
snapshot and scrolling passed | Real CLI subprocesses against a
synthetic local page |
| Create/switch/close real browser tabs | Closing active middle,
inactive and active last tabs preserved the CLI-selected page and
subsequent titles | Verifies tab identity and active-page
synchronization |
| Actual generated Claude/Codex Browse wrapper | Opened and submitted a
synthetic form and read the expected title; passed for both subsets |
Shared external wrapper works with the workspace CLI |
| Native/wrapper cleanup | No remaining PID/socket files after shutdown
settled | Smoke resources cleaned up |
| GitHub CI after the README reconciliation | Dependency-only parent: 46
passed, 9 skipped. Combined head while stacked: 9 passed, 15 skipped;
none failed or pending | The earlier combined source tree also passed
the full 47-check CI run; check selection depends on the PR diff |

The native adapter's obsolete-command failure predates the dependency
switch; the build fix itself does not require the adapter changes.
Browser smokes used local Chromium with an isolated daemon directory and
a test-only `--no-sandbox` launcher for the container. Remote adapter
sessions, paid model evals, and package publishing were not performed
locally.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant