Conversation
|
Contributor
There was a problem hiding this comment.
No issues found across 5 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Architecture diagram
sequenceDiagram
participant EvalPkg as @browserbasehq/stagehand-evals
participant Pnpm as pnpm (workspace resolver)
participant BrowseCli as Browse CLI package (workspace)
participant Lockfile as pnpm-lock.yaml
participant Release as Release tooling (changeset version)
participant Test as Eval test suite
Note over EvalPkg,BrowseCli: Workspace Dependency Resolution Flow
EvalPkg->>EvalPkg: package.json declares browse: "workspace:*"
Pnpm->>Pnpm: Resolve workspace dependency
Pnpm->>BrowseCli: Link local package (packages/cli)
BrowseCli-->>Pnpm: Confirmed available
Pnpm->>Lockfile: Record workspace importer
Lockfile-->>Pnpm: Lockfile entry synced
Note over Release,Lockfile: Release Versioning Sequence
Release->>Pnpm: Run changeset version
Pnpm->>BrowseCli: Bump version (e.g., 0.9.7)
Pnpm->>Lockfile: Update CLI package version
alt PR pre-fix (published version pinned)
Pnpm->>Pnpm: Rewrite eval dependency to new version
Pnpm->>Lockfile: Mismatch - no lockfile update
Note over Pnpm,Lockfile: Frozen install fails on next run
else CHANGED: workspace protocol
Pnpm->>EvalPkg: Keep workspace:* reference
Pnpm->>Lockfile: Lockfile remains consistent
end
Pnpm-->>Release: Versioning succeeds
Note over Test,BrowseCli: Runtime Eval Flow (unchanged external wrapper)
Test->>Test: Import workspace Browse CLI
Test->>BrowseCli: Resolve tool metadata
BrowseCli-->>Test: Version + entrypoint (packages/cli/bin/run.js)
Test->>BrowseCli: Execute Browse wrapper (external)
BrowseCli->>Browser: Drive local Chromium
Browser-->>BrowseCli: Page interactions complete
BrowseCli-->>Test: Test results pass
alt Frozen install verification
Pnpm->>Lockfile: pnpm install --frozen-lockfile
alt Lockfile consistent
Lockfile-->>Pnpm: Success
Pnpm-->>EvalPkg: Linked workspace dependency
else Lockfile mismatch
Lockfile-->>Pnpm: Failure (blocked release)
end
end
shrey150
added this pull request to stack #2968
September 17, 2026 21:53
shrey150
removed this pull request from stack #2968
September 17, 2026 22:07
shrey150
added a commit
that referenced
this pull request
Sep 17, 2026
Release versioning rewrites evals' pinned Browse dependency without updating the lockfile, causing the next frozen install to fail ([failed release](https://github.com/browserbase/stagehand/actions/runs/35200720996/job/105134544106)). Use `browse: "workspace:*"`, update its lockfile importer, and make metadata tests read the workspace CLI version and entrypoint. The native Browse eval adapter also sends obsolete commands and fails at `--json` with both published Browse 0.9.5 and the workspace CLI. Update startup, tab/mouse commands, timeout and screenshot flags to the supported grammar. Track tabs by stable target IDs, follow the CLI-selected active tab after closing a page, and share short cryptographically random session names with the external wrapper. Regression tests cover tab identity, active-tab selection, malformed responses, and session-name safety/uniqueness. This PR targets `main` and includes the dependency/build changes from #2965 together with the adapter fixes. #2965 is superseded. Changesets policy and public-package versions are unchanged; broader wrapper restrictions and abort/cleanup-timeout work from #2838 remain outside this PR. ## E2E Test Matrix Current head `2ea510e1b` has the same code, dependencies, lockfile, and tests as verified commit `09f3feb7c`; only the upstream README was updated afterward. Retargeting this PR to `main` did not change its head commit. Both the dependency-only subset and the combined implementation were tested independently with pnpm 11.10.0. | Command / flow | Observed output | Confidence / sufficiency | | --- | --- | --- | | `CI=true pnpm install --frozen-lockfile` | Passed for dependency-only and combined changes | Confirms workspace dependency resolution and lockfile consistency | | `CI=true pnpm exec changeset version`, `CI=true pnpm exec tsx scripts/release/consolidate-changelogs.ts`, then `CI=true pnpm install --frozen-lockfile` | Passed for both; Browse became 0.9.7; eval manifest and lockfile stayed unchanged | Exercises the failing release sequence; generated release files were restored afterward | | `pnpm exec turbo run build --filter=@browserbasehq/stagehand-evals` | 14/14 build tasks passed for both | Builds Browse and eval bundles | | `pnpm exec vitest run --config packages/evals/vitest.config.ts --maxWorkers=4` | Combined: 92 files / 768 tests passed. Dependency-only subset: 90 files / 761 tests passed | Full eval unit suites, with the inherited devbox OpenAI key unset | | Actual native adapter against local Chromium | Startup, navigation, waits, fill/hover/coordinate click, press/typing, PNG/JPEG screenshots, snapshot and scrolling passed | Real CLI subprocesses against a synthetic local page | | Create/switch/close real browser tabs | Closing active middle, inactive and active last tabs preserved the CLI-selected page and subsequent titles | Verifies tab identity and active-page synchronization | | Actual generated Claude/Codex Browse wrapper | Opened and submitted a synthetic form and read the expected title; passed for both subsets | Shared external wrapper works with the workspace CLI | | Native/wrapper cleanup | No remaining PID/socket files after shutdown settled | Smoke resources cleaned up | | GitHub CI after the README reconciliation | Dependency-only parent: 46 passed, 9 skipped. Combined head while stacked: 9 passed, 15 skipped; none failed or pending | The earlier combined source tree also passed the full 47-check CI run; check selection depends on the PR diff | The native adapter's obsolete-command failure predates the dependency switch; the build fix itself does not require the adapter changes. Browser smokes used local Chromium with an isolated daemon directory and a test-only `--no-sandbox` launcher for the container. Remote adapter sessions, paid model evals, and package publishing were not performed locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by #2963, which now targets
mainand includes every change from this PR together with the adapter fixes. No separate merge of this PR is needed.Release versioning rewrites evals' pinned Browse dependency without updating the lockfile, causing the next frozen install to fail (failed release). Use
browse: "workspace:*"so evals consume the current checkout and their dependency remains stable across release bumps.This build-fix PR changes only the eval dependency, its lockfile importer, and workspace path/version assertions (four files). Native adapter runtime fixes are in the stacked child #2963 and are not required to merge this release fix. This implements the workspace approach as an alternative to #2961's Changesets flag.
Stack
main: workspace dependency and build fix.Merge this PR first. The child preserves the already-reviewed adapter fixes separately.
E2E Test Matrix
Verified on
a7711cac6with pnpm 11.10.0. The parent was checked out and tested independently of the adapter fixes. Current head468ae0a70merges main's README cleanup; all files except README are identical to the verified head, and README exactly matches main and passes formatting.468ae0a70CI=true pnpm install --frozen-lockfileCI=true pnpm exec changeset version,CI=true pnpm exec tsx scripts/release/consolidate-changelogs.ts, thenCI=true pnpm install --frozen-lockfile|
pnpm exec turbo run build --filter=@browserbasehq/stagehand-evals| 14/14 tasks passed | Builds the workspace CLI and eval bundles ||
pnpm exec vitest run --config packages/evals/vitest.config.ts --maxWorkers=4| 90 files / 761 tests passed | Full eval unit suite, with the inherited devbox OpenAI key unset || Actual generated Claude/Codex Browse wrapper against local Chromium | Opened a synthetic form, filled/clicked it, and read the expected title; daemon files cleaned up | Proves the existing external wrapper works with the workspace dependency without child adapter fixes |
| Evals typecheck and repository formatting check | Passed | No type or formatting errors |
The existing native adapter's obsolete-command failure also occurs with published Browse 0.9.5. It is intentionally repaired in #2963. This PR does not claim native runtime compatibility or perform a live package publish.