fix(cli): restore V3 network capture through a CDP sidecar

[shrey150]

Summary

Restore Browse V3 network-capture behavior on Stagehand V4 through a CLI-private CDP sidecar, without committing core Stagehand, its protocol, or generated SDKs to a public network-event schema.

Stack (#2872)

1. chore(cli): import Browse V3 baseline #2833 — exact Browse V3 baseline import
2. refactor(cli): migrate Browse runtime and commands to Stagehand V4 #2834 — Stagehand V4 runtime and standard command parity
3. feat(cli): restore cursor overlay through page.evaluate #2869 — CLI-owned cursor overlay
4. fix(cli): restore V3 network capture through a CDP sidecar #2849 — CLI-private CDP sidecar; V3 network parity
5. refactor(cli): remove coordinate XPath return flag #2835 — remove --return-xpath; supported V3 parity/release checkpoint
6. test(evals): exercise the workspace V4 CLI #2838 — eval and packaging integration
7. fix(cli): persist context names in Browserbase #2839 — managed Context names (fast-follow)
8. refactor(cli): consume shared Functions core #2701 — shared Functions core consumer (fast-follow)

Architecture

• Lazily open one CLI-private browser-level CDP WebSocket using the endpoint already held by the initialized Stagehand client.
• Attach a flattened CDP session to the active V4 page target.
• Present the unchanged V3 network writer with the same on / off / send session shape.
• On network off, remove listeners, send Network.disable, and detach the page target.
• Keep the browser-level sidecar WebSocket alive across off/on cycles; closing an auxiliary Browserbase socket during the live session can disconnect the browser.
• Close the sidecar only during Browse session cleanup.
• Never log or return the signed browser WebSocket endpoint.

The request correlation and request/response JSON writer are inherited from the V3 CLI. No public protocol schema, extension, SDK, or generated-client changes are included. #2832 remains open outside the landing stack for that separate API-design discussion.

User-visible behavior

The V3 command and file surface is retained:

browse network on
browse network path
browse network clear
browse network off

Each request directory contains request.json and, when available, response.json, including the existing treatment of POST bodies, response bodies, failures, redirects, cache hits, and binary responses.

E2E Test Matrix

Full post-propagation implementation verification ran against exact clean #2849 head adbe80d8bbfc337fea9dc38e41a88da18c287046. Its seven-file network patch has the same stable patch ID (f062ac789108ddf640bc912776dafa210411f50c) as the previously stress-tested head, so the deterministic V3/V4 and MSN/CNN evidence below applies unchanged. Frozen install, fresh builds, focused/full tests, and the real Browserbase lifecycle were rerun on adbe80d8b. Current review-fix head 9887732b6f0c32cdef0966ebfd2ac2c454321d4b changes only the test helper timeout diagnostic; on that exact head, the targeted network-capture tests passed 2/2 and Browse formatting, ESLint, and TypeScript checks passed. The comparison CLI was the exact built V3 implementation at 7365a20d52955c10d72606f2e6ddd74791609d13. Every CLI flow used a unique daemon directory; no pre-existing daemon was reused or stopped.

Command / flow	Observed output	Confidence / sufficiency
pnpm install --frozen-lockfile; build extension, local Stagehand SDK, then browse	Frozen install and all three builds passed; the CLI manifest was generated from the tested workspace.	Proves the clean stacked head installs from its lockfile and the subprocess tests exercised fresh V4 SDK/extension/CLI artifacts, not stale dist output.
Built V3 vs built V4 deterministic fixture: open → network on → eval(GET, cached GET twice, POST, redirect, binary, abort) → path → off → path → clear → stop	V3: 8 requests / 7 responses. V4: 8 requests / 7 responses. Evaluation results matched; normalized request/response diff count was 0; command-shape diff was []; clear left 0 entries for both.	Exact parity for the stable V3 command and on-disk contract. Normalization was limited to request ID, timestamp/duration, HTTP Date, and fixture origin/port.
Concurrent real CLI enables: two browse network on subprocesses launched together, followed by one marked navigation	One subprocess returned { enabled: true }; the other returned { alreadyEnabled: true, enabled: true }. The marked navigation produced exactly 1 request record with 1 unique request ID.	Proves overlapping network on calls serialize and attach listeners once rather than duplicating capture.
Real CLI on → navigate → off → on → navigate → off without clearing	The two records were numbered 000 and 001; both URLs remained present, and the SHA-256 of the first request.json was unchanged after the second cycle.	Proves off/on resumes at the next on-disk counter and does not overwrite retained captures.
Real CLI permissions and lifecycle	Capture directories were 0700; request.json and response.json were 0600; network clear left 0 entries; browse stop reported stopped.	Proves the V3 private-file contract and cleanup behavior through the built CLI.
Real Browserbase, two network on → open Example Domain → off cycles in one session, then navigation with capture off	Both cycles captured GET 200 and the complete Example Domain body; the Browserbase session stayed stable; post-off open --wait networkidle worked; final status was connected, initialized, and remote; stop completed.	Proves the auxiliary sidecar survives real remote off/on while the owning Stagehand browser remains usable. No session ID or signed endpoint is included here.
Real Browserbase stress, V3 and V4, public MSN then CNN; each cycle ran on → path → open → scroll → collect 12s → off → path → navigate while off → status → clear	All 21 commands completed for each CLI. V3 captured MSN 415/401 and CNN 192/187 request/response records. V4 captured MSN 498/481 and CNN 194/187. Across all four cycles: 0 malformed request JSON, 0 malformed response JSON, 0 response-ID mismatches, 0 new records after off, and 0 entries after clear. Both sessions stayed remote/connected, stopped cleanly, and emitted no WebSocket endpoint.	Proves bounded high-concurrency, cross-origin Document/Script/Fetch/XHR/Image/Font/Media traffic. Live request counts are intentionally not expected to match because sites and ad auctions are nondeterministic; contract invariants and command shapes matched.
pnpm --filter browse lint	Passed formatting, ESLint, and TypeScript checks at full-verification head adbe80d8b; passed again at current review-fix head 9887732b6.	Static support for both the fully exercised implementation head and the exact current head.
pnpm --filter browse test with isolated daemon directory	At full-verification head adbe80d8b: 27 files / 390 tests passed; focused network subset: 3 files / 30 tests passed. At current review-fix head 9887732b6: targeted network-capture tests passed 2/2.	Covers sidecar routing, response bodies, listener teardown, attach/detach, request-write races, counter reuse, and overlapping enable serialization in addition to the real flows above; the current-head rerun is scoped to the test-only diagnostic change.

The deterministic artifact comparison includes method, URL/path, request/response headers and bodies, status/status text, MIME type, resource type, error shape, counter naming, and file modes. The live stress test also observed GET/POST/OPTIONS, failures, base64 bodies, cross-origin documents, and request-only records for traffic still in flight at the bounded off point.

This matrix does not claim WebSocket-frame, SSE-message, service-worker, or every out-of-process-iframe edge-case coverage; those are outside the V3 JSON request/response file contract proven here.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 99fe4ca

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 7 files

Architecture diagram

sequenceDiagram
    participant User as Browse CLI User
    participant CLI as Browse Command Handler
    participant Manager as Driver Session Manager
    participant Stagehand as Stagehand V4 Client
    participant Browser as Remote Browser
    participant Sidecar as CLI CDP Sidecar
    participant Capture as Network Capture Writer
    participant Files as Private Network Files

    Note over User,Files: Browse V3-compatible network capture on Stagehand V4

    User->>CLI: browse network on
    CLI->>Manager: Get active page
    Manager-->>CLI: V4 page with pageId
    CLI->>Manager: Get browser CDP debugger URL
    Manager->>Stagehand: Read browserWebSocketDebuggerUrl
    Stagehand-->>Manager: Signed browser WebSocket endpoint
    Manager-->>CLI: Endpoint kept internal
    CLI->>Capture: enable(pageId, endpoint)
    Capture->>Sidecar: attach(endpoint, pageId)
    Sidecar->>Browser: Open one browser-level CDP WebSocket
    Browser-->>Sidecar: WebSocket connected
    Sidecar->>Browser: Target.attachToTarget(flatten, pageId)
    Browser-->>Sidecar: CDP sessionId
    Sidecar-->>Capture: Network CDP session
    Capture->>Sidecar: Network.enable
    Capture->>Sidecar: Register request and response listeners
    CLI-->>User: enabled=true, network path

    Note over Browser,Files: CDP events are routed through the flattened page session

    Browser-->>Sidecar: Network.requestWillBeSent
    Sidecar-->>Capture: Request event
    Capture->>Files: Write request.json and allocate counter directory
    Browser-->>Sidecar: Network.responseReceived
    Sidecar-->>Capture: Response metadata
    Capture->>Sidecar: Network.getResponseBody
    Sidecar->>Browser: Fetch response body
    Browser-->>Sidecar: Body or CDP error
    Sidecar-->>Capture: Response body result
    Capture->>Files: Write response.json when available

    alt Concurrent network on requests
        User->>CLI: browse network on
        CLI->>Capture: enable(...)
        Capture-->>CLI: alreadyEnabled=true
    else CDP sidecar connection failure
        Sidecar-->>Capture: Connection or command error
        Capture-->>CLI: network sidecar error
        CLI-->>User: Capture enable failed
    end

    User->>CLI: browse network off
    CLI->>Capture: disable()
    Capture->>Sidecar: Remove registered listeners
    Capture->>Sidecar: Network.disable
    Sidecar->>Browser: Disable Network domain
    Capture->>Sidecar: Target.detachFromTarget
    Sidecar->>Browser: Detach page session
    Sidecar-->>Capture: Page session detached
    Note over Sidecar,Browser: Browser-level WebSocket remains open across off/on cycles
    Capture-->>CLI: enabled=false, retained network path
    CLI-->>User: Capture stopped

    opt Later network on in the same Browse session
        User->>CLI: browse network on
        CLI->>Capture: enable(next pageId, same endpoint)
        Capture->>Sidecar: Attach a new flattened page session
        Sidecar->>Browser: Target.attachToTarget(flatten, pageId)
        Capture->>Files: Continue from highest existing request counter
        CLI-->>User: Capture resumed without overwriting files
    end

    User->>CLI: browse stop
    CLI->>Manager: Close Browse session
    Manager->>Capture: close()
    Capture->>Sidecar: Disable and detach active page session
    Capture->>Sidecar: Close browser-level WebSocket
    Sidecar->>Browser: Close auxiliary CDP connection
    Manager->>Stagehand: Close Stagehand client
    CLI-->>User: Session stopped

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>