Replay refused top-level navigations in the browser VM

[ulziibay-kernel]

Summary

A client asks for a page, the site answers 429 Too Many Requests, and the client is left holding a block page. A person in that position hits reload. An agent usually does not — it reads the document it was handed, concludes the site is unavailable, and reports back.

This adds server/lib/pagerecovery, which does that reload early enough that the caller never sees the refusal.

It intercepts main-frame document responses at the Fetch response stage and answers a refusal with a 307 back to the same URL instead of letting it through. Chromium treats that as one more hop in the navigation already in flight, so a page.goto() resolves once, on the page it asked for, having waited out the retries. It is not a second navigation, so nothing the client is waiting on is interrupted.

Off by default behind PAGE_RECOVERY_ENABLED.

What is replayed

Condition	Replayed
408, 429, 502, 503, 504, 507	yes
ConnectionReset, ConnectionClosed, ConnectionFailed, ConnectionAborted, TimedOut	yes
403	no — as often a settled answer about the session as a throttle
502 with X-Kernel-Proxy-Error	no — our own egress failing, already reported by cdpmonitor as a typed proxy_error
Subresources, iframe documents, non-GET navigations	no

Non-GET is excluded because the replay preserves method and body: a POST a gateway refused may still have been recorded upstream.

Retries are bounded by attempts (default 2) and wall clock (default 8s) per session and URL, sized to fit inside a caller's goto timeout with the real page load still to come. Retry-After is honoured when it fits what the budget has left; otherwise backoff is exponential with full jitter from 300 ms. Past the budget the refusal goes through, so a block never turns into a navigation that looks hung.

Notes from building it

Two things were verified against real Chromium rather than assumed:

• Cookies survive the replay. The network stack applies Set-Cookie before the request is paused, so a clearance cookie handed out by a block page is present on the retry — which is the mechanism that makes a manual reload work at all.
• A client's own Fetch interception still works. Playwright's page.route installs interception on its own session; both interceptors see every request. There is a test for it.

A fulfillment must carry a body, even an empty one: Chromium treats a fulfillment without one as no fulfillment at all and lets the original response through. That silently produced a no-op until the e2e test caught it.

Scope

This handles only the half of the problem that can be decided from the response itself. A block that answers 200 and puts an interstitial in the document needs a reading of the rendered page and a real reload after the document has run; that is deliberately not here.

There is no API-level or SDK-level surface yet — this is the in-VM mechanism plus an env flag.

Testing

go test -race ./lib/pagerecovery ./lib/metrics ./cmd/config
KERNEL_PAGERECOVERY_CHROME_E2E=1 go test -race ./lib/pagerecovery -count=1 -v

Both pass locally, including the real-Chromium suite: the transparent case, an exhausted budget landing the caller on the site's own answer, an unrefused navigation going untouched, a 403 left alone, a client interceptor still seeing every request, and six tabs refused at once each recovering independently.

lib/devtoolsproxy's TestUpstreamManagerDetectsChromiumAndRestart flakes on TempDir cleanup under -count=3 on this machine. It does so identically on a clean tree and is unrelated to this change.

🤖 Generated with Claude Code

[ulziibay-kernel]

Superseded by kernel/kernel-images-private#406 — working around a site's refusals belongs with the rest of the evasion surface rather than in the public image.

The private version also carries two fixes found after this PR was opened: a replay that cannot be delivered now passes the refusal through instead of leaving the navigation paused, and the concurrency test's tab-coverage gate was wrong (it counted the pre-existing about:blank tab, so the last tab could be navigated before interception was installed).