Skip to content

Bound the live view connect stages separately - #415

Closed
robertjamesprior wants to merge 7 commits into
hypeship/live-view-connect-failure-reportingfrom
hypeship/staged-live-view-connect-bounds
Closed

robertjamesprior wants to merge 7 commits into
hypeship/live-view-connect-failure-reportingfrom
hypeship/staged-live-view-connect-bounds

Conversation

@robertjamesprior

@robertjamesprior robertjamesprior commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #403 — retarget to main once that lands.

Summary

A connect walks transport (socket open) → signaling (signal/provide + offer) → media
(ICE starts). One 15s watchdog covered all three, so a socket that never opened burned
the full budget and reported only "connection timeout", and a stuck media path was
indistinguishable from a stuck socket.

Each stage now arms its own bound and reports its own reason:

stage starts bound retry
transport in connect(), before the socket opens 15s yes
signaling on ws.onopen 3s yes
media after setRemoteOffer, once a peer exists 2s no

media is terminal because a peer that exists but never reaches ICE checking is the
deterministic class — a retry repeats it. That also caps the wait on that path, and
transport/signaling now fail on their own bounds instead of one shared clock.

KERNEL_CONNECTION_TIMEOUT's reason is now "<stage> timeout" rather than
"connection timeout", and KERNEL_CONNECTION_FAILED carries the same stage. Both
events are recent enough that this reads as filling them in rather than changing them,
but it is a payload change.

Calibration

The bounds come from 130 connects against real headful sessions — us-east, eu-west,
ap-southeast, plus relay-only (iceTransportPolicy: 'relay') in two of them. A local
Chromium loaded the live view with the WebSocket and RTCPeerConnection constructors
instrumented before any page script ran; the client itself is unchanged.

stage (ms) p50 p95 p99 max
transport, direct 238 700 718 722
transport, relay-only 236 241 241 241
signaling + offer, direct 22 79 138 195
signaling + offer, relay-only 24 117 129 145
media, direct 2 93 94 96
media, relay-only 89 94 96 96
  • media is local, not network-bound. ICE reaches checking as soon as the local
    description is set, because the remote candidates arrive in the offer. Direct media is
    1–16ms even in ap-southeast, where transport is 700ms. Relay-only is a flat ~90ms
    (the TURN allocation), independent of the last mile. 2s is ~20x the worst case.
  • transport stays at 15s, deliberately off the measurement. It is last-mile RTT
    (35ms intra-region → 238ms eu-west → 700ms ap-southeast), but it is also the stage
    the live-view proxy's own wake path lands in, and that path can spend ~12s waking a
    browser before the socket opens. A bound set from the measured p99 would fire on every
    slow wake and never let the wake finish. This is the one stage where a stale bound is
    cheaper than a tight one.
  • signaling is one round trip on an open socket. p99 under 200ms, including cold
    sessions (the first connect after session creation is the slow one — ~135ms offer vs
    ~20ms median).

All 130 connects succeeded; this measures the healthy path and does not reproduce the
field failure.

Consequence worth stating: only media gets faster. The transport stage keeps the
budget it had, so a transport-class failure still spends up to ~47s across three attempts
(3 × 15s + 2 × 1s) before the terminal event — the same total as before, now with a
reason attached. What changes is that the stuck-media class, which is the failure we can
actually see in the field, ends in ~2.5s instead of 15.4s.

Validation on a real image

Hotpatch A/B in one session on the deployed image
(the currently deployed headful image, serving app.416afd11.js) against a local
build of this branch (app.47e216f9.js). The live view was embedded in an iframe on a
local parent page, so the parent-frame messages are observable — the client only posts
them when embedded. Faults were injected before any page script ran:

  • throw — RTCPeerConnection construction throws. The failing boundary from the field.
  • media-stall — candidate lines stripped from the remote offer and addIceCandidate
    neutralised, so ICE never leaves new.
scenario baseline (fb437e9) patched
healthy connected 638ms, playing 1809ms, no failure events connected 526ms, playing 1591ms, no failure events
throw KERNEL_CONNECTION_TIMEOUT 15373ms, reason connection timeout KERNEL_CONNECTION_FAILED 432ms, reason injected: RTCPeerConnection blocked
media-stall KERNEL_CONNECTION_TIMEOUT 15373ms, reason connection timeout KERNEL_CONNECTION_TIMEOUT + KERNEL_CONNECTION_FAILED 2528ms, reason media timeout, attempts 1
  • Healthy connects post nothing, so the payload change does not touch viewers that work.
  • The swallowed throw is now reported at 0.4s with its cause, where the deployed client
    waited 15.4s and said only connection timeout.
  • The media bound fires at ~2.5s and names the stage, and is terminal on attempt 1 — no
    retry loop on the deterministic class.
  • One thing to know for the parent-side gate: in the artificial media-stall,
    KERNEL_PLAYING still fired (baseline 15376ms, patched 2532ms). The <video> element
    emits playing even when the peer never connected, so it is not on its own a
    "media is flowing" signal. In the field failure signature (throw) it never fired.
  • A hotpatch validates runtime behavior of the bundle, not image construction or release
    packaging, and the stall is injected at the RTCPeerConnection boundary rather than by
    a real network fault.

Not in this PR

  • The watcher is one vantage point with no last-mile emulation, so these are not mobile
    or satcom numbers. That matters most for signaling, which is the bound derived
    tightest from measurement.
  • signal/provide re-arming the media stage on a renegotiation is unhandled; a second
    provide replaces the peer and restarts the bound, which is the desired behavior, but
    there is no test for it.
  • Reason-branched retry is unchanged: the parent frame still decides whether to remount.

Testing

35 pass, 0 fail in images/chromium-headful/client (bun test tests), including 4 new
cases covering stage entry, the terminal media timeout, and ICE checking clearing the
media bound. vue-cli-service lint clean on changed files.

CI

Test Live View client and Test for the server/ directory (incl. both docker builds,
server unit, server e2e) were dispatched against 9a7a14c and both passed — this repo's
workflows only trigger on PRs targeting main, so a stacked PR gets no PR checks of its
own. Rerun them on retarget.

robertjamesprior and others added 5 commits September 18, 2026 23:39
onMessage is assigned directly to ws.onmessage, so a throw from createPeer
or setRemoteOffer became a discarded rejection: no error, no overlay, no
parent-frame message. The only thing left running was the 15s connect
watchdog, which closed the socket and reconnected into the identical failure
indefinitely.

Route a throw to onDisconnected, cap consecutive attempts that never reach a
connected peer, and post the terminal reason to the parent frame so embedders
can react. An unsupported browser is now terminal too rather than retried.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bound check incremented before comparing, so giving up after three
attempts reported four, contradicting its own message. Count only
attempts that opened a socket.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@robertjamesprior

Copy link
Copy Markdown
Contributor Author

Collapsed into #417. The retry this was stacked to bound is gone — retries are removed entirely, so the stage bounds and the per-stage reason are now the whole change rather than a layer on top of one. #417 carries both and targets main directly. Branch hypeship/staged-live-view-connect-bounds is now orphaned and can be deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant