fix(observability): bound orphaned-run finalization POST during shutdown (SDK-7518) - #180
fix(observability): bound orphaned-run finalization POST during shutdown (SDK-7518)#180rounak610 wants to merge 2 commits into
Conversation
…own (SDK-7518) finalizeOrphanedRuns() posts synthetic TestRun/HookRunFinished events immediately before the build-stop call, in both the onComplete path and the detached exit-cleanup rescue. That POST had no timeout, so a hung connection could stall shutdown before stopBuildUpstream ran, leaving the build "running" until the server-side inactivity timeout (~60 min). It fires only when an orphaned run exists (a worker killed mid-test), which is why it is intermittent and shows up more with multiple products enabled. Bound it with an optional per-call timeout on batchAndPostEvents, passed by finalizeOrphanedRuns (ORPHAN_FINALIZE_POST_TIMEOUT_MS), mirroring the already- hardened stopBuildUpstream. Normal in-run event uploads pass no timeout and are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
shivam5643
left a comment
There was a problem hiding this comment.
✅ Good to go
0 blocking findings. Every diff region judged clean (8/8 across 4 files).
What this changes
O11y builds occasionally stayed running for ~an hour after the run ended. During shutdown, finalizeOrphanedRuns() POSTs synthetic TestRunFinished/HookRunFinished events for runs orphaned by a killed worker, immediately before the build-stop call. That POST (batchAndPostEvents) had no timeout, unlike its sibling stopBuildUpstream (hardened under SDK-7229). On a transient network blip it could hang indefinitely, so stopBuildUpstream was never reached and the build sat until the server-side inactivity reap.
The fix:
util.ts—batchAndPostEventsgains an optional 4th paramtimeoutMs?, wiring anAbortControllerinto thefetchsignal, with the timer cleared in afinallyon every outcome.constants.ts— newORPHAN_FINALIZE_POST_TIMEOUT_MS = 10000, mirroringSTOP_BUILD_ATTEMPT_TIMEOUT_MS.openRunsJournal.ts—finalizeOrphanedRuns()passes that constant through.- Changeset (patch bump) included and consistent with the code change.
Why it holds up
- Mirrors an already-hardened sibling. Same
AbortController+setTimeout+finally-cleared timer shape asstopBuildUpstream, so no timer leak on the success path. - The unrelated caller is provably unaffected. The param is optional;
listener.ts's in-runBATCH_DATAupload passes nothing, sosignal: controller?.signalresolves toundefinedexactly as before. - Abort degrades safely. An aborted fetch rejects → existing
catchinbatchAndPostEvents→ rethrown →finalizeOrphanedRuns' pre-existing outercatchlogs at debug and returns0. A timeout here cannot crash shutdown or surface in the customer's test run — consistent with the graceful-degradation requirement. - Both shutdown call sites inherit the fix without needing a diff —
launcher.ts(onComplete) andcleanup.ts(detached exit-rescue) both callfinalizeOrphanedRuns(); confirmed by code search.
Coverage ledger
| Unit | Files | Regions | Verdict |
|---|---|---|---|
| u001 | .changeset/pr-180.md |
1 | clean |
| u002 | constants.ts, openRunsJournal.ts, util.ts |
7 | clean |
| cross-cutting | whole diff | — | pass |
8/8 regions judged, regions_unjudged: []. No unit hit high-risk, so no second pass. No falsification pass — nothing was flagged to falsify.
🤖 Automated review via Claude Code. Posted as a COMMENT-event recommendation, not a formal approval.
|
RUN_TESTS |
|
RUN_TESTS |
What is this about?
Observability builds occasionally stay "running" for hours after the run has ended (reported for WebdriverIO + App Automate with multiple products enabled).
During shutdown,
finalizeOrphanedRuns()posts syntheticTestRunFinished/HookRunFinishedevents for any runs orphaned by a killed worker — and it does so immediately before the build-stop call, in both theonCompletepath (launcher.ts) and the detached exit-cleanup rescue (cleanup.ts). That POST (batchAndPostEvents) had no timeout, unlike the siblingstopBuildUpstream(hardened in SDK-7229 with anAbortController+ deadline) andnodeRequest. On a transient network blip the request could hang indefinitely, sostopBuildUpstreamwas never reached — leaving the build "running" until the server-side inactivity timeout (~60 min).It only fires when an orphaned run exists (a worker killed mid-test), so it is intermittent and correlates with heavier, multi-product runs (more chance a worker is killed mid-test).
Fix: add an optional per-call
timeoutMstobatchAndPostEventsand havefinalizeOrphanedRunspassORPHAN_FINALIZE_POST_TIMEOUT_MS(10s), mirroring the already-hardenedstopBuildUpstream. Normal in-run event uploads pass no timeout and are unchanged (minimal blast radius).Related Jira task/s
stopBuildUpstream; this closes the remaining unbounded call on the same shutdown path)Release (mandatory for every PR — required for the
ready-for-reviewlabel)Version bump: (required — tick exactly one)
Release notes type: (optional)
Release notes (customer-facing): (optional but encouraged)
Release notes (internal): (required — engineer-facing; what actually changed / why)
batchAndPostEventsgains an optionaltimeoutMs; when set it bounds thefetchwith anAbortController.finalizeOrphanedRunspassesORPHAN_FINALIZE_POST_TIMEOUT_MS(10s) so the shutdown-path orphan finalization can no longer hang and stallstopBuildUpstream. All otherbatchAndPostEventscallers are unchanged.Checklist
PR Validations
Run Tests: Comment RUN_TESTS to trigger sanity tests.
🤖 Generated with Claude Code