Skip to content

fix(observability): bound orphaned-run finalization POST during shutdown (SDK-7518) - #180

Open
rounak610 wants to merge 2 commits into
mainfrom
fix/sdk-7518-orphan-finalization-timeout
Open

fix(observability): bound orphaned-run finalization POST during shutdown (SDK-7518)#180
rounak610 wants to merge 2 commits into
mainfrom
fix/sdk-7518-orphan-finalization-timeout

Conversation

@rounak610

Copy link
Copy Markdown
Collaborator

What is this about?

Observability builds occasionally stay "running" for hours after the run has ended (reported for WebdriverIO + App Automate with multiple products enabled).

During shutdown, finalizeOrphanedRuns() posts synthetic TestRunFinished/HookRunFinished events for any runs orphaned by a killed worker — and it does so immediately before the build-stop call, in both the onComplete path (launcher.ts) and the detached exit-cleanup rescue (cleanup.ts). That POST (batchAndPostEvents) had no timeout, unlike the sibling stopBuildUpstream (hardened in SDK-7229 with an AbortController + deadline) and nodeRequest. On a transient network blip the request could hang indefinitely, so stopBuildUpstream was never reached — leaving the build "running" until the server-side inactivity timeout (~60 min).

It only fires when an orphaned run exists (a worker killed mid-test), so it is intermittent and correlates with heavier, multi-product runs (more chance a worker is killed mid-test).

Fix: add an optional per-call timeoutMs to batchAndPostEvents and have finalizeOrphanedRuns pass ORPHAN_FINALIZE_POST_TIMEOUT_MS (10s), mirroring the already-hardened stopBuildUpstream. Normal in-run event uploads pass no timeout and are unchanged (minimal blast radius).

Related Jira task/s

  • SDK-7518 (reopen of SDK-7229 — the earlier "builds running forever" fix hardened stopBuildUpstream; this closes the remaining unbounded call on the same shutdown path)

Release (mandatory for every PR — required for the ready-for-review label)

Version bump: (required — tick exactly one)

  • minor (backwards-compatible feature)
  • patch (bug fix or other small change)

Release notes type: (optional)

  • New Feature
  • Bug Fix
  • Other Improvement

Release notes (customer-facing): (optional but encouraged)

  • Fixed builds occasionally showing as "running" for hours after the run finished, caused by an unbounded network request during shutdown.

Release notes (internal): (required — engineer-facing; what actually changed / why)

  • batchAndPostEvents gains an optional timeoutMs; when set it bounds the fetch with an AbortController. finalizeOrphanedRuns passes ORPHAN_FINALIZE_POST_TIMEOUT_MS (10s) so the shutdown-path orphan finalization can no longer hang and stall stopBuildUpstream. All other batchAndPostEvents callers are unchanged.

Checklist

  • Ready to review
  • Has it been tested locally?

PR Validations

Run Tests: Comment RUN_TESTS to trigger sanity tests.


🤖 Generated with Claude Code

…own (SDK-7518)

finalizeOrphanedRuns() posts synthetic TestRun/HookRunFinished events immediately
before the build-stop call, in both the onComplete path and the detached
exit-cleanup rescue. That POST had no timeout, so a hung connection could stall
shutdown before stopBuildUpstream ran, leaving the build "running" until the
server-side inactivity timeout (~60 min). It fires only when an orphaned run
exists (a worker killed mid-test), which is why it is intermittent and shows up
more with multiple products enabled.

Bound it with an optional per-call timeout on batchAndPostEvents, passed by
finalizeOrphanedRuns (ORPHAN_FINALIZE_POST_TIMEOUT_MS), mirroring the already-
hardened stopBuildUpstream. Normal in-run event uploads pass no timeout and are
unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@rounak610
rounak610 requested a review from a team as a code owner September 7, 2026 10:57
@rounak610
rounak610 requested review from 07souravkunda and removed request for pri-gadhiya September 7, 2026 11:31

@shivam5643 shivam5643 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Good to go

0 blocking findings. Every diff region judged clean (8/8 across 4 files).

What this changes

O11y builds occasionally stayed running for ~an hour after the run ended. During shutdown, finalizeOrphanedRuns() POSTs synthetic TestRunFinished/HookRunFinished events for runs orphaned by a killed worker, immediately before the build-stop call. That POST (batchAndPostEvents) had no timeout, unlike its sibling stopBuildUpstream (hardened under SDK-7229). On a transient network blip it could hang indefinitely, so stopBuildUpstream was never reached and the build sat until the server-side inactivity reap.

The fix:

  • util.tsbatchAndPostEvents gains an optional 4th param timeoutMs?, wiring an AbortController into the fetch signal, with the timer cleared in a finally on every outcome.
  • constants.ts — new ORPHAN_FINALIZE_POST_TIMEOUT_MS = 10000, mirroring STOP_BUILD_ATTEMPT_TIMEOUT_MS.
  • openRunsJournal.tsfinalizeOrphanedRuns() passes that constant through.
  • Changeset (patch bump) included and consistent with the code change.

Why it holds up

  • Mirrors an already-hardened sibling. Same AbortController + setTimeout + finally-cleared timer shape as stopBuildUpstream, so no timer leak on the success path.
  • The unrelated caller is provably unaffected. The param is optional; listener.ts's in-run BATCH_DATA upload passes nothing, so signal: controller?.signal resolves to undefined exactly as before.
  • Abort degrades safely. An aborted fetch rejects → existing catch in batchAndPostEvents → rethrown → finalizeOrphanedRuns' pre-existing outer catch logs at debug and returns 0. A timeout here cannot crash shutdown or surface in the customer's test run — consistent with the graceful-degradation requirement.
  • Both shutdown call sites inherit the fix without needing a diff — launcher.ts (onComplete) and cleanup.ts (detached exit-rescue) both call finalizeOrphanedRuns(); confirmed by code search.

Coverage ledger

Unit Files Regions Verdict
u001 .changeset/pr-180.md 1 clean
u002 constants.ts, openRunsJournal.ts, util.ts 7 clean
cross-cutting whole diff pass

8/8 regions judged, regions_unjudged: []. No unit hit high-risk, so no second pass. No falsification pass — nothing was flagged to falsify.


🤖 Automated review via Claude Code. Posted as a COMMENT-event recommendation, not a formal approval.

@rounak610

Copy link
Copy Markdown
Collaborator Author

RUN_TESTS

@rounak610

Copy link
Copy Markdown
Collaborator Author

RUN_TESTS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants