Skip to content

[SDK Sentinel] Fix WorkerFactory shutdown await race - #3093

Open
sdk-sentinel-bot wants to merge 1 commit into
temporalio:mainfrom
sdk-sentinel-forks:automation/sdk-sentinel/ci-flake/java/35546651673-1-97ec2ebdb3de
Open

sdk-sentinel-bot wants to merge 1 commit into
temporalio:mainfrom
sdk-sentinel-forks:automation/sdk-sentinel/ci-flake/java/35546651673-1-97ec2ebdb3de

Conversation

@sdk-sentinel-bot

Copy link
Copy Markdown
Contributor

Caution

This PR contains untrusted AI-generated code. Do not approve or run CI until a maintainer has reviewed the diff. SDK Sentinel verified that GitHub Actions remained approval-gated with zero executable jobs when this PR was opened.

Summary

Prevent WorkerFactory.awaitTermination from returning before asynchronous worker shutdown completes. The [first captured Edge failure](https://github.com/temporalio/sdk-java/actions/runs/35512726248/job/106083372368) affected the autoscaling sticky-poller variant; no preceding equivalent success was available.

Root cause

Poller shutdown and task-executor shutdown became sequential future stages. awaitTermination could observe the poller as terminated before its continuation initiated task-executor shutdown, then return while the factory remained active.

Fix

Retain the aggregate shutdown future and await it using the caller's remaining timeout. Extend the lifecycle regression test to require this additional wait. There are no API, replay, workflow-history, breaking, or server changes.

Validation

The regression suite and both shutdown-now poller variants pass on Temurin JDK 23. Formatting and all test classes compile. The unchanged canonical baseline passed, but candidate canonical validation was blocked by sandbox ownership of Gradle and Git submodule metadata. The Edge JDK 23 job and canonical validator remain required.

Validation status: validation-incomplete

  • Flake confidence: high
  • Fix confidence: high
  • Value: medium
  • Patch scope: standard — 2 changed files; 3.8 KiB
  • Local reproduction: not-reproduced — One unchanged focused run passed on Temurin JDK 23.0.2; the captured timing, parameter contrast, and asynchronous shutdown chain exposed the race causally.
  • Regression coverage: pass-after-only — The regression assertion verifies that awaitTermination waits both the command worker and aggregate asynchronous shutdown chain; it and the original failing test pass after the fix.
  • Unchanged baseline (Gradle formatting and test-source compilation) — passed
  • Independent candidate (Gradle formatting and test-source compilation) — did not pass (failure)

Investigator-run checks

  • GRADLE_USER_HOME=.ci-flake-runtime/tmp/gradle-home ./gradlew --offline --no-daemon :temporal-sdk:test -x :temporal-serviceclient:updateSubmodules -x :temporal-sdk:compileJava17Java --tests 'io.temporal.worker.shutdown.StickyWorkflowDrainShutdownTest.testShutdownNow*' — passed; 1 attempt(s). The unchanged test passed locally on Temurin JDK 23.0.2; natural reproduction did not occur.
  • GRADLE_USER_HOME=.ci-flake-runtime/tmp/gradle-home ./gradlew --offline --no-daemon :temporal-sdk:test -x :temporal-serviceclient:updateSubmodules -x :temporal-sdk:compileJava17Java --tests 'io.temporal.worker.WorkerShutdownTest' --tests 'io.temporal.worker.shutdown.StickyWorkflowDrainShutdownTest.testShutdownNow*' — passed; 1 attempt(s). The final candidate passed the regression suite and both shutdown-now poller variants.
  • .ci-flake-runtime/input/validate.sh — failed; 2 attempt(s). The trusted unchanged baseline passed. Candidate attempts were sandbox-blocked by the root-owned Gradle daemon registry and read-only Git submodule metadata before patch compilation.
  • GRADLE_USER_HOME=.ci-flake-runtime/tmp/gradle-home ./gradlew --offline --no-daemon spotlessCheck testClasses -x test -x :temporal-serviceclient:updateSubmodules -x :temporal-sdk:compileJava17Java — passed; 1 attempt(s). The permitted equivalent formatting and test-compilation check passed for all modules.

Required target CI

  • Fixed repository validator: .ci-flake-runtime/input/validate.sh
  • Continuous Integration / Unit test with in-memory test service [Edge] on Linux with Temurin JDK 23

Residual risks

  • The bounded evidence contains only one CI occurrence and no same-SHA rerun.
  • Canonical candidate validation was blocked by sandbox filesystem permissions.
  • The full test suite was not executed locally; formatting and all test classes compiled with sandbox-specific exclusions.

Automation provenance

If this finding should not be fixed, apply the sdk-sentinel:false-positive label and close the PR. Sentinel will suppress the finding until its affected source changes.

SDK Sentinel recurrence history

  • Unique failed CI run attempts: 1
  • First occurrence: 2026-09-20 13:21 UTC
  • Latest occurrence: 2026-09-20 13:21 UTC
  • Recent occurrences:

@sdk-sentinel-bot
sdk-sentinel-bot requested a review from a team as a code owner September 21, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant