Keep time skipping after activity timeouts

[1fanwang]

What was changed

Tests can stall after an activity calls testEnv.sleep() past its start-to-close timeout. This keeps time skipping working so retries and later workflow timers can complete.

Why?

The sleep temporarily releases a time-skipping lock. If the activity times out before the sleep returns, the balance can briefly become negative. The test server throws java.lang.IllegalStateException: Unbalanced lock and unlock calls: instead of recording that decrement. When the sleep returns, it leaves an extra lock.

Lock handles still reject a second unlock.

Checklist

1. Closes Time-skipping past activity’s startToCloseTimeout causes worker not to close #2246
2. How was this tested: A TypeScript worker connected over gRPC to a test server built from source. The same delayed-activity case stalled on the base and completed with this change.
3. Any docs updates needed? No public API or coordinated server change is needed.

Testing Done

JDK 21, Node 22, Temporal TypeScript 1.24.0. The unmodified base was 961a35e.

#	Scenario	Command	Observed result
1	Unmodified base	node run.cjs	The 20-second watchdog fired; the process exited 1.
2	This change	node run.cjs	The workflow completed after two attempts; the process exited 0.

Raw output from the base:

WATCHDOG: workflow did not complete within 20 seconds
workflow runtime: 27.692s

Raw output with the fix:

{"result":"Hello, Temporal!","attempts":[1,2]}
workflow runtime: 2.005s

The late-completion NOT_FOUND warning remains expected for an attempt that has already timed out.

Self-contained runtime reproduction

Build the checked-out revision and use its executable, rather than a downloaded test-server release:

./gradlew --no-daemon --max-workers=2 :temporal-test-server:installDist
export TEST_SERVER_PATH="$PWD/temporal-test-server/build/install/temporal-test-server/bin/temporal-test-server"

In a separate directory, save these three files. Keep TEST_SERVER_PATH pointed at the checked-out server.

package.json:

{
  "name": "temporal-2246-repro",
  "private": true,
  "version": "0.0.0",
  "dependencies": {
    "@temporalio/activity": "1.24.0",
    "@temporalio/testing": "1.24.0",
    "@temporalio/worker": "1.24.0",
    "@temporalio/workflow": "1.24.0"
  }
}

workflows.js:

const { proxyActivities } = require("@temporalio/workflow");

const { greet } = proxyActivities({ startToCloseTimeout: "1 minute" });

async function example(name) {
  return await greet(name);
}

module.exports = { example };

run.cjs:

const assert = require("node:assert/strict");
const { activityInfo } = require("@temporalio/activity");
const { TestWorkflowEnvironment } = require("@temporalio/testing");
const { DefaultLogger, Runtime, Worker } = require("@temporalio/worker");

Runtime.install({ logger: new DefaultLogger("WARN") });

async function run(serverPath) {
  assert.ok(serverPath, "Set TEST_SERVER_PATH to the built test-server executable.");
  const environment = await TestWorkflowEnvironment.createTimeSkipping({
    client: { identity: "activity-timeout-repro" },
    server: { executable: { type: "existing-path", path: serverPath } },
  });
  let waited = false;
  let watchdog;
  const attempts = [];
  console.time("workflow runtime");
  try {
    const worker = await Worker.create({
      connection: environment.nativeConnection,
      identity: "activity-timeout-repro",
      taskQueue: "activity-timeout-repro",
      workflowsPath: require.resolve("./workflows.js"),
      shutdownGraceTime: "1 second",
      shutdownForceTime: "5 seconds",
      activities: {
        greet: async (name) => {
          attempts.push(activityInfo().attempt);
          if (!waited) {
            await environment.sleep("61 seconds");
            waited = true;
          }
          return "Hello, " + name + "!";
        },
      },
    });
    watchdog = setTimeout(() => {
      console.error("WATCHDOG: workflow did not complete within 20 seconds");
      worker.shutdown();
    }, 20_000);
    const result = await worker.runUntil(
      () => environment.client.workflow.execute("example", {
        workflowId: "activity-timeout-repro",
        taskQueue: "activity-timeout-repro",
        args: ["Temporal"],
      }),
      { promiseCompletionTimeout: "1 second" },
    );
    assert.equal(result, "Hello, Temporal!");
    assert.ok(attempts.length >= 2);
    console.log(JSON.stringify({ result, attempts }));
  } finally {
    clearTimeout(watchdog);
    await environment.teardown();
    console.timeEnd("workflow runtime");
  }
}

run(process.env.TEST_SERVER_PATH).catch(error => {
  console.error(error);
  process.exitCode = 1;
});

Run:

npm install --ignore-scripts --no-audit --no-fund --registry=https://registry.npmjs.org
node run.cjs

Controls for normal completion, bounded retries, and default retries followed by termination also passed. Java validation passed across the test-server suite, selected SDK timeout/timer cases, and testing-module checks.

• Local code review completed: timer accounting, duplicate-unlock protection, and regression coverage.