functions serve: intermittent HTTP 500 `WorkerAlreadyRetired` — bootstrap should retry like edge-runtime's reference main service

[basjee]

Affected area

Edge Functions

Supabase CLI version

2.114.0

Operating system

Ubuntu 24.04 (GitHub Actions runner). The runtime behaviour also reproduces on macOS 26.5 with Docker Desktop.

Installation method

GitHub release binary

Command

supabase functions serve --no-verify-jwt --env-file <env-file>

Actual output

wall clock duration warning: isolate: 202bcd02-1526-48d1-acd0-b7b0a281c777
user worker failed to respond: request cannot be handled because the worker has already retired
WorkerAlreadyRetired: request cannot be handled because the worker has already retired
    at async Function.allSettled (<anonymous>)
    at async UserWorker.fetch (ext:user_workers/user_workers.js:85:63)
    at async Object.handler (file:///var/tmp/sb-compile-edge-runtime/root/index.ts:1360:14)
    at async mapped (ext:runtime/http.js:246:20)
early termination has been triggered: isolate: 202bcd02-1526-48d1-acd0-b7b0a281c777

# The client receives the bootstrap's generic fallback, HTTP 500:
{"code":"Internal Server Error","message":"Request failed due to an internal server error","trace":"\"WorkerAlreadyRetired: request cannot be handled because the worker has already retired\\n    at async Function.allSettled (<anonymous>)\\n    ..."}

Under steady local traffic (for example a Playwright suite), a request to a function sometimes gets this HTTP 500 without ever reaching the function. CPU time soft limit reached: isolate: … triggers the same retirement.

Expected behavior

edge-runtime rejects such a request with WorkerAlreadyRetired before dispatch, so the function never saw it. The bootstrap should serve it with a fresh worker, as edge-runtime's reference main service does, instead of answering HTTP 500.

Steps to reproduce

The race window is narrow, so this reproduces it deterministically on edge-runtime v1.74.3, the image that functions serve uses in 2.114.0. A minimal main service dispatches to a worker right after it retired, and answers errors with the CLI bootstrap's fallback byte for byte.

1. Save repro/main/index.ts:

// Deterministic stand-in for the CLI bootstrap's race: /slow creates a worker
// (4 s wall clock, retired at ~2 s) and keeps it busy. The first /fast is
// dispatched to that same worker object, as if `create()` had returned it just
// before retirement, and gets a real WorkerAlreadyRetired. Every later /fast
// calls `create()` like the CLI does, which hands out a fresh worker. Errors
// are answered with the CLI 2.114.0 bootstrap fallback, byte for byte.
let worker: any;
let staleDispatchUsed = false;
const options = {
  servicePath: '/work/fn', memoryLimitMb: 64, workerTimeoutMs: 4000, noModuleCache: false,
  envVars: [], forceCreate: false, cpuTimeSoftLimitMs: 10000, cpuTimeHardLimitMs: 20000,
};
function prepare(req: Request) {
  const clone = new Request(new URL(req.url), req.clone());
  EdgeRuntime.applySupabaseTag(req, clone);
  return clone;
}
Deno.serve(async (req) => {
  if (new URL(req.url).pathname === '/health') return new Response('ok');
  try {
    if (new URL(req.url).pathname.endsWith('/slow')) {
      worker = await EdgeRuntime.userWorkers.create(options);
      return await worker.fetch(prepare(req));
    }
    const target = staleDispatchUsed ? await EdgeRuntime.userWorkers.create(options) : worker;
    staleDispatchUsed = true;
    return await target.fetch(prepare(req));
  } catch (p) {
    console.error(p);
    return new Response(JSON.stringify({ code: 'Internal Server Error', message: 'Request failed due to an internal server error', trace: JSON.stringify(p.stack) }), { status: 500, headers: { 'Content-Type': 'application/json' } });
  }
});

2. Save repro/fn/index.ts:

Deno.serve(async (req) => {
  if (new URL(req.url).pathname.endsWith('/slow')) await new Promise((r) => setTimeout(r, 3000));
  return new Response('fn-ok');
});

3. Start the runtime and wait until curl -s http://127.0.0.1:18081/health answers ok:

docker run --rm -v "$PWD/repro:/work:ro" -p 127.0.0.1:18081:8081 --entrypoint edge-runtime \
  ghcr.io/supabase/edge-runtime:v1.74.3 start --main-service=/work/main --port=8081

4. Run:

curl -s http://127.0.0.1:18081/slow &   # creates the worker (4 s wall clock) and keeps it busy
sleep 2.4; curl -s http://127.0.0.1:18081/fast

At about 2 s the worker is retired while /slow is still in flight. /fast is then dispatched to the retired worker and receives the bootstrap's HTTP 500 carrying WorkerAlreadyRetired. The runtime logs the same four lines as above.

Crash report ID

No response

Docker and service versions

Docker 29.8.0 (local reproduction)
edge-runtime v1.74.3 (ghcr.io/supabase/edge-runtime:v1.74.3), default per_worker policy

Additional context

Cause. Under per_worker, the supervisor retires a reused worker at half its wall-clock budget (workerTimeoutMs 400 s, so at 200 s) or at the 1 s cumulative CPU soft limit (crates/base/src/worker/supervisor/strategy_per_worker.rs).

1. The bootstrap first calls EdgeRuntime.userWorkers.create(), which returns the still-active worker, and then worker.fetch().
2. If the retirement lands between the two, WorkerPool::send_request rejects the dispatch with WorkerAlreadyRetired before the request is delivered (crates/base/src/worker/pool.rs).
3. edge-runtime's reference main service handles exactly this by calling create() again, which hands out a fresh worker (examples/main/index.ts: if (e instanceof Deno.errors.WorkerAlreadyRetired) return await callWorker();).
4. The CLI bootstrap has no such branch (2.114.0 and current develop), so the rejection falls through to the generic 500. DENO_SB_ERROR_MAP covers only InvalidWorkerCreation, InvalidWorkerResponse and WorkerRequestCancelled.

The replay is safe for any method: the rejection happens before demand is incremented and before anything is sent to the worker, so the function never saw the request.

The 500 only appears for requests without a body. With a body, edge-runtime's UserWorker.fetch never settles after this rejection, so the request hangs until the gateway times out. I reported that separately to supabase/edge-runtime.

Impact. Local test suites intermittently see HTTP 500 for requests the function never received. In our CI this failed 1 of about 20 pull-request runs on the day we investigated.

Proposed fix (against develop @ 75b9e81). I checked the retry semantics on edge-runtime v1.74.3: after WorkerAlreadyRetired, a second create() plus fetch() of the same original request, re-prepared with prepareUserRequest, returns the function's response.

--- a/apps/cli/src/shared/functions/serve.main.ts
+++ b/apps/cli/src/shared/functions/serve.main.ts
@@ -1,4 +1,4 @@
-import { Cause, Config, ConfigProvider, Console, Data, Effect, Exit, Option, Schema } from "effect";
+import { Cause, Config, ConfigProvider, Console, Data, Effect, Exit, Option, Schedule, Schema } from "effect";
 import { dirname, join, STATUS_CODE, STATUS_TEXT, toFileUrl } from "./serve-main-deps.ts";
 
 import * as jose from "jose";
@@ -117,6 +117,15 @@
   [Deno.errors.InvalidWorkerResponse, SB_SPECIFIC_ERROR_CODE.InvalidWorkerResponse],
   [Deno.errors.WorkerRequestCancelled, SB_SPECIFIC_ERROR_CODE.WorkerLimit],
 ]);
+
+// edge-runtime rejects a dispatch to a worker that retired (wall-clock or CPU
+// budget) after `userWorkers.create` returned it. The request never reached the
+// function, so it is safe to replay once with a fresh worker.
+function isWorkerAlreadyRetired(error: unknown): boolean {
+  const workerAlreadyRetired = Deno.errors.WorkerAlreadyRetired;
+  return workerAlreadyRetired !== undefined && error instanceof workerAlreadyRetired;
+}
+
 const GENERIC_FUNCTION_SERVE_MESSAGE = `Serving functions on http://127.0.0.1:${HOST_PORT}/functions/v1/<function-name>`;
 export enum RequestErrors {
   MissingAuthHeader = "UNAUTHORIZED_NO_AUTH_HEADER",
@@ -510,6 +519,12 @@
         });
 
         return yield* workerRequest.pipe(
+          // Same recovery as edge-runtime's reference main service
+          // (examples/main/index.ts): `create` hands out a fresh worker.
+          Effect.retry({
+            schedule: Schedule.recurs(1),
+            while: ({ cause }) => isWorkerAlreadyRetired(cause),
+          }),
           Effect.catchTag("BootstrapOperationError", ({ cause }) =>
             Console.error("[functions] worker error", cause).pipe(
               Effect.andThen(Effect.succeed(getWorkerErrorResponse(cause))),

The embedded Go template (templates/main.bundled.js) would need regenerating. Suggested test: stub EdgeRuntime.userWorkers.create so that the first worker's fetch rejects with Deno.errors.WorkerAlreadyRetired. Assert that:

• create runs twice and the second response is returned;
• a second consecutive rejection is not retried;
• any other error is not retried.

I'm happy to open a pull request with this once the issue is marked open-for-contribution.