E2E flake: global-setup prereq check reds a shard on transient Supabase auth admin-API {} timeouts (28-shard load)

[TortoiseWolfe]

Summary

The E2E global-setup.ts prerequisite check intermittently reds a shard when a Supabase auth admin-API call transiently times out — surfacing as an empty {} error. It is not a corrupted user and not the failing PR's diff; the same shared users are healthy. Under the full 28-shard push-to-main matrix (chromium + firefox + webkit), the concurrent load spikes the auth admin API and a single transient failure fails that shard's entire setup, red-lining main until a re-run.

This is the second, distinct failure mode from #338's shared-user corruption (where the user signs in fine but has no profile/keys → 406). Here the admin API call itself errors {}.

Evidence (2026-07-24, shipping #321 / PR #344)

• PR run 30091431004 (chromium-only): chromium-msg + chromium-msg-iso failed at global-setup:
  ❌ PRIMARY user sign-in failed: {} (~50s), while all 6 chromium-gen shards + the RLS/conformance gate passed. Re-ran → green.
• Post-merge run 30094104243 on f597f369 (full 28-shard):
  • webkit-gen 3/6, firefox-gen 3/6: ❌ Could not verify PRIMARY/TERTIARY exists — listUsers error: {} (~26s)
  • chromium-gen 4/6: payment/06-realtime-dashboard.spec.ts:124 60s timeout (co-occurring flaky-watch under the same load)
  • Re-ran the 4 failed jobs → green.
• Users verified healthy throughout: direct GoTrue password grant for both PRIMARY and TERTIARY returned HTTP 200 (curl -4 …/auth/v1/token?grant_type=password). The {} was a transient admin-API timeout, not a missing/corrupted user.

Root cause

tests/e2e/global-setup.ts makes three admin-API calls whose transient failure pushes a hard prereq error (→ throw at L232, aborting the shard), with no retry:

1. L86 — adminClient.auth.admin.listUsers({ perPage: 1 }) connectivity probe (catch at L96).
2. L138-183 — per-user existence check via paginated listUsers; a listUsers error becomes enumerationError → Could not verify ${name} exists — listUsers error: {} (L171).
3. L197-220 — PRIMARY signInWithPassword verify → PRIMARY user sign-in failed: {} (L215).

#338 deliberately made a listUsers error fail loudly instead of destructively recreating the shared user (the right call — that recreate path is what corrupted the user). But "fail loudly" on a transient now reds the shard for a non-issue. The missing piece is retry/backoff: distinguish "the API blipped" from "the prerequisite is genuinely unmet."

Proposed fix

• Wrap each of the three admin-API calls (connectivity listUsers, per-user listUsers, PRIMARY signInWithPassword) in a retry-with-backoff helper (e.g. 3 attempts, 2-4s backoff) before recording a prereq error. Only an error that persists across retries should red the shard. Empty/{}/timeout errors are exactly the retryable class.
• Consider staggering global-setup start or lowering shard concurrency so 28 shards don't hit the auth admin API simultaneously (secondary — the retry is the primary fix).
• Keep payment/06-realtime-dashboard.spec.ts on the flaky-watch (its 60s timeout co-occurs under the same load).

Acceptance

• A single transient {}/timeout on any of the three admin-API calls no longer fails a shard's setup (retried).
• A genuine prereq failure (invalid creds, missing env, sustained outage) still fails fast and loudly after retries are exhausted — the fix(e2e): harden test-user cleanup so a partial delete can't corrupt the shared user #338 no-destructive-recreate guarantee is preserved.
• Document the retry in the prereq-check comments.

Refs

• fix(e2e): harden test-user cleanup so a partial delete can't corrupt the shared user #338 / Test-infra: apply #338's FK-safe delete cleanup to the 2nd deleteTestUser + cleanup-stale-impl (webhook_events gap) #341 — the shared-user corruption fix (the other failure mode; do not conflate).
• E2E flaky watch: game-3d touch-drag camera (new on #297's run; check main's next run) #300 — flaky-gate arc. Keep FLAKY_GATE_MODE=annotate in e2e.yml until this is hardened; this is a real recurring source of main-red-then-green-on-rerun.
• Memory: lesson_e2e_shared_user_corruption.md documents how to tell the two modes apart.