Skip to content

fix: fal probe handles IN_PROGRESS + infra-skip; re-enable live queue leg#332

Merged
jpr5 merged 2 commits into
mainfrom
fix/fal-probe-in-progress
Jul 23, 2026
Merged

fix: fal probe handles IN_PROGRESS + infra-skip; re-enable live queue leg#332
jpr5 merged 2 commits into
mainfrom
fix/fal-probe-in-progress

Conversation

@jpr5

@jpr5 jpr5 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

What

Proper fix for the fal live queue-lifecycle drift probe (falQueueLifecycleCanary + the live block in src/__tests__/drift/fal-queue.drift.ts, merged #327), and re-enables the live leg that #330 temporarily descoped. Probe-only — no change to aimock's fal.ts core.

The bug (verified live in CI, root cause)

The probe assumed fal jobs sit in IN_QUEUE long enough to cancel for $0. Funded flux/schnell returns status:"IN_PROGRESS" on the submit envelope immediately. By cancel time the job is running/completed, so fal's cancel envelope is {detail:"..."}no status field. The live leg hard-asserted typeof cancel.body.status === "string" → AssertionError → the collector could not map it → exit-5 quarantine → the drift-live-pr base step (main's code) went red on every PR. #330 was the fast, reversible descope; this is the proper fix.

The fix

  1. Lifecycle — state-agnostic submit → status → CANCEL. Accept any 2xx submit and any valid queued/running/completed status (IN_QUEUE or IN_PROGRESS …), never assert the job is specifically queued. Cancel still fires even if the status poll throws (a job is never left running).
  2. Envelope shapes / drift — the cancel assertion now requires only a JSON envelope ({status} or {detail}); precise shape grading is left to triangulate against the mock, so a genuine 2xx-wrong-shape still reports critical drift.
  3. Infra-skip (folds in fix: fal drift probe skips on 403/locked/infra instead of poisoning baseline #329, which is being closed in favor of this PR)401/402/403 (locked/exhausted) /429/5xx raise a distinguished FalCanarySkip → honest ctx.skip, never an InfraError that quarantines the baseline. Includes fix: fal drift probe skips on 403/locked/infra instead of poisoning baseline #329's fal-canary-skip.test.ts guard tests.
  4. Re-enable — removes the FAL_LIVE_QUEUE opt-in gate fix: descope fal live-queue drift leg from CI base (unpoison drift baseline) #330 added; the corrected leg runs live in CI again (still gated on FAL_KEY).

Cost safety

Cheapest reliably-available model (fal-ai/flux/schnell), cancel ASAP, the paid response_url result endpoint is never fetched. Residual: at most one sub-cent generation if the model races to completion before the immediate cancel lands.

Red-green proof (local, drives the real canary + real triangulate)

RED captured against pre-fix probe (origin/main @ d43701c):

× RED(a): IN_PROGRESS submit + {detail} cancel — old assert `typeof cancel.body.status === "string"`
    → AssertionError: expected 'undefined' to be 'string'
× RED(b): 403 user-locked at submit
    → rejects with InfraError: "INFRA_ERROR: fal Queue Lifecycle …" (quarantine-causing, not a skip)

GREEN post-fix (src/__tests__/drift/fal-canary-skip.test.ts, 4/4 pass):

  • (a) IN_PROGRESS submit + {detail} cancel → canary handles it, triangulate reports no critical → leg passes.
  • (b) 403 user-locked → FalCanarySkip (not InfraError); 401/402 also skip.
  • (c) guard: 2xx submit whose envelope diverges from the mock → still critical drift (fix did not swallow genuine drift).

Checks (local)

prettier ✓ · eslint ✓ · tsc --noEmit (tests-inclusive) 0 errors ✓ · test:drift 16 passed / 9 skipped ✓ · full suite 4693 passed ✓ · tsdown build ✓

Live validation (corrected leg against funded fal) = CI drift-live-pr on this PR — see checks below.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KamK73Wu5heLxJEogM6hHC

@pkg-pr-new

pkg-pr-new Bot commented Jul 23, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@332

commit: d809d1f

@jpr5
jpr5 force-pushed the fix/fal-probe-in-progress branch from dce5be9 to ce57d9e Compare July 23, 2026 01:59
… leg

The live fal queue-lifecycle canary (#327) assumed jobs sit in IN_QUEUE long
enough to cancel for $0. Funded flux/schnell returns status:"IN_PROGRESS"
on submit immediately; by cancel time the job is running/completed and fal's
cancel envelope is {detail:...} (no status field). The leg hard-asserted
`typeof cancel.body.status === 'string'` -> AssertionError -> collector
exit-5 quarantine -> the drift-live-pr base step red on every PR (descoped in
#330).

Fix (probe-only, no fal.ts change):
- Lifecycle: state-agnostic submit -> status -> CANCEL; accept any valid
  queued/running/completed status; cancel always fires even if status polls
  throw. Relax the brittle cancel assertion to require only a JSON envelope
  ({status} OR {detail}); leave shape grading to triangulate so a genuine
  2xx-wrong-shape still reports critical drift.
- Infra-skip (folds in #329, which is being closed): 401/402/403 locked-or-
  exhausted/429/5xx raise a distinguished FalCanarySkip -> honest ctx.skip,
  never an InfraError that quarantines the baseline.
- Re-enable: drop the FAL_LIVE_QUEUE opt-in gate #330 added so the corrected
  leg runs live in CI again (still gated on FAL_KEY).

Cost-safe: cheapest model, cancel ASAP, result endpoint never fetched.
Residual: <= one sub-cent generation if the model races to completion.

Red-green in fal-canary-skip.test.ts drives the real canary + real triangulate:
IN_PROGRESS+{detail} handled (leg passes), 403 -> FalCanarySkip, 2xx-wrong-
shape still critical.
@jpr5
jpr5 force-pushed the fix/fal-probe-in-progress branch 2 times, most recently from e383942 to bfc0082 Compare July 23, 2026 02:27
…le HTTP codes

fal's queue API returns DIFFERENT HTTP status codes at each lifecycle step
purely as a function of job timing: a fast model (flux/schnell) is IN_PROGRESS
on submit; the status poll answers 202 while running vs 200 once COMPLETED; and
cancel answers a 2xx (CANCELLATION_REQUESTED) while cancellable vs 4xx
(ALREADY_COMPLETED) on a completion race. The live leg hard-asserted specific
codes (`statusPoll.status === 200`, `cancel.status in [200,400]`), so whenever
the job's timing differed from the lucky COMPLETED-by-poll race the assertion
threw the raw envelope as an AssertionError the drift collector could not parse
-> exit-5 quarantine -> drift-live-pr red.

Root-cause fix (not code-by-code whack-a-mole): STOP asserting timing-variable
HTTP status codes at the lifecycle steps entirely. Grade only the ENVELOPE
SHAPE via triangulate at every step — the shape is the real drift signal, the
status code is timing noise. Submit still requires a 2xx (enqueue success);
infra/auth codes (401/402/403/429/5xx) remain the one meaningful class and are
converted to an honest FalCanarySkip by the canary before assertion.

Anti-cheat preserved: shape grading is untouched, so a genuine 2xx-wrong-shape
(wrong-typed field, unknown lifecycle status value) STILL reports critical
drift. Extracts the assertion sequence into a pure, unit-testable
`assertLiveFalQueueEnvelopes` (fal-queue-contract.ts) with stubbed-state
red-green + anti-cheat + code-ignored coverage (fal-queue-contract.test.ts).
@jpr5
jpr5 force-pushed the fix/fal-probe-in-progress branch from bfc0082 to d809d1f Compare July 23, 2026 02:33
@jpr5
jpr5 marked this pull request as ready for review July 23, 2026 02:36
@jpr5
jpr5 merged commit 3565ec4 into main Jul 23, 2026
27 checks passed
@jpr5
jpr5 deleted the fix/fal-probe-in-progress branch July 23, 2026 02:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant