fix: fal probe handles IN_PROGRESS + infra-skip; re-enable live queue leg#332
Merged
Conversation
commit: |
jpr5
force-pushed
the
fix/fal-probe-in-progress
branch
from
July 23, 2026 01:59
dce5be9 to
ce57d9e
Compare
… leg The live fal queue-lifecycle canary (#327) assumed jobs sit in IN_QUEUE long enough to cancel for $0. Funded flux/schnell returns status:"IN_PROGRESS" on submit immediately; by cancel time the job is running/completed and fal's cancel envelope is {detail:...} (no status field). The leg hard-asserted `typeof cancel.body.status === 'string'` -> AssertionError -> collector exit-5 quarantine -> the drift-live-pr base step red on every PR (descoped in #330). Fix (probe-only, no fal.ts change): - Lifecycle: state-agnostic submit -> status -> CANCEL; accept any valid queued/running/completed status; cancel always fires even if status polls throw. Relax the brittle cancel assertion to require only a JSON envelope ({status} OR {detail}); leave shape grading to triangulate so a genuine 2xx-wrong-shape still reports critical drift. - Infra-skip (folds in #329, which is being closed): 401/402/403 locked-or- exhausted/429/5xx raise a distinguished FalCanarySkip -> honest ctx.skip, never an InfraError that quarantines the baseline. - Re-enable: drop the FAL_LIVE_QUEUE opt-in gate #330 added so the corrected leg runs live in CI again (still gated on FAL_KEY). Cost-safe: cheapest model, cancel ASAP, result endpoint never fetched. Residual: <= one sub-cent generation if the model races to completion. Red-green in fal-canary-skip.test.ts drives the real canary + real triangulate: IN_PROGRESS+{detail} handled (leg passes), 403 -> FalCanarySkip, 2xx-wrong- shape still critical.
jpr5
force-pushed
the
fix/fal-probe-in-progress
branch
2 times, most recently
from
July 23, 2026 02:27
e383942 to
bfc0082
Compare
…le HTTP codes fal's queue API returns DIFFERENT HTTP status codes at each lifecycle step purely as a function of job timing: a fast model (flux/schnell) is IN_PROGRESS on submit; the status poll answers 202 while running vs 200 once COMPLETED; and cancel answers a 2xx (CANCELLATION_REQUESTED) while cancellable vs 4xx (ALREADY_COMPLETED) on a completion race. The live leg hard-asserted specific codes (`statusPoll.status === 200`, `cancel.status in [200,400]`), so whenever the job's timing differed from the lucky COMPLETED-by-poll race the assertion threw the raw envelope as an AssertionError the drift collector could not parse -> exit-5 quarantine -> drift-live-pr red. Root-cause fix (not code-by-code whack-a-mole): STOP asserting timing-variable HTTP status codes at the lifecycle steps entirely. Grade only the ENVELOPE SHAPE via triangulate at every step — the shape is the real drift signal, the status code is timing noise. Submit still requires a 2xx (enqueue success); infra/auth codes (401/402/403/429/5xx) remain the one meaningful class and are converted to an honest FalCanarySkip by the canary before assertion. Anti-cheat preserved: shape grading is untouched, so a genuine 2xx-wrong-shape (wrong-typed field, unknown lifecycle status value) STILL reports critical drift. Extracts the assertion sequence into a pure, unit-testable `assertLiveFalQueueEnvelopes` (fal-queue-contract.ts) with stubbed-state red-green + anti-cheat + code-ignored coverage (fal-queue-contract.test.ts).
jpr5
force-pushed
the
fix/fal-probe-in-progress
branch
from
July 23, 2026 02:33
bfc0082 to
d809d1f
Compare
jpr5
marked this pull request as ready for review
July 23, 2026 02:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Proper fix for the fal live queue-lifecycle drift probe (
falQueueLifecycleCanary+ the live block insrc/__tests__/drift/fal-queue.drift.ts, merged #327), and re-enables the live leg that #330 temporarily descoped. Probe-only — no change to aimock'sfal.tscore.The bug (verified live in CI, root cause)
The probe assumed fal jobs sit in
IN_QUEUElong enough to cancel for $0. Fundedflux/schnellreturnsstatus:"IN_PROGRESS"on the submit envelope immediately. By cancel time the job is running/completed, so fal's cancel envelope is{detail:"..."}— nostatusfield. The live leg hard-assertedtypeof cancel.body.status === "string"→ AssertionError → the collector could not map it → exit-5 quarantine → thedrift-live-prbase step (main's code) went red on every PR. #330 was the fast, reversible descope; this is the proper fix.The fix
IN_QUEUEorIN_PROGRESS…), never assert the job is specifically queued. Cancel still fires even if the status poll throws (a job is never left running).{status}or{detail}); precise shape grading is left totriangulateagainst the mock, so a genuine 2xx-wrong-shape still reports critical drift.401/402/403(locked/exhausted)/429/5xxraise a distinguishedFalCanarySkip→ honestctx.skip, never anInfraErrorthat quarantines the baseline. Includes fix: fal drift probe skips on 403/locked/infra instead of poisoning baseline #329'sfal-canary-skip.test.tsguard tests.FAL_LIVE_QUEUEopt-in gate fix: descope fal live-queue drift leg from CI base (unpoison drift baseline) #330 added; the corrected leg runs live in CI again (still gated onFAL_KEY).Cost safety
Cheapest reliably-available model (
fal-ai/flux/schnell), cancel ASAP, the paidresponse_urlresult endpoint is never fetched. Residual: at most one sub-cent generation if the model races to completion before the immediate cancel lands.Red-green proof (local, drives the real canary + real
triangulate)RED captured against pre-fix probe (origin/main @ d43701c):
GREEN post-fix (
src/__tests__/drift/fal-canary-skip.test.ts, 4/4 pass):{detail}cancel → canary handles it,triangulatereports no critical → leg passes.FalCanarySkip(notInfraError); 401/402 also skip.Checks (local)
prettier✓ ·eslint✓ ·tsc --noEmit(tests-inclusive) 0 errors ✓ ·test:drift16 passed / 9 skipped ✓ · full suite 4693 passed ✓ ·tsdownbuild ✓Live validation (corrected leg against funded fal) = CI
drift-live-pron this PR — see checks below.🤖 Generated with Claude Code
https://claude.ai/code/session_01KamK73Wu5heLxJEogM6hHC