Skip to content

fix(codex): bind scoped quota suppression to resolved alternate - #576

Open
luvs01 wants to merge 9 commits into
devfrom
codex/propose-fix-for-scoped-quota-failure
Open

luvs01 wants to merge 9 commits into
devfrom
codex/propose-fix-for-scoped-quota-failure

Conversation

@luvs01

@luvs01 luvs01 commented Sep 20, 2026

Copy link
Copy Markdown
Owner

Motivation

  • Scoped organization/project quota codes (HTTP 402/429) were suppressing alternate-account failover before the alternate credential's workspace identity was known, which can wrongly prevent a healthy credential in a different workspace from serving.

Description

  • Preserve the broad bounded failover for pre-stream 402/429 responses until an alternate account has been resolved by making shouldRetryCodexPoolAccountQuota return true initially for those statuses and deferring suppression to post-resolution logic in the retry path.
  • Add shouldRetryCodexScopedQuotaOnAlternate(response, firstWorkspaceAccountId, alternateWorkspaceAccountId) to check scoped exhaustion evidence only after the alternate's chatgptAccountId is known and to withhold rotation only when the alternate is proven to share the same workspace (organization-level cases), while keeping project-scoped exhaustion retryable.
  • Update classification in src/codex/quota-rejection.ts so scoped-quota-exhaustion remains alternateRetryEligible: true at pre-stream classification and let the rotation path perform the binding check once an alternate is resolved.
  • Add focused regression tests in tests/codex-integration/codex-quota-rejection.test.ts and update structure docs (structure/providers/openai-tiers.md, structure/transports/responses.md) to reflect the post-resolution scope check.

Testing

  • Ran the focused regression suite ./node_modules/.bin/bun test tests/codex-integration/codex-quota-rejection.test.ts which passed (89 tests).
  • Ran type checking with ./node_modules/.bin/bun run typecheck, ./node_modules/.bin/bun run structure:check, and ./node_modules/.bin/bun run privacy:scan, all of which completed successfully for the change set.
  • Attempted broader test runs; ./node_modules/.bin/bun run test:changed was not usable in this checkout (no resolvable dev comparison ref), and a full ./node_modules/.bin/bun run test was stopped due to unrelated container/full-suite timeouts in other subsystems, while the focused tests validating this change remained green.

Codex Task


Devin Review

@coderabbitai

coderabbitai Bot commented Sep 20, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository: luvs01/opencodex/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: ea9ce84b-2e92-4651-824b-f90a67d9c34d


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 20, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-20T13:02:38.058200Z 23a3694 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

devin-ai-integration[bot]

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

@github-actions

Copy link
Copy Markdown

Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Sep 20, 2026
devin-ai-integration Bot and others added 3 commits September 20, 2026 13:08
The scoped-quota re-export grew src/server/responses/core.ts past its committed 210-line cap (213). Collapse the two-name re-export back to one line; the file's export list already carries longer single-line statements.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…main

- record the normalized 429/402 outcome before returning no-alternate on a
  suppressed same-workspace move, so a 5xx-wrapped quota refusal still cools
  the refused account instead of reading as transient
- bind a request-owned `main` alternate by the caller credential's own
  workspace id (chatgpt-account-id header, else the bearer token's account
  claim) via callerCodexWorkspaceAccountId
- apply the same scoped-quota workspace gate to the single bounded alternate
  send in the native /responses/compact path
- cover all three in tests and update the transport doc

Co-Authored-By: Epinephrine <luvs01@hanmail.net>
@devin-ai-integration

Copy link
Copy Markdown

Fixed in 5267593. The suppression branch now runs recordWrappedQuotaOutcome() — the same normalized 429/402 recording the no-usable-alternate branch uses — before returning no-alternate, so a 5xx-wrapped scoped refusal still lands the account's quota cooldown (including codexQuotaOutcomeMeta headers, affinity, model, probe, and writer-generation fields) instead of only a transient failure. The shared closure is also used by the no-usable-alternate branch above.

Covered by a new regression test: a 502 carrying organization_spend_limit_exceeded with a same-workspace alternate returns the original 502, sends only to account A, and leaves A with a quota cooldown (cooldownUntil/cooldownSource) rather than transient failure evidence.

@devin-ai-integration

Copy link
Copy Markdown

Fixed in 5267593. handleResponsesCompact now applies shouldRetryCodexScopedQuotaOnAlternate after resolving its single bounded alternate, exactly like the regular path: a proven same-workspace alternate is withheld for organization-scoped 402/429 refusals and released without a send, while distinct/unknown workspaces and project-scoped codes still rotate. structure/transports/responses.md now states that the gate covers the compact endpoint's alternate as well.

Covered by a same-workspace alternate is withheld for a scoped 429/402 refusal in tests/responses/responses-compaction-routing.test.ts (pool-b re-credentialed into pool-a's workspace; the original rejection is preserved with no second send).

@devin-ai-integration

Copy link
Copy Markdown

Fixed in 5267593. kind: "main" alternates are always the request-owned caller credential, so instead of undefined the gate now passes callerCodexWorkspaceAccountId(callerAuthHeaders) — the same chatgpt-account-id that credential would materialize upstream (explicit header first, else the bearer token's chatgpt_account_id claim), matching materializeCodexUpstreamAuth. A same-workspace Pool→main move is now suppressed for organization-scoped codes.

Coverage in tests/server/server-auth.test.ts: for both 429 and 402 scoped refusals the request-owned main (via the chatgpt-account-id header, and separately via the bearer token's account claim) is bound by its workspace id and the retry is withheld — one dispatch, original status preserved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 2 commits September 21, 2026 02:27
…the size cap

server-auth.test.ts grew to 4684 against a 4589 baseline cap, so the
file-size ratchet failed shard 3/4. The three scoped-quota suppression
cases move byte-for-byte into server-auth-scoped-quota.test.ts, and the
pool-retry harness they share is extracted to
tests/helpers/pool-retry-harness.ts (per-run OPENCODEX_HOME dir, so each
importing file keeps its own module state under bun test --isolate).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The workspace classification in shouldRetryCodexScopedQuotaOnAlternate
reads the first response body asynchronously, so a client disconnect can
land after the earlier abort check but before the branch records the
first account, cancels its body, and sends the alternate. Re-check the
abort signal immediately after the await in both paths: compact returns
the 499 client_cancelled response after releasing the alternate lease,
and the regular path releases its permit and lease and returns
no-alternate while still recording the first account's real outcome.

Co-Authored-By: Epinephrine <luvs01@hanmail.net>
@devin-ai-integration

Copy link
Copy Markdown

Fixed in c52b64b. Both paths now re-check the abort signal immediately after shouldRetryCodexScopedQuotaOnAlternate resolves: compact releases the alternate's probe lease and returns the 499 client_cancelled path before recording/cancelling/sending; retryCodexPoolOnAlternateAccount releases its move permit and the alternate lease and returns no-alternate, while still attributing the first account's real outcome (normalized quota or transient) — the refusal is evidence regardless of who is still listening.

devin-ai-integration[bot]

This comment was marked as resolved.

The 499 exits around the scoped-quota classification return a fresh
response while the first rejection's body is still open; cancel it so
the abandoned upstream connection and tee resources are released.

Co-Authored-By: Epinephrine <luvs01@hanmail.net>
@devin-ai-integration

Copy link
Copy Markdown

Fixed in 813efccupstream.body?.cancel() applied as suggested, on both abort checks in this region (the post-resolution one above had the same discarded-body leak).

devin-ai-integration[bot]

This comment was marked as resolved.

upstream.body.cancel() can wait on a custom or stalled source; awaiting
it at the abort checkpoints would park the 499 reply on cleanup. Fire
it with the request's abort reason and swallow rejection, the same
best-effort shape bufferCompactResponse already uses.

Co-Authored-By: Epinephrine <luvs01@hanmail.net>
@devin-ai-integration

Copy link
Copy Markdown

Fixed in 2d1ee69 — both checkpoints now fire upstream.body?.cancel(req.signal.reason) unawaited with .catch(() => undefined), the same best-effort shape bufferCompactResponse uses at line ~570, so a never-settling cancel can't park the 499 reply.

…sses.test.ts)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aardvark bug Something isn't working codex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant