fix(anthropic): complete AgentRouter streams that end before terminal frames - #896
fix(anthropic): complete AgentRouter streams that end before terminal frames#896Yuxin-Qiao wants to merge 1 commit into
Conversation
📝 WalkthroughWalkthroughAdds ChangesAnthropic EOF tolerance
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant AgentRouter
participant AnthropicAdapter
participant ToolInputParser
participant OpenCodexClient
AgentRouter->>AnthropicAdapter: stream content and tool JSON deltas
AgentRouter-->>AnthropicAdapter: EOF before message_stop
AnthropicAdapter->>ToolInputParser: validate accumulated tool input
ToolInputParser-->>AnthropicAdapter: complete JSON object or invalid input
AnthropicAdapter->>OpenCodexClient: completion or truncation error
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs-site/src/content/docs/reference/configuration/providers.md`:
- Line 94: Update the Anthropic Messages streaming documentation in the adapters
reference to list the supported SSE lifecycle events: message_start,
content_block_start, content_block_delta, content_block_stop, message_delta,
message_stop, and error. Document that strict EOF handling fails truncated
streams, while anthropicEofTolerance permits early completion only after visible
text or complete JSON-object tool input; incomplete or contentless streams must
still fail.
In `@src/adapters/anthropic.ts`:
- Around line 992-1010: Track completed tool calls separately from
currentToolCallId in the EOF handling around the Anthropic stream flow, and
allow early completion only for visible non-empty text or open tool arguments
that parse to a non-null, non-array JSON object; update
src/adapters/anthropic.ts lines 992-1010 accordingly, while preserving
already-emitted tool_call_end state. In src/adapters/anthropic.ts lines 892-894,
set sawVisibleText only when the text delta is non-empty. Add coverage in
tests/anthropic-eof-tolerance.test.ts lines 57-104 for pre-EOF tool closure,
missing or non-object tool JSON, and empty text deltas.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 62830ced-ce24-4509-9437-6fda3989d238
📒 Files selected for processing (9)
docs-site/src/content/docs/ja/reference/configuration/providers.mddocs-site/src/content/docs/ko/reference/configuration/providers.mddocs-site/src/content/docs/reference/configuration/providers.mddocs-site/src/content/docs/ru/reference/configuration/providers.mddocs-site/src/content/docs/zh-cn/reference/configuration/providers.mdsrc/adapters/anthropic.tssrc/providers/free-directory.tssrc/types.tstests/anthropic-eof-tolerance.test.ts
| | `thinkingBudgetModels?` | `string[]` | Chat models using integer `thinking_budget`; effort maps to a budget fraction. | | ||
| | `noVisionModels?` | `string[]` | Text-only models sent through the vision sidecar; matching tolerates an Ollama `:size` tag. | | ||
| | `escapeBuiltinToolNames?` | `boolean` | Escape built-in tool names for Anthropic-compatible gateways and restore them in returned calls. | | ||
| | `anthropicEofTolerance?` | `boolean` | Let an Anthropic-compatible gateway complete a stream that ends before `message_stop`, only when visible text or a complete JSON-object tool input was received. Off by default. | |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 5 'Anthropic|Messages API|message_stop|content_block_stop|SSE|EOF|tool JSON' \
docs-site/src/content/docs/reference/adapters.mdRepository: lidge-jun/opencodex
Length of output: 3674
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- adapter documentation ---'
cat -n docs-site/src/content/docs/reference/adapters.md | sed -n '54,72p'
printf '%s\n' '--- provider reference ---'
cat -n docs-site/src/content/docs/reference/configuration/providers.md | sed -n '86,100p'
printf '%s\n' '--- anthropicEofTolerance references ---'
rg -n -C 4 'anthropicEofTolerance|message_stop|content_block_stop|input_json_delta' \
--glob '!node_modules' --glob '!dist' --glob '!build' .
printf '%s\n' '--- candidate Anthropic adapter files ---'
fd -i 'anthropic' . --type f | head -80Repository: lidge-jun/opencodex
Length of output: 50378
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- Anthropic parser event cases ---'
sed -n '850,1020p' src/adapters/anthropic.ts
printf '%s\n' '--- adapter documentation context ---'
cat -n docs-site/src/content/docs/reference/adapters.md | sed -n '54,70p'
printf '%s\n' '--- required documentation terms ---'
python3 - <<'PY'
from pathlib import Path
doc = Path("docs-site/src/content/docs/reference/adapters.md").read_text()
terms = [
"Messages API",
"message_start",
"content_block_start",
"content_block_delta",
"content_block_stop",
"message_delta",
"message_stop",
"error",
"anthropicEofTolerance",
"strict",
"truncation",
"complete JSON",
]
for term in terms:
print(f"{term}: {'present' if term in doc else 'missing'}")
PYRepository: lidge-jun/opencodex
Length of output: 11037
Document Anthropic Messages streaming and EOF behavior.
docs-site/src/content/docs/reference/adapters.md:54-66 omits the supported SSE lifecycle events and anthropicEofTolerance behavior. Document message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop, and error. State that strict EOF handling fails on truncation, while enabled tolerance allows early completion only after visible text or complete JSON-object tool input; incomplete or contentless streams remain errors.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs-site/src/content/docs/reference/configuration/providers.md` at line 94,
Update the Anthropic Messages streaming documentation in the adapters reference
to list the supported SSE lifecycle events: message_start, content_block_start,
content_block_delta, content_block_stop, message_delta, message_stop, and error.
Document that strict EOF handling fails truncated streams, while
anthropicEofTolerance permits early completion only after visible text or
complete JSON-object tool input; incomplete or contentless streams must still
fail.
Source: Path instructions
| } else if (provider.anthropicEofTolerance === true) { | ||
| // AgentRouter-style compatibility profile (#658): the upstream can close the stream | ||
| // after valid content without terminal frames. Complete only when visible text was | ||
| // received or an open tool call has complete JSON-object arguments; everything else | ||
| // (incomplete tool JSON, no usable content, transport failure) stays a truncation | ||
| // error, matching the strict default. | ||
| if (currentToolCallId) { | ||
| if (streamedToolArgumentsParse(currentToolCallJson)) { | ||
| budget.closeCall(currentToolCallId); | ||
| currentToolCallId = ""; | ||
| yield { type: "tool_call_end" }; | ||
| yield* emitDone(); | ||
| } else { | ||
| yield { type: "error", message: "upstream stream ended before message_stop — possible truncation" }; | ||
| } | ||
| } else if (sawVisibleText) { | ||
| yield* emitDone(); | ||
| } else { | ||
| yield { type: "error", message: "upstream stream ended before message_stop — possible truncation" }; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Implement the documented EOF completion predicate.
A valid tool call that receives content_block_stop clears currentToolCallId before Line 1007. The EOF path then reports truncation even though it already emitted tool_call_end. Conversely, streamedToolArgumentsParse("") returns true and also accepts arrays, scalars, and null, so an open tool call without a complete JSON object can incorrectly complete. An empty text_delta also sets sawVisibleText at Line 893.
src/adapters/anthropic.ts#L992-L1010: Track completed tool calls separately. At EOF, accept an open tool call only if its trimmed arguments parse to a non-null, non-array JSON object.src/adapters/anthropic.ts#L892-L894: Set the visible-text state only for non-empty text.tests/anthropic-eof-tolerance.test.ts#L57-L104: Add cases for a tool call closed before EOF, missing or non-object tool JSON, and an empty text delta.
Based on PR objectives, only visible text or a complete JSON object may permit early completion.
📍 Affects 2 files
src/adapters/anthropic.ts#L992-L1010(this comment)src/adapters/anthropic.ts#L892-L894tests/anthropic-eof-tolerance.test.ts#L57-L104
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/adapters/anthropic.ts` around lines 992 - 1010, Track completed tool
calls separately from currentToolCallId in the EOF handling around the Anthropic
stream flow, and allow early completion only for visible non-empty text or open
tool arguments that parse to a non-null, non-array JSON object; update
src/adapters/anthropic.ts lines 992-1010 accordingly, while preserving
already-emitted tool_call_end state. In src/adapters/anthropic.ts lines 892-894,
set sawVisibleText only when the text delta is non-empty. Add coverage in
tests/anthropic-eof-tolerance.test.ts lines 57-104 for pre-EOF tool closure,
missing or non-object tool JSON, and empty text deltas.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c7494f64c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| bytez: openAi("https://api.bytez.com/models/v2/openai/v1", "https://bytez.com", { verification: "unverified", lastVerified: undefined, documentationUrl: "https://docs.bytez.com/", discovery: "static", liveModels: false, models: ["meta-llama/Llama-3.3-70B-Instruct", "mistralai/Mistral-7B-Instruct-v0.3", "Qwen/Qwen2.5-72B-Instruct"], note: "The recurring-credit classification is retained from the requested catalog, but the current reset terms could not be independently verified." }), | ||
| "nous-research": openAi("https://inference-api.nousresearch.com/v1", "https://portal.nousresearch.com", { discovery: "static", liveModels: false, models: ["Hermes-4-405B", "Hermes-4-70B"] }), | ||
| agentrouter: { baseUrl: "https://agentrouter.org", dashboardUrl: "https://agentrouter.org", adapter: "anthropic", authKind: "key", supportLevel: "experimental", verification: "primary", modelsUrl: "https://agentrouter.org/v1/models", lastVerified: LAST_VERIFIED, discovery: "live", liveModels: true }, | ||
| agentrouter: { baseUrl: "https://agentrouter.org", dashboardUrl: "https://agentrouter.org", adapter: "anthropic", authKind: "key", supportLevel: "experimental", verification: "primary", modelsUrl: "https://agentrouter.org/v1/models", lastVerified: LAST_VERIFIED, discovery: "live", liveModels: true, anthropicEofTolerance: true }, |
There was a problem hiding this comment.
For existing AgentRouter configurations that do not manually add anthropicEofTolerance, this metadata never enables the new behavior: a repo-wide search shows FREE_PROVIDER_DIRECTORY is consumed only by tests, and tests/provider-registry-parity.test.ts:814-830 explicitly keeps it isolated from runtime providers. Because routedProviderConfig() reads the canonical registry and user config instead, AgentRouter streams still use the strict default and report truncation. Add the capability to the canonical provider/derivation flow or otherwise persist it into the actual provider configuration.
AGENTS.md reference: src/AGENTS.md:L18-L18
Useful? React with 👍 / 👎.
| } else if (sawVisibleText) { | ||
| yield* emitDone(); |
There was a problem hiding this comment.
When a tool-only AgentRouter stream sends valid arguments and content_block_stop but then closes before message_delta/message_stop, the normal block-stop handling has already cleared currentToolCallId; this branch therefore falls through to the truncation error because no text was emitted. This is a cleaner and more complete tool call than the open-tool case accepted above, yet the advertised tolerance still rejects it. Track that a usable tool call completed and allow that state to reach emitDone().
AGENTS.md reference: src/AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
| if (streamedToolArgumentsParse(currentToolCallJson)) { | ||
| budget.closeCall(currentToolCallId); | ||
| currentToolCallId = ""; | ||
| yield { type: "tool_call_end" }; | ||
| yield* emitDone(); |
There was a problem hiding this comment.
When a tolerant stream exceeds the translator budget after opening a tool call, the catch at lines 966-974 emits translation_buffer_limit but execution continues here; if the overflow happened before the fragment was retained, the empty argument buffer is considered valid and the adapter subsequently emits tool_call_end and done. Direct or unguarded consumers therefore receive a successful terminal after a terminal error, potentially completing an empty tool call. Record the failure or return immediately after emitting the budget error so EOF tolerance cannot run.
AGENTS.md reference: src/AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
| const delta = data.delta as Record<string, unknown> | undefined; | ||
| if (!delta) break; | ||
| if (delta.type === "text_delta" && typeof delta.text === "string") { | ||
| sawVisibleText = true; |
There was a problem hiding this comment.
If a tolerant gateway emits text_delta with text: "" and then closes, this assignment marks the response usable and the EOF branch returns done with an empty answer. That contradicts the stated requirement that EOF without usable content remain a truncation error. Set sawVisibleText only when the delta actually contains visible content.
AGENTS.md reference: src/AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
| } else if (sawVisibleText) { | ||
| yield* emitDone(); |
There was a problem hiding this comment.
When a tolerant stream emits valid text and its next SSE record is truncated or contains malformed JSON, the parse failure at lines 858-863 is only debug-dropped, so this branch treats the ensuing EOF as clean and emits done; for example, a cut-off second text_delta silently loses that text while completing the response. Track whether any malformed record was dropped and keep EOF strict in that state rather than accepting a demonstrably damaged stream.
AGENTS.md reference: src/AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
| for (let i = opens.length - 1; i >= 0 && tried < maxCandidates; i--, tried++) { | ||
| const candidate = input.slice(opens[i]); | ||
| try { | ||
| const parsed = JSON.parse(candidate) as unknown; | ||
| if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) return candidate; |
There was a problem hiding this comment.
When the final concatenated object is itself truncated but ends with a complete nested object, scanning every opening brace accepts that nested value as the repaired tool arguments. For example, {}{"outer":{"value":42} is converted to {"value":42} and emitted as a completed tool call even though the second top-level object never closed. Parse top-level object boundaries with string/escape awareness so nested fragments from incomplete payloads cannot become executable arguments.
AGENTS.md reference: src/AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
| for (let i = opens.length - 1; i >= 0 && tried < maxCandidates; i--, tried++) { | ||
| const candidate = input.slice(opens[i]); |
There was a problem hiding this comment.
When a tolerant non-stream response contains a large malformed tool-input string with many braces, each of up to 32 suffix candidates is a near-full copy that JSON.parse scans synchronously, followed by up to 32 more prefix parses; the initial loop also retains every brace offset. A 4 MiB all-{ input already spends roughly half a second in this path, and inputs near the 32 MiB translator limit can block Bun's event loop for several seconds per request. Cap the repairable input size or replace the repeated suffix parsing with a single bounded top-level scan.
Useful? React with 👍 / 👎.
| // (incomplete tool JSON, no usable content, transport failure) stays a truncation | ||
| // error, matching the strict default. | ||
| if (currentToolCallId) { | ||
| if (streamedToolArgumentsParse(currentToolCallJson)) { |
There was a problem hiding this comment.
When a tolerant tool stream ends with a syntactically valid scalar or array such as null, 42, or [], streamedToolArgumentsParse() returns true and this branch emits tool_call_end followed by done. The capability contract and documentation require a complete JSON object, so these values should remain truncation/protocol errors instead of becoming completed function calls. Validate that the parsed result is non-null, object-shaped, and not an array before accepting EOF.
AGENTS.md reference: src/AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
| | `thinkingBudgetModels?` | `string[]` | Chat models using integer `thinking_budget`; effort maps to a budget fraction. | | ||
| | `noVisionModels?` | `string[]` | Text-only models sent through the vision sidecar; matching tolerates an Ollama `:size` tag. | | ||
| | `escapeBuiltinToolNames?` | `boolean` | Escape built-in tool names for Anthropic-compatible gateways and restore them in returned calls. | | ||
| | `anthropicEofTolerance?` | `boolean` | Let an Anthropic-compatible gateway complete a stream that ends before `message_stop`, only when visible text or a complete JSON-object tool input was received. Off by default. | |
There was a problem hiding this comment.
When an operator enables anthropicEofTolerance, parseResponse() also changes non-stream behavior by extracting a JSON object from malformed string-valued tool input, but this row—and the matching translated rows—describes only early stream completion. Users therefore cannot tell that the option can alter and complete tool calls on ordinary non-stream requests. Document the repair behavior in every locale or separate it behind a configuration key whose scope matches the documented EOF behavior.
AGENTS.md reference: docs-site/AGENTS.md:L7-L10
Useful? React with 👍 / 👎.
|
Landed in #892 as 538d755 (authorship preserved), with three review folds on top: the repair scan is now backward/lastIndexOf with no offset arrays plus an exact UTF-8 byte cap (surrogate-safe), only non-empty text authorizes tolerant completion, and a budget overflow returns as the single terminal event (9794e24, d8b707e, daf7069). One honest note: FREE_PROVIDER_DIRECTORY is currently consumed by tests only, so the AgentRouter row declaration is inert metadata — the capability activates via the provider config flag, matching your explicit-capability design. Closing as superseded. |
2 participants
Summary
Closes #658: AgentRouter's Anthropic-compatible endpoint can close the stream before
content_block_stop,message_delta, andmessage_stop. The Anthropic adapter previously treated every such EOF as a fatal truncation and discarded otherwise valid text and complete tool-call arguments.This PR implements the accepted provider-compatibility feature as an explicit capability (
anthropicEofTolerance, default off) rather than a hard-coded hostname branch:{}{"value":42}is repaired to the last valid JSON object ({"value":42}) instead of being discarded to{}. The strict path is unchanged.tool_use.idkeeps the existing stable synthesized-id path (anthropic adapter: baseUrl ending in /v1/messages doubles the path, input_json_delta is not scoped to tool_use, and Anthropic-compatible relays need tool-call quirk tolerance #765), now pinned by a regression test for this provider.src/providers/free-directory.tsdeclares the capability.Repro (wire shape)
Before:
upstream stream ended before message_stop — possible truncation, visible text lost. After (withanthropicEofTolerance: true): text preserved and the stream completes withdone.Tests
tests/anthropic-eof-tolerance.test.ts(8 cases): text EOF on/off, complete/incomplete tool EOF, no-content EOF, synthesized tool id, non-stream concatenation on/off, directory row capability.bun run typecheckpasses.anthropic-compatible-stream.test.ts,provider-registry-parity.test.ts,zhipu-bigmodel-provider.test.ts) pass.devbaseline in the same environment: baseline has pre-existing failing tests (GUI/network-dependent, unrelated to this change); this branch introduces no additional failures, and the 4 flaky WP040 probe tests that were branch-only pass in isolation.Docs
providers.mdconfiguration table row added in English plus ja/ko/ru/zh-cn locales.Summary by CodeRabbit