Skip to content

fix(core): make web search limits non-fatal and report real errors - #50938

Open
rekram1-node wants to merge 2 commits into
v2from
websearch-limits
Open

rekram1-node wants to merge 2 commits into
v2from
websearch-limits

Conversation

@rekram1-node

@rekram1-node rekram1-node commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Type of change

  • Bug fix

What does this PR do?

A websearch call failed with only Unable to search the web for <query>. Firecrawl had returned 450 KB, 424 KB of it one Hacker News thread, and WebSearchMcp.call fails any response over 256 KB. The tool then replaced that error, like every error without an HTTP status, with the generic message.

Limits no longer fail a search. All six providers (Exa, Firecrawl, Parallel, Tavily, TinyFish, and OpenCode Web Search) now read through plugin/websearch/response.ts:

 WebSearchMcp.call / Tavily / OpenCode Web Search
-  filterStatusOk, read body (MCP: over 256 KB → fail; Tavily and OpenCode: no cap)
+  WebSearchResponse.execute
+    read at most 1 MiB, stop downloading past it
+    non-2xx → same HttpClientError, plus the provider's explanation (read for at most 1 s)
+  parse the body (as partial JSON when it was cut off)
+  decode each result on its own, drop the one that was cut
 WebSearch.query
+  cap each result's content at 4,000 characters

The per-result cap keeps one huge page from pushing the other results out of the 50 KiB tool output; 10 results at the cap still fit. Measured across providers, only Firecrawl (up to 424 KB) and some Exa results (up to 8 KB) exceed it. Parallel sends structuredContent after a text copy of the same JSON, so a cut response falls back to the text copy.

Errors say what happened. The web search errors now have messages, and the tool reports Unable to search the web for <query> (<provider>): <reason>. MCP isError results and JSON-RPC errors now fail instead of becoming "No search results found". Results from real providers:

Case Before After
Firecrawl response over the cap Unable to search the web for … 4 complete results
Firecrawl, free tier rate limit No search results found… … (firecrawl): You've hit Firecrawl's free MCP rate limit… Fix: Create an API key at https://www.firecrawl.dev/app/api-keys…
Exa, invalid key No search results found… … (exa): web_search_exa error (401): Invalid API key
Tavily, invalid key Web search authentication failed (HTTP 401) … (tavily): HTTP 401: Unauthorized: missing or invalid API key.
TinyFish, keyless quota used Web search authentication failed (HTTP 401) … (tinyfish): HTTP 401: Free daily Search quota used (50/50). Sign up for continued access: https://agent.tinyfish.ai/sign-up…
Timeout Unable to search the web for … … (exa): web_search_exa request timed out

Messages never use the HTTP client's own error text, because it includes the request URL and Exa passes its key in the query string. The 429 failover still reads the status and Retry-After from the same HttpClientError.

Web search tool failures no longer attach the underlying error. toSessionError shows an attached error's message in place of the tool's, which is why the generic text only appeared when that message was empty. Permission errors are still attached and unwrap as before.

How did you verify your code works?

  • Real sessions: the dev TUI with a private server from this branch (bun dev --standalone) and Claude Sonnet 5 calling websearch, pinned to each provider through project config:

    • Exa returned 8 results, 3 of them capped at 4,000 characters.
    • OpenCode Web Search returned 8 results.
    • TinyFish and Firecrawl, both out of free quota, stored the provider messages shown in the table.
  • Service-level runs against all five external providers without keys and with invalid keys, including Firecrawl's 1.1 MB response cut at 1 MiB.

  • New tests:

    • every external provider against a response over the cap, with the oversized result in the middle
    • the per-result cap, including a cut at an emoji
    • MCP isError and JSON-RPC errors
    • provider explanations on non-2xx for TinyFish and Tavily
    • OpenCode Web Search with an oversized response and a 503 { _tag, message } body
    • tool messages for a timeout, a connection failure, and disabled search

    The oversized Firecrawl test fails on v2 with firecrawl_search response exceeded 262144 bytes. The existing test that a rejected OpenCode response is closed promptly still passes.

  • packages/core: 5461 pass. The 8 failures are the same shell-tool and test-environment tests that fail on v2. sdk's "embedded client exposes plugin-backed web search" also fails identically on v2.

  • bun run check, Prettier, and lint:effect-patterns are clean for the changed files.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Oversized provider responses failed the whole search: WebSearchMcp.call
rejected bodies over 256 KB, and a single 424 KB Firecrawl result was
enough. The tool then replaced the cause with a generic message.

All providers now read at most 1 MiB, parse a cut-off body as partial
JSON, and keep the results that arrived complete. WebSearch.query caps
each result's content at 4,000 characters so one page cannot crowd out
the rest of the bounded tool output.

Web search errors now carry readable messages, including the provider's
own explanation for non-2xx responses, MCP isError results, and
JSON-RPC errors, which previously surfaced as no results.
OpenCode Web Search, the default search provider for Console-connected
users, made its own request with no size cap and discarded the body of
failed responses. It now uses the shared response handling, and OpenCode
API error bodies ({ _tag, message }) supply the failure explanation.

Reading a failed response for its explanation is bounded to one second
so a stalled error body still fails promptly.

Also formats the earlier changes with Prettier and tidies the tests.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant