fix(core): read a Codex other web_search action as a page read - #286
Draft
czenko wants to merge 1 commit into
Draft
fix(core): read a Codex other web_search action as a page read#286czenko wants to merge 1 commit into
other web_search action as a page read#286czenko wants to merge 1 commit into
Conversation
Every docs eval carries a check that the referenced page was retrieved with content, and it red on runs that had plainly read the page. Docs evals run only on Codex, so the check was close to unpassable. CLI 0.138 round-trips `openPage` and `findInPage` through a snake_case enum with no such variants, so both arrive as `other`. `search` survives the trip intact and is matched earlier, which makes an explicit `other` positive evidence of a page read rather than an absence of evidence. The url does not survive on the action, only in the query, so the page is attributed to that. Codex reports no result for these calls, the same way web_fetch carries no body here. hasContent says the channel delivers page text to the model, not that the harness captured it. Scoped to an explicit `other`. An action type we do not know still falls through to the url-shape fallback and stays unknown. Verified by replaying the tool calls recorded on three runs of build-docs-007-edge-function-secrets, which now resolve as read.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every docs eval carries a check that the referenced page was retrieved with content. It reds on runs that plainly read the page, and docs evals run only on
codex-gpt-5.6-luna-no-skills, so the check is close to unpassable on the one experiment that matters.Codex CLI 0.138 round-trips
openPageandfindInPagethrough a snake_case enum that has no such variants, so both page-reading actions arrive asother. The action carries no url, only the query does, and the url-shape fallback leaveshasContentunset. Three runs ofbuild-docs-007-edge-function-secretsrecorded exactly this:{ "query": "https://supabase.com/docs/guides/functions/secrets.md", "action": { "type": "other" } }Solution
otheraction with a url-shaped Supabase query as a page read.searchsurvives the round trip intact and is matched earlier, so reaching that branch withotheris positive evidence of a page read rather than an absence of evidence.other. An action type we do not know still falls through to the url-shape fallback and stays unknown.Codex reports no result for these calls, the same way
web_fetchcarries no body here.hasContentsays the channel delivers page text to the model, not that the harness captured it.Manual testing
cd packages/core && npx vitest run. 134 pass, including three new cases: the exact recorded body, anotherpointing off supabase.com, and anotherwhose query is a search term rather than a url.buildDocsResult. Each resolves two calls reaching the guide, both with content, so the check flips from fail to pass on all three. The bodies come from the run'sraw-resultsartifact, which keepstoolCallseven though published results drop them.pnpm typecheck. Passes.