Skip to content

fix(core): read a Codex other web_search action as a page read - #286

Draft
czenko wants to merge 1 commit into
mainfrom
core/codex-page-read-detection
Draft

fix(core): read a Codex other web_search action as a page read#286
czenko wants to merge 1 commit into
mainfrom
core/codex-page-read-detection

Conversation

@czenko

@czenko czenko commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Problem

Every docs eval carries a check that the referenced page was retrieved with content. It reds on runs that plainly read the page, and docs evals run only on codex-gpt-5.6-luna-no-skills, so the check is close to unpassable on the one experiment that matters.

Codex CLI 0.138 round-trips openPage and findInPage through a snake_case enum that has no such variants, so both page-reading actions arrive as other. The action carries no url, only the query does, and the url-shape fallback leaves hasContent unset. Three runs of build-docs-007-edge-function-secrets recorded exactly this:

{ "query": "https://supabase.com/docs/guides/functions/secrets.md",
  "action": { "type": "other" } }

Solution

  • Treat an explicit other action with a url-shaped Supabase query as a page read. search survives the round trip intact and is matched earlier, so reaching that branch with other is positive evidence of a page read rather than an absence of evidence.
  • Attribute the page to the query, since the url does not survive on the action.
  • Scoped to an explicit other. An action type we do not know still falls through to the url-shape fallback and stays unknown.

Codex reports no result for these calls, the same way web_fetch carries no body here. hasContent says the channel delivers page text to the model, not that the harness captured it.

Manual testing

  1. cd packages/core && npx vitest run. 134 pass, including three new cases: the exact recorded body, an other pointing off supabase.com, and an other whose query is a search term rather than a url.
  2. Replay the tool calls recorded on the three runs through buildDocsResult. Each resolves two calls reaching the guide, both with content, so the check flips from fail to pass on all three. The bodies come from the run's raw-results artifact, which keeps toolCalls even though published results drop them.
  3. pnpm typecheck. Passes.

Every docs eval carries a check that the referenced page was retrieved with
content, and it red on runs that had plainly read the page. Docs evals run only
on Codex, so the check was close to unpassable.

CLI 0.138 round-trips `openPage` and `findInPage` through a snake_case enum
with no such variants, so both arrive as `other`. `search` survives the trip
intact and is matched earlier, which makes an explicit `other` positive
evidence of a page read rather than an absence of evidence. The url does not
survive on the action, only in the query, so the page is attributed to that.

Codex reports no result for these calls, the same way web_fetch carries no body
here. hasContent says the channel delivers page text to the model, not that the
harness captured it.

Scoped to an explicit `other`. An action type we do not know still falls
through to the url-shape fallback and stays unknown.

Verified by replaying the tool calls recorded on three runs of
build-docs-007-edge-function-secrets, which now resolve as read.
@vercel

vercel Bot commented Sep 11, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
evals Ignored Ignored Sep 11, 2026 10:57pm UTC

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant