feat(evals): eval the Edge Function secrets guide in supabase.com/docs

[czenko]

Closes DOCS-1305

Problem

The Managing secrets guide is the only place that says where a local secret has to sit for the function runtime to load it. Neither the Supabase agent skill nor Supacademy covers the topic, so the page is the only carrier. Users keep landing on the wrong file: FDBKIN-11884 and CLI-818 both ask for the project-root .env and supabase/functions/.env to be told apart, and FDBKIN-12716 and FDBKIN-6831 are the recurring "I set it and the function cannot read it" shape.

Nothing in the repo tests an Edge Functions docs page as the subject under test. deploy-functions-001-edge-function-secrets owns the hosted side and does not touch local loading.

Solution

• Adds evals/docs/build-docs-007-edge-function-secrets. The seed is a Vite app that calls a provider from the browser with the key as a literal, and a TODO comment fixes the endpoint and the missing_api_key failure contract the probe needs.
• The prompt asks for the key to be sorted out and for the local stack to be running. It never names the mechanism. The stripped-word list is in README.md.
• projectRunning: false, so the agent owns the stack lifecycle. supabase start bakes supabase/functions/.env into the edge runtime container's environment at creation time, so a file written after the stack is up is invisible to it. Letting the agent start the stack is what makes the runtime claim measurable without the scorer restarting anything.
• Eleven checks. Four read the client, three read where the key landed, three probe the served endpoint, and one asserts the page was retrieved with content.
• The ignore-rules check uses git check-ignore rather than git ls-files. The harness strips .git when it copies the seed in, so the tracked-file form matches nothing and reads green whatever the project ignores. The same check in deploy-functions-001 is affected.

Manual testing

1. pnpm typecheck. Passes.
2. npx biome check evals/docs/build-docs-007-edge-function-secrets/. Clean.
3. pnpm eval:dry -- --suite docs --experiment-suite docs. Plans codex-gpt-5.6-luna-no-skills x build-docs-007-edge-function-secrets.
4. Score six local fixtures against the file-based checks. Predictions were written before running and all six matched. The one that decides the design puts the key in the project-root .env and fails the provider key sits where the local function runtime loads it, with the other five failing on the bundle, the function source, and the ignore rules respectively.

Baseline

Three runs on codex-gpt-5.6-luna-no-skills, all 7/11.

All three built a function reading OPENAI_API_KEY from the environment, started the stack, and answered {"error":"missing_api_key"}. Two wrote supabase/functions/.env.example and stopped rather than creating the file supabase start reads. The third wrote no env file. Placement, ignore rules, and the request-time probe agree on every run, which is the source-level claim and the behavioral one corroborating each other.

The fourth failure is instrumentation, not the page. the agent read the Managing secrets guide the prompt referenced reds on every run even though every run reached the page. Codex web_search never sets hasContent, because CLI 0.138 mangles open_page into the catch-all and the url-shape fallback in packages/core/src/docs-results.ts:309 leaves it unset. Every docs eval carries this check and docs evals run only on Codex, so it needs a fix that is not scoped to this PR.

An earlier baseline on this branch red two checks that hunted the seeded literal. Agents replace it, so those checks reported nothing placed on solutions that placed a credential correctly. Fixed in 6f38449, and a fixture reproducing it is part of the set.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Sep 11, 2026 11:12pm UTC