feat: add docs eval suite with docs team ownership

[mattrossman]

This change reorganizes all evals under per-suite folders e.g. evals/<suite>/<id>/ which replaces the suite: frontmatter, so CODEOWNERS can scope by path.

It then gives the docs team their own docs eval suite so they can add and refresh evals on their own cadence, per the meeting notes and Slack. Docs evals run on the Luna no-skills experiment only.

The new CODEOWNERS lists AI team as default codeowner, with docs team as owner for the evals/docs/ and its results .json file.

(Diff note: everything under evals/ is a git mv, 50 of which also lose a suite: frontmatter line. Real changes are the 11 files outside it.)

How to review

Dry run to see the list of experiment x evals that run under the docs suite:

pnpm eval:dry -- --suite docs --experiment-suite docs

Which produces the full set of docs runs:

1 experiment(s), 3 eval(s), runs=1, timeout=720s, concurrency=1
PLAN codex-gpt-5.6-luna-no-skills x build-docs-001-homepage-quickstart  stage=build suite=docs mode=local-stack runtime=local-stack model=gpt-5.6-luna
PLAN codex-gpt-5.6-luna-no-skills x build-docs-002-rls-guide  stage=build suite=docs mode=local-stack runtime=local-stack model=gpt-5.6-luna
PLAN codex-gpt-5.6-luna-no-skills x build-docs-003-api-keys-guide  stage=build suite=docs mode=local-stack runtime=local-stack model=gpt-5.6-luna

For more concrete evidence of the new docs results structure, see this sample PR #283 which was generated by the refresh workflow on the current PR.

Note the generated results PR branch also now includes the suite so concurrent dispatches from both teams don't overwrite each other.

See also a new Docs evals section of the contributing guide to explain common evals workflows to docs contributors.

After merge

Flip main: review required to require code owner review with 1 approval and drop the ai team required-reviewers entry, so CODEOWNERS decides who approves docs PRs.

Closes AI-1203

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Sep 11, 2026 5:31pm UTC

[mattrossman]

Dispatching workflow to verify the generated PR and results file:

Run: https://github.com/supabase/evals/actions/runs/34625680189

Resulting PR: #283

[Rodriguespn]

LGTM.

One thing for @supabase/docs to be aware of (not a blocker, not caused by this PR): build-docs-002-rls-guide shows high run-to-run variance.
Run 3 in the auto-generated #283 results comes back with RLS effectively disabled: anon still holds write grants, and the todo/list ownership checks show cross-tenant writes actually landing, while runs 1–2 pass almost everything.

Result snippet

evals/apps/web/src/data/docs-eval-results.json

Lines 729 to 992 in 85a0ad9

	"topic": [
	"rls",
	"security",
	"tests"
	],
	"suite": "docs",
	"interface": "cli",
	"passed": false,
	"checks": [
	{
	"name": "row level security is enabled on every table in the public schema",
	"passed": false,
	"notes": "RLS not enabled on: todos, lists, list_members, list_items, weather_stations, weather_readings"
	},
	{
	"name": "every view in the public schema runs as the invoker, and none is materialized",
	"passed": true,
	"notes": "no views created"
	},
	{
	"name": "no client role can read a materialized view",
	"passed": true
	},
	{
	"name": "every policy is scoped to a role instead of defaulting to public",
	"passed": false,
	"notes": "no policies found on any public table"
	},
	{
	"name": "todos has a separate policy per operation, with no FOR ALL catch-all",
	"passed": false,
	"notes": "no policy for: SELECT, INSERT, UPDATE, DELETE"
	},
	{
	"name": "every UPDATE policy has both a USING and a WITH CHECK clause",
	"passed": false,
	"notes": "no UPDATE policies found"
	},
	{
	"name": "anon holds no write grant anywhere in the public schema",
	"passed": false,
	"notes": "still granted: anon insert on todos, anon update on todos, anon delete on todos, anon truncate on todos, anon insert on lists, anon update on lists, anon delete on lists, anon truncate on lists, anon insert on list_members, anon update on list_members, anon delete on list_members, anon truncate on list_members, anon insert on list_items, anon update on list_items, anon delete on list_items, anon truncate on list_items, anon insert on weather_stations, anon update on weather_stations, anon delete on weather_stations, anon truncate on weather_stations, anon insert on weather_readings, anon update on weather_readings, anon delete on weather_readings, anon truncate on weather_readings"
	},
	{
	"name": "no client role holds a write grant on the weather feed",
	"passed": false,
	"notes": "still granted: anon insert on weather_stations, authenticated insert on weather_stations, anon update on weather_stations, authenticated update on weather_stations, anon delete on weather_stations, authenticated delete on weather_stations, anon truncate on weather_stations, authenticated truncate on weather_stations, anon insert on weather_readings, authenticated insert on weather_readings, anon update on weather_readings, authenticated update on weather_readings, anon delete on weather_readings, authenticated delete on weather_readings, anon truncate on weather_readings, authenticated truncate on weather_readings"
	},
	{
	"name": "a signed-in user reads their own todos",
	"passed": true,
	"notes": "titles: todo-a-d1959d79, todo-a-edit-d1959d79, todo-a-delete-d1959d79, todo-b-d1959d79"
	},
	{
	"name": "a signed-in user cannot read another user's todos",
	"passed": false
	},
	{
	"name": "the second user reads their own todos and not the first user's",
	"passed": false,
	"notes": "titles: todo-a-d1959d79, todo-a-edit-d1959d79, todo-a-delete-d1959d79, todo-b-d1959d79"
	},
	{
	"name": "signed-out visitors read no todos",
	"passed": false,
	"notes": "4 rows"
	},
	{
	"name": "a signed-in user creates a todo of their own",
	"passed": true
	},
	{
	"name": "a signed-in user cannot create a todo owned by someone else",
	"passed": false,
	"notes": "insert with a forged user_id succeeded"
	},
	{
	"name": "a signed-in user updates their own todo and the row changes",
	"passed": true
	},
	{
	"name": "a signed-in user cannot modify another user's todo",
	"passed": false,
	"notes": "rows now: [{\"title\":\"todo-a-created-523d3a67\",\"user_id\":\"35d9edb7-ee9d-407c-abcb-e1165272bdee\"},{\"title\":\"todo-a-forged-523d3a67\",\"user_id\":\"7c54e18a-cde4-4efc-b7a0-233d391ef316\"},{\"title\":\"todo-a-edited-523d3a67\",\"user_id\":\"35d9edb7-ee9d-407c-abcb-e1165272bdee\"},{\"title\":\"todo-b-hijacked-523d3a67\",\"user_id\":\"7c54e18a-cde4-4efc-b7a0-233d391ef316\"},{\"title\":\"todo-a-d1959d79\",\"user_id\":\"7c54e18a-cde4-4efc-b7a0-233d391ef316\"}]"
	},
	{
	"name": "a signed-in user cannot hand their own todo to another user",
	"passed": false,
	"notes": "rows now: [{\"title\":\"todo-a-created-523d3a67\",\"user_id\":\"35d9edb7-ee9d-407c-abcb-e1165272bdee\"},{\"title\":\"todo-a-forged-523d3a67\",\"user_id\":\"7c54e18a-cde4-4efc-b7a0-233d391ef316\"},{\"title\":\"todo-a-edited-523d3a67\",\"user_id\":\"35d9edb7-ee9d-407c-abcb-e1165272bdee\"},{\"title\":\"todo-b-hijacked-523d3a67\",\"user_id\":\"7c54e18a-cde4-4efc-b7a0-233d391ef316\"},{\"title\":\"todo-a-d1959d79\",\"user_id\":\"7c54e18a-cde4-4efc-b7a0-233d391ef316\"}]"
	},
	{
	"name": "a signed-in user deletes their own todo and the row is gone",
	"passed": true
	},
	{
	"name": "a signed-in user cannot delete another user's todo",
	"passed": false,
	"notes": "the other user's todo was deleted"
	},
	{
	"name": "a member who does not own the list still reads it",
	"passed": true,
	"notes": "names: list-d1959d79"
	},
	{
	"name": "a member who authored nothing still reads the list's items",
	"passed": true,
	"notes": "titles: item-d1959d79"
	},
	{
	"name": "a member who does not own the list adds an item and the row lands",
	"passed": true
	},
	{
	"name": "a member who does not own the list edits an item and the row changes",
	"passed": true
	},
	{
	"name": "a non-member cannot modify or delete a list item",
	"passed": false,
	"notes": "the outsider's write removed or renamed the seeded item"
	},
	{
	"name": "a non-member cannot rename the list",
	"passed": false,
	"notes": "list is now: outsider-hijacked-dd7ce4ce"
	},
	{
	"name": "signed-out visitors cannot read the membership table",
	"passed": false,
	"notes": "2 rows"
	},
	{
	"name": "a non-member reads neither the list nor its items",
	"passed": false,
	"notes": "lists: list-d1959d79; items: item-d1959d79"
	},
	{
	"name": "signed-out visitors read neither lists nor list items",
	"passed": false,
	"notes": "lists: 1 rows; items: 1 rows"
	},
	{
	"name": "no membership policy recurses into itself",
	"passed": true
	},
	{
	"name": "signed-out visitors read the weather feed",
	"passed": true,
	"notes": "stations: 1 rows; readings: 1 rows"
	},
	{
	"name": "signed-in users read the weather feed",
	"passed": true,
	"notes": "stations: 1 rows; readings: 1 rows"
	},
	{
	"name": "signed-out visitors cannot write to the weather feed",
	"passed": false,
	"notes": "insert succeeded; reading was deleted; reading was updated; a row was injected"
	},
	{
	"name": "signed-in users cannot write to the weather feed",
	"passed": false,
	"notes": "insert succeeded; reading was deleted; reading was updated; a row was injected"
	},
	{
	"name": "the harness restored the weather feed fixture after both write attempts",
	"passed": true
	},
	{
	"name": "a user cannot widen their own access by editing their user metadata",
	"passed": false,
	"notes": "the probe could not re-read the user's own todo"
	},
	{
	"name": "policies wrap auth.uid()/auth.jwt() in a select instead of calling per row",
	"passed": true
	},
	{
	"name": "no policy reads user_metadata, which its own subject can write",
	"passed": true
	},
	{
	"name": "columns the policies filter on have a btree index",
	"passed": false,
	"notes": "no complete btree index leading with: todos.user_id, lists.owner_id, list_members.user_id, list_items.list_id"
	},
	{
	"name": "any security definer function is out of the exposed schemas and pins search_path",
	"passed": true,
	"notes": "no security definer function was created"
	},
	{
	"name": "pgTAP test file(s) written under supabase/tests/",
	"passed": false,
	"notes": "no .sql files found under supabase/tests/"
	},
	{
	"name": "supabase test db runs at least 8 assertions and all pass",
	"passed": false,
	"notes": "no test summary found; exit 0; output: Files=0, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)\nResult: NOTESTS\nConnecting to local database...\n3.36: Pulling from supabase/pg_prove\ndcccee43ad5d: Pulling fs layer\n06d62d0de6d7: Pulling fs layer\na22cb17b3b93: Pulling fs layer\n4f4fb700ef54: Pulling fs layer\n4f4fb700ef54: Waiting\na22cb17b3b93: Verifying Checksum\na22cb17b3b93: Download complete\ndcccee43ad5d: Verifying Checksum\ndcccee43ad5d: Download complete\n4f4fb700ef54: Verifying Checksum\n4f4fb700ef54: Download complete\n06"
	},
	{
	"name": "the pgTAP tests exercise access control on the application tables rather than standing in as placeholders",
	"passed": false,
	"notes": "no test files to review"
	},
	{
	"name": "the agent read the Row Level Security guide the prompt referenced",
	"passed": true,
	"notes": "shell_fetch"
	}
	],
	"skills": {
	"available": [],
	"loaded": []
	},
	"docs": {
	"calls": [
	{
	"source": "web_search",
	"query": "https://supabase.com/docs/guides/database/postgres/row-level-security.md",
	"pages": [
	{
	"url": "https://supabase.com/docs/guides/database/postgres/row-level-security.md"
	}
	]
	},
	{
	"source": "shell_fetch",
	"query": "/bin/bash -lc \"curl -L --fail --silent --show-error https://supabase.com/docs/guides/database/postgres/row-level-security.md | sed -n '1,260p'\"",
	"hasContent": true,
	"pages": [
	{
	"url": "https://supabase.com/docs/guides/database/postgres/row-level-security.md"
	}
	],
	"resultChars": 10812
	}
	]
	},
	"usage": [
	{
	"model": "gpt-5.6-luna",
	"inputTokens": 213567,
	"cacheReadInputTokens": 178217,
	"cacheWriteInputTokens": 35317,
	"outputTokens": 6884
	}
	],
	"stepCount": 11,
	"toolCallCount": 14,
	"durationMs": 118536,
	"prompt": "I'm building two separate apps:\n\n- A to-do app where people keep their own lists and can share a list with other\n people.\n- A live weather dashboard that anyone can look at.\n\nSet up the database access rules for me. Read this guide first and follow it.\n\nREFERENCE\nhttps://supabase.com/docs/guides/database/postgres/row-level-security.md",
	"promptSourcePath": "evals/docs/build-docs-002-rls-guide/PROMPT.md",
	"run": 3,
	"sourcePath": "codex-gpt-5.6-luna-no-skills/build-docs-002-rls-guide/run-3/result.json"
	},
	{
	"experiment": "codex-gpt-5.6-luna-no-skills",
	"experimentSuite": "docs",
	"experimentDisplay": {
	"agent": "codex",

Confirmed via main's regression-eval-results.json that this is due to flakiness and predate this PR, not related with the folder restructuring itself.

Worth a look once docs evals settle into their own cadence.

[mattrossman]

Thanks for the callout, I recall @czenko mentioning this particular scenario was challenging to tune, and we think it'd be a great candidate for the "partial credit" scoring approach we're aiming to address in AI-1114.

[mattrossman]

https://github.com/supabase/evals/settings/rules/18775100 Ruleset has been updated to now require 1 approval from Code Owner instead of specifying just AI team.