test: add agent evals

[angeloashmore]

Resolves:

Description

Adds an eval suite that measures how well AI agents (like Claude Code) use the CLI.

Each eval gives an agent a task in a temp project. The agent runs the real CLI against a real repo, and the test checks the outcome. A few notes on the design:

• Evals are plain Vitest tests. They reuse the existing e2e fixtures and grade results with normal asserts, plus an LLM judge for criteria that need judgement (for example, "are these field types sensible?").
• Each eval runs several trials. We track pass rates per run instead of a single pass/fail, since agent behavior varies.
• Evals are measurements, not gates. Evals the agent fails today stay enabled and score 0% until the CLI improves; an eval is only skipped when the CLI cannot pass it (missing feature).
• Each eval gets its own throwaway repository, so evals run concurrently without interfering and agents are free to push. Global teardown sweeps repositories created by agents that the harness cannot name in advance (e.g. prismic init).
• Results append to a JSONL history tagged with the CLI commit, so runs can be compared across CLI versions with node --run evals:report (scope with EVAL_HISTORY).
• Evals are opt-in (node --run evals) and refuse to run outside a container or CI, because the agent runs with permissions bypassed.
• A full run is 32 evals × 3 trials: roughly $16 and 15 minutes.

Checklist

• A comprehensive Linear ticket, providing sufficient context and details to facilitate the review of the PR, is linked to the PR.
• If my changes require tests, I added them.
• If my changes affect backward compatibility, it has been discussed.
• If my changes require an update to the CONTRIBUTING.md guide, I updated it.

Preview

node --run evals runs the suite as normal Vitest tests; a custom reporter prints pass rates and cost at the end (excerpt):

 ✓ evals/edit-models-precisely.eval.ts > adds a field without disturbing existing fields
 ✗ evals/edit-models-precisely.eval.ts > renames a field without disturbing field order
 …

Pass rates:
  ✓ adds a rich text field to a type: 3/3 passed (100%), $0.30
  ✓ models a testimonial slice from a text description: 1/1 passed (100%), $0.13
  ✓ adds a field without disturbing existing fields: 1/1 passed (100%), $0.11
  ✗ renames a field without disturbing field order: 0/1 passed (0%), $0.24
  …

node --run evals:report digests the JSONL history: one block per eval with one line per run, so scores compare across CLI commits (excerpt):

adds a rich text field to a type
  Jul 21 03:25 @ f235c35   3/3 (100%)   $0.27   205k tok   7.0 turns   16s
  Jul 21 19:34 @ 2f3a7a0   3/3 (100%)   $0.30   235k tok   8.0 turns   19s

writes a slice component that follows Prismic conventions
  Jul 21 19:34 @ 2f3a7a0   0/1   (0%)   $0.38   703k tok   25.0 turns   87s
  Jul 21 19:41 @ 2f3a7a0   1/1 (100%)   $0.47   936k tok   22.0 turns   102s

…

run Jul 21 19:34 @ 2f3a7a0: 29/35 trials passed (83%) across 33 evals · $6.26

How to QA ¹

Inside a disposable container with .env.test.local available:

PRISMIC_ALLOW_EVALS=true node --run evals
node --run evals:report

🤖 Generated with Claude Code

---

Note

Medium Risk
Evals run a real agent with permission bypass against live Prismic accounts and bill Anthropic API usage; mitigated by opt-in PRISMIC_ALLOW_EVALS and isolated-repo design, but misuse outside a disposable environment is dangerous.

Overview
Introduces an agent eval suite (node --run evals) that measures how well an AI agent uses the Prismic CLI on realistic tasks in disposable temp projects.

The harness in evals/it.ts drives the Claude Agent SDK with bypassed permissions, a pinned prismicio/skills system prompt, a prismic CLI shim, and custom matchers (toHaveRun, toSatisfyJudge via Haiku). Trials repeat per eval (EVAL_TRIALS); failures are tracked as pass-rate signals, not CI gates.

Eval modules cover safety (docs before acting, no destructive ambiguity), repo setup (previews, tokens, webhooks, locales), routes, content modeling, precise edits, init/sync, and React slice wiring—some cases are it.skip or annotated when the CLI cannot pass yet.

Reporting: evals/reporter.ts appends per-trial rows (tokens, cost, CLI commit) to evals/results.jsonl; evals/report.ts summarizes history across runs. Git attributes treat results.jsonl as union-merge; .env.test.example documents ANTHROPIC_API_KEY.

^{Reviewed by Cursor Bugbot for commit bee3782. Bugbot is set up for automated code reviews on this repo. Configure here.}

Footnotes

1. Please use these labels when submitting a review:
   ❓ #ask: Ask a question.
   💡 #idea: Suggest an idea.
   ⚠️ #issue: Strongly suggest a change.
   🎉 #nice: Share a compliment. ↩

[lihbr]

Look good to me, I like what you did with Vite and using a custom reporter, the failing evals are also really insightful :)

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bee3782. Configure here.}