ci: run evals weekly and record results on main

[angeloashmore]

Resolves:

Description

Runs the eval suite in CI: weekly on Monday 06:00 UTC, and on demand via workflow dispatch. Results land on main through a bot-opened PR the job squash-merges immediately, since main's ruleset requires changes to arrive by PR. The results PR body carries the run's digest, so watching the repo's PRs doubles as a weekly report.

Eval failures do not fail the job; evals are measurements, not gates. The job fails only when no rows were recorded (the commit of an unchanged results.jsonl is the gate), meaning the harness died before grading. The eval step never receives GH_TOKEN because the agent under test can read the step's environment.

Requires one new repository secret: ANTHROPIC_API_KEY (already set).

Checklist

• A comprehensive Linear ticket, providing sufficient context and details to facilitate the review of the PR, is linked to the PR.
• If my changes require tests, I added them.
• If my changes affect backward compatibility, it has been discussed.
• If my changes require an update to the CONTRIBUTING.md guide, I updated it.

Preview

How to QA ¹

After this merges into #229's branch and reaches main: Actions → evals → Run workflow. The run takes roughly 15–25 minutes and about $22; confirm it opens and squash-merges a test(eval): record eval run at … PR whose body shows the digest.

🤖 Generated with Claude Code

Footnotes

1. Please use these labels when submitting a review:
   ❓ #ask: Ask a question.
   💡 #idea: Suggest an idea.
   ⚠️ #issue: Strongly suggest a change.
   🎉 #nice: Share a compliment. ↩