From ba5a4a74fb45988fbdf8e54089be5dd97354ca7b Mon Sep 17 00:00:00 2001 From: PauBadiaM Date: Wed, 9 Sep 2026 16:50:45 -0700 Subject: [PATCH 01/18] Replace draft/bench/improve loop with `acumen epoch` + knowledge wiki Reshape the optimization loop into a resumable training epoch. `acumen epoch` benches the current arm on the training split, distils each task's runs into a persistent per-task wiki (observations.md + hypothesis.md, one [version][model] block per epoch from noskill up), creates or improves the skill from that wiki, then benches the new version on the held-out split. - New `wiki.py`: cumulative, resume-safe per-task notes (one agent per task, in parallel); terse-by-design prompt; filtered-source isolation. - Rework `improve.py`: reads the wiki + parent skill + staged train transcripts + filtered package source; creates v1 when no version exists, improves after. - New `epoch.py` + `acumen epoch`; add `acumen wiki`; retire `acumen draft`. - Rename the held-out split `test` -> `valid` (reserving `test` for a future train/valid/test split) across paths, tasks, report, check, review, taskgen, scaffold, prompts, and tests. - bench: free codex sandbox preflight probe so a namespace/sandbox failure is reported up front instead of per paid run; raise the auth-probe budget cap (PREFLIGHT_MAX_USD) so premium models can pass preflight. - Rewrite the README (minimal) and the shipped acumen self-skill for the new loop. Validated end-to-end on a toy package across six models; the skill lifted the weakest model's held-out pass rate without regressing the others. Co-Authored-By: Claude Opus 4.8 --- README.md | 272 ++------- src/acumen/__init__.py | 33 +- src/acumen/_skills/data/SKILL.md | 54 +- .../_skills/data/references/authoring.md | 68 ++- .../_skills/data/references/benchmark.md | 8 +- .../_skills/data/references/python-api.md | 31 +- src/acumen/_skills/data/references/setup.md | 10 +- src/acumen/agents.py | 49 ++ src/acumen/bench.py | 24 +- src/acumen/check.py | 2 +- src/acumen/cli.py | 549 ++++++++++++++---- src/acumen/draft.py | 235 -------- src/acumen/epoch.py | 63 ++ src/acumen/improve.py | 455 +++++---------- src/acumen/paths.py | 6 +- src/acumen/prompts.py | 499 +++++++++------- src/acumen/report.py | 50 +- src/acumen/review.py | 2 +- src/acumen/scaffold.py | 8 +- src/acumen/taskgen.py | 8 +- src/acumen/tasks.py | 16 +- src/acumen/wiki.py | 547 +++++++++++++++++ tests/conftest.py | 4 +- tests/test_cli.py | 38 +- tests/test_core.py | 287 +++++---- 25 files changed, 1922 insertions(+), 1396 deletions(-) delete mode 100644 src/acumen/draft.py create mode 100644 src/acumen/epoch.py create mode 100644 src/acumen/wiki.py diff --git a/README.md b/README.md index 9dac23f..08ec1be 100644 --- a/README.md +++ b/README.md @@ -10,263 +10,58 @@ [badge-tests]: https://img.shields.io/github/actions/workflow/status/scverse/acumen/test.yaml?branch=main [badge-docs]: https://app.readthedocs.org/projects/acumen/badge/ -Agentic skills, tool instructions writen in plain text, allow agents to use tools more succesfuly and efficient. -However most python packages do not ship skills with them because developers have no easy way to build and benchmark skills for their tools. -Acumen closes this gap. Point it at a Python package and a few evaluation tasks, and it drafts a skill, benchmarks it and improves it across a train/test split so the gains are generalizable. +An agentic skill is a short, plain-text set of instructions that helps a coding agent use a tool +correctly. Most Python packages ship none — maintainers have no easy way to write one, or to tell +whether it actually helps. acumen closes that loop: point it at a package and a few tasks, and it +learns a skill in **training epochs**, each one benchmarking the current skill, distilling what went +right and wrong into a knowledge wiki, and rewriting the skill — always measured against a no-skill +baseline on a held-out split, so the gains generalize instead of memorizing answers. +## Installation -Many good tools are unusable by coding agents because their maintainers have no way to write -a skill for them — or, having written one, no way to tell whether it helps. acumen closes -that loop: point it at a Python package and a few tasks, and it drafts a skill, benchmarks it -against a no-skill baseline, and improves it across a train/test split so the gains are real -generalization, not memorized answers. - -- **`acumen check`** — rerun the script behind each task's answer, then have an agent judge - whether the prompt actually asks for what that script and answer produce. Both before a - benchmark pass pays to find out. -- **`acumen draft`** — write `skills/v1` from the package's own source. -- **`acumen bench`** — score a skill against a no-skill baseline, in a scrubbed sandbox where - the skill is the only difference between arms. Agent guidance the target ships itself is - removed from the venv first, so the baseline really is skill-free. -- **`acumen improve`** — refine the skill from its train results, then benchmark again. -- **`acumen report`** — aggregate every run into one self-contained `report.html`: success - rate per version, train vs. test. Bars are coloured by model, with a grey bar pooling all - of them; pass `--palette claude-opus-5=#3b7ea1` (repeatable) to recolour any of them. +Python 3.12+ is required. Install the backend you actually run — both are optional, and either +alone is a complete install: -You decide when to stop. Every version is benchmarked on both splits, and only train results -reach the improver — so a widening train/test gap is a visible sign a skill is overfitting -rather than genuinely helping. +| you run | install | also needs | +|---|---|---| +| Claude only | `pip install acumen[claude]` | an Anthropic key or a `claude` login | +| Codex only | `pip install acumen` | the `codex` CLI on `PATH`, plus a Codex login or key | +| both | `pip install acumen[all]` | both of the above | -## Quickstart +To install the acumen skill (guidance for using acumen itself) into your agent: ```bash -# 1. Scaffold a starter config.yaml and tasks.yaml -acumen init - -# 2. Fill in config.yaml (repo). Write tasks.yaml by hand, or generate it: -acumen tasks # mine the package for real analyses -> tasks.yaml + tasks/ -acumen check # is the ground truth right, and does each prompt ask for it? - -# 3. Then run the loop: -acumen bench --no-skill # the baseline arm -acumen draft # generate skills/v1 from the package source, or write by hand -acumen bench --skill v1 # benchmark the skill against the baseline -acumen improve # generate skills/v2 from v1's train results, or write by hand -acumen bench --skill v2 -acumen bench # or: every arm at once (baseline + each skills/vN) -acumen report # aggregate every run into report.html - -# 4. Once a version proves out, ship it into the package itself: -acumen ship --skill v2 # add a -install-skills console script (PR, or local edit) +acumen-install-skills --agent claude # or codex, agents, claude-science; or --dest ``` -`acumen ship` packages the chosen skill version into the target: the package gains a -`-install-skills` command that installs the skill into the skills directory of whichever -agent the user names — `--agent {claude,codex,agents,claude-science}`, or an explicit `--dest` — -so the package's own users get the guidance with one command, wherever they run their agent. The -same bundle installs verbatim into every framework. - -## Checking the ground truth - -A task is only worth benchmarking if its recorded answer is actually correct. A wrong answer makes -every model fail that task: real money spent, and the failure reads in the report as the model's -fault rather than the task's. `acumen check` catches the two ways that happens. - -**Does the answer still come out of running the code?** Each task keeps a **reproducer** at -`tasks/-.py`, a self-contained script that redoes the analysis in the target venv and -writes its answer to `answer.md` — the same contract a benchmark run has, graded the same way. -`acumen tasks` writes them as it generates the tasks; `acumen check` reruns them: +## Quickstart ```bash -acumen check # every task, both splits -acumen check --task bulk --split train # one cell, while you fix it -acumen check --jobs 8 --timeout 600 # or: --keep to inspect what a script wrote -``` - -You get one row per task and split — reproduced, wrong answer, script error, timed out, or no -script at all — then the summary statistics: how much of the task set has a reproducer, how much -of it reproduces, and how many tasks reproduce on both splits. Before running anything it checks -that the package imports in the venv at all, since that one failure would otherwise be reported -once per task. - -**Does the prompt actually ask for what the script and answer produce?** Reproducing an answer -proves the code and the answer agree. It says nothing about the prompt, and a prompt describing -something else fails every agent that reads it correctly. A real example: - -> Find the 3 most deactivated PROGENy pathways in Megakaryocytes … Report only the pathway names -> sorted by score **(ascending)**. +acumen init # scaffold config.yaml + tasks.yaml +# edit config.yaml: point `repo` at your package (a GitHub URL or local path) -The script sorted descending, the recorded answer was descending, the reproducer check said `ok` — -and every agent that honoured the prompt produced the reverse order and was graded wrong. So after -the scripts run, one agent reads every split's prompt, recorded answer and reproducer together and -adds a `review` column of `ok` or `mismatch`, with one line naming the contradiction and one naming -the fix. It never edits `tasks.yaml`: which of the three artifacts to repair is your call. +acumen tasks # mine the package for real tasks + ground-truth reproducers +acumen check # verify each task's answer reproduces before you spend -``` -task split status review detail -scell train ok ok MAPK;Estrogen;TGFb -scell test ok mismatch Trail;JAK-STAT;Estrogen +acumen epoch # one training round: bench train → wiki → create/improve skill → bench held-out +acumen epoch # run again for the next round (v2, v3, …) -1 split the review flagged - scell/test prompt says ascending; script and answer are descending - fix: say descending in the prompt, or reverse the answer +acumen report # aggregate every run into a self-contained report.html +acumen ship --skill v2 # ship the chosen skill into your package ``` -The review is on by default and picks its model from `meta_model`; it is the one phase that costs -money, so `acumen check --no-review` runs the reproducers alone and spends nothing — what you want -while iterating on a script. `check` takes the same `--auth`, `--stream` and `--log-dir` flags as -the other agentic commands, and `--max-turns`/`--max-usd` bound the reviewer. - -Either phase failing exits non-zero, so `acumen check` works as a gate before a pass. - -A task that needs no code to answer (a licence, a supported species, a documented default) sets -`needs_script: false`; its reproducer column reads `n/a` rather than counting as a gap, and its -prompt and answer are still reviewed. - -The reproducers hold the answers to the held-out test split, so nothing must feed them to an agent -under test. They are safe where they are: `bench`, `draft`, and `improve` confine their agents to -explicit read roots that never include your project directory, and the reviewer reads a staged copy -with no path back to `tasks.yaml`. - -`acumen tasks`, `acumen draft`, and `acumen improve` each accept `--feedback "…"` to steer the -agent with context it can't infer — which functionality to skip when generating tasks, what a -skill should emphasise or fix. The guidance is added to the prompt without overriding the -train/test isolation, and for `draft`/`improve` it is recorded in the version's `meta.json` and -shown in the report. (Don't paste held-out test answers into `improve` feedback — that would -defeat the split.) - -Claude and Codex can run side by side. Put both model families in `models` to compare them -in one matrix; model IDs beginning with `claude` use Claude Code, while `gpt-*`, `o1`, -`o3`, `o4`, and `codex-*` use Codex: - -```yaml -models: - - claude-opus-5 - - claude-sonnet-5 - - claude-haiku-4-5-20251001 - - gpt-5.6-sol - - gpt-5.6-terra - - gpt-5.6-luna -``` - -This spans each provider's quality/cost range; it is not a claim that the tiers are -one-to-one equivalents. - -Neither backend is required. Claude is an optional dependency and Codex is an external CLI, -so install only the one you run — `pip install acumen[claude]`, or plain `acumen` plus the -`codex` CLI on `PATH`. Selecting a model whose backend is missing fails immediately, with the -install command, before acumen prepares a target or spends anything. - -Claude API runs use `ANTHROPIC_API_KEY`; Codex API runs use `CODEX_API_KEY` (or -`OPENAI_API_KEY`). The meta-agent commands also accept a Codex model through the -`meta_model` config key or `--model`. +`acumen epoch` is the loop. Each round benchmarks the current arm on the training tasks, records +what happened per task in `wiki/`, creates or improves the skill from that wiki, and benchmarks the +new version on the held-out validation tasks. Re-run it to keep going; you decide when to stop. -Every agentic command — `bench` included — takes `--auth {auto,session,api}` and defaults to -the provider's logged-in subscription, falling back to its API key. Both billing modes report -tokens, so Acumen can calculate the same API-rate estimate for either. Under `session`, that -estimate is what the run *would* have cost at API rates, not money billed — so each run records -its `auth_mode` alongside the figure. +The stages also run on their own — `acumen bench`, `acumen wiki`, and `acumen improve`. -If the selected subscription runs out of usage or the API account runs out of credit, Acumen -invalidates the pass instead of scoring that as an agent failure: it prints the provider error, -cancels remaining cells for that provider, lets other providers finish all running and queued -cells, and exits non-zero. Replenish the credential and rerun the same command; automatic resume -retries the invalid and cancelled cells. Reports and `improve` refuse invalid quota/credit -evidence. - -`max_turns` and `max_usd` apply to both providers, but they are not equally strict for Codex, -which has no cap of its own — acumen enforces both against its event stream: - -- **`max_turns` bounds the run.** One `codex exec` is a single Codex turn however much work - happens inside it, so turns are counted in completed model actions (a message, a command, a - file change, a tool or search call) and the agent is stopped at the cap. -- **`max_usd` cannot.** Codex reports usage once, when the turn ends, so a breach is only - visible after the money is spent. The run is recorded as a budget failure — the same outcome - Claude gives it — but bound Codex spend with `max_turns`. acumen prints this before the pass. - -**Every cost acumen shows is inferred from tokens.** Each run records its breakdown (fresh -input, cache reads, cache writes, and output) and Acumen prices it with the rate table stored -in `result.json`. That gives Claude and Codex one comparable basis and prevents an old -benchmark from being silently re-priced, so it is what `cost_usd` holds and what every figure, -table, CSV column and console line reports. Where a backend supplies a dollar figure of its own -it is recorded beside it as `provider_cost_usd` (`recorded_cost_usd` in the report's sidecar -CSV), with the gap between the two, but nothing is plotted or tallied from it: Claude's SDK -total covers nested subagents that the run's own usage block does not, so a console reading it -would disagree with the report it summarises. A model no layer prices stays unpriced even when -the provider reported dollars, since one run on a basis the rest of the pass is not on is worse -than a visible gap. - -**Rates are read from the providers' pricing pages, never shipped with the package.** Prices -move, and each run's cost is frozen into its `result.json` and never recomputed, so a table -compiled into a release would store numbers that were already wrong. `bench` resolves rates -before it spends anything and **fails the pass** if the pages cannot be read: cost is a headline -metric, and a benchmark that cannot establish rates has not earned the numbers it would print. -`draft`, `improve`, `tasks`, `check`, and `ship` fetch too but degrade to unpriced instead — their -cost line is progress reporting, not stored evidence. - -Alongside the rates themselves each run records `price_source` (`config` or `fetched`) and -`price_rates_as_of`, so a pass run in August and another in October stay individually -attributable and one report can cover both without restating either. When arms in a report were -priced on different dates, the report says so: the cost gap between them includes the price -change, not only the skill's effect. - -```bash -acumen prices # the rates in use today, and where each came from -acumen prices --refresh # check pinned rates against what the providers publish -``` - -Pin rates with a `prices:` block in `config.yaml` to price a model the providers don't publish, -to price a gateway, or to record negotiated rates — pins outrank a live fetch, since only you -know what you are billed. They are also the only rates that can drift unnoticed, which is what -`--refresh` checks; it prints a diff for you to accept and never rewrites anything, because -picking the wrong tier or context band would silently misprice future runs. A model no layer -prices records its tokens and leaves report cost unavailable — never zero, which would read as -free. - -> One consequence worth knowing: Codex's `max_usd` cap is enforced from these same rates, so an -> unpriced model under Codex has no enforceable budget cap. Bound those runs with `max_turns`, -> or pin the rates. - -`draft`, `improve`, `tasks`, `ship`, and `check`'s review phase each drive an autonomous agent. -Every run writes a live `logs/acumen--.jsonl` (one event per step, flushed as it -goes — so you can watch progress by reading the file) and a rendered `.html` transcript. Add -`--stream` to mirror the conversation to the terminal, or `--log-dir` to change where the logs -land. +`acumen ship` gives your package a `-install-skills` command so its users can install the +skill into whichever agent they use (`--agent {claude,codex,agents,claude-science}`, or `--dest`). ## Getting started -Please refer to the [documentation][], -in particular, the [API documentation][]. - -## Installation - -You need to have Python 3.12 or newer installed on your system. -If you don't have Python installed, we recommend installing [uv][]. - -Install the backend you actually run — both are optional, and either alone is a complete -install: - -| you run | install | also needs | -|---|---|---| -| Claude only | `pip install acumen[claude]` | an Anthropic key or a `claude` login | -| Codex only | `pip install acumen` | the `codex` CLI on `PATH`, plus a Codex login or key | -| both | `pip install acumen[all]` | both of the above | - - - -And to install the acumen skill that ships with the package into your agent's skills directory, -run `acumen-install-skills --agent {claude,codex,agents,claude-science}` (or `--dest ` to -choose the directory yourself): - -```bash -acumen-install-skills --agent claude -``` +Please refer to the [documentation][], in particular the [API documentation][]. ## Release notes @@ -274,7 +69,7 @@ See the [changelog][]. ## Contact -For questions and help requests, you can reach out in the [scverse discourse][]. +For questions and help requests, reach out on the [scverse discourse][]. If you found a bug, please use the [issue tracker][]. ## Citation @@ -288,4 +83,3 @@ If you found a bug, please use the [issue tracker][]. [documentation]: https://acumen.readthedocs.io [changelog]: https://acumen.readthedocs.io/page/changelog.html [api documentation]: https://acumen.readthedocs.io/page/api.html -[pypi]: https://pypi.org/project/acumen diff --git a/src/acumen/__init__.py b/src/acumen/__init__.py index 542ce72..e40e817 100644 --- a/src/acumen/__init__.py +++ b/src/acumen/__init__.py @@ -29,7 +29,6 @@ summarize_checks, ) from acumen.config import Config, ConfigError, load_config, parse_config -from acumen.draft import DraftError, DraftResult, draft_skill from acumen.env import ( AuthMode, EnvError, @@ -41,15 +40,14 @@ scrubbed_env, session_auth_available, ) +from acumen.epoch import EpochPlan, resolve_epoch from acumen.grade import Grade, Reason, grade_answer, grade_run from acumen.improve import ( ImproveError, ImproveResult, - TrainRun, - collect_train_runs, - find_test_access, + find_valid_access, improve_skill, - make_test_guard, + make_valid_guard, ) from acumen.logs import LiveLog from acumen.paths import RunKey, Split, arm_name, is_complete, parse_run_dir, run_dir, skill_from_arm @@ -112,8 +110,26 @@ render_codex_transcript, render_transcript, ) +from acumen.wiki import ( + RunRecord, + TaskWikiResult, + WikiError, + collect_arm_runs, + recorded_arms, + update_wiki, +) __all__ = [ + "EpochPlan", + "resolve_epoch", + "find_valid_access", + "make_valid_guard", + "RunRecord", + "TaskWikiResult", + "WikiError", + "collect_arm_runs", + "recorded_arms", + "update_wiki", "AuthMode", "AgentError", "AgentOptions", @@ -126,8 +142,6 @@ "CheckSummary", "Config", "ConfigError", - "DraftError", - "DraftResult", "EnvError", "Grade", "Harvest", @@ -160,7 +174,6 @@ "TaskGenResult", "TaskSplit", "Trajectory", - "TrainRun", "__version__", "api_auth_available", "arm_metrics", @@ -176,11 +189,8 @@ "build_report", "check_task_split", "check_tasks", - "collect_train_runs", - "draft_skill", "dump_tasks", "find_skill_access", - "find_test_access", "generate_tasks", "grade_answer", "grade_run", @@ -191,7 +201,6 @@ "installer_exists", "is_complete", "make_skill_guard", - "make_test_guard", "latest_version", "load_config", "locate_transcript", diff --git a/src/acumen/_skills/data/SKILL.md b/src/acumen/_skills/data/SKILL.md index 0a7fbf3..1b93578 100644 --- a/src/acumen/_skills/data/SKILL.md +++ b/src/acumen/_skills/data/SKILL.md @@ -1,6 +1,6 @@ --- name: acumen -description: Use for any question or task involving the Python package `acumen` (its CLI or its API) — setting up a benchmark project against a target package, writing or generating benchmark tasks, drafting/improving/hand-editing an agent Skill, running and interpreting skill-vs-baseline benchmark passes, shipping a skill into the target package, diagnosing a run, and choosing what to do next in that loop — open it before answering or running anything, because acumen's defaults, guardrails and correct next step are not guessable from the command names. +description: Use for any question or task involving the Python package `acumen` (its CLI or its API) — setting up a benchmark project against a target package, writing or generating benchmark tasks, running training epochs that create/improve an agent Skill from a knowledge wiki, running and interpreting skill-vs-baseline benchmark passes, shipping a skill into the target package, diagnosing a run, and choosing what to do next in that loop — open it before answering or running anything, because acumen's defaults, guardrails and correct next step are not guessable from the command names. --- # acumen @@ -13,15 +13,20 @@ run from the project dir and default to `config.yaml`, `tasks.yaml`, `skills/`, ```bash acumen init # scaffold config.yaml + tasks.yaml (placeholders the user must fill) acumen tasks # optional: mine the package for tasks, run them for ground truth -acumen bench --no-skill # baseline arm (bare `acumen bench` runs every arm there is) -acumen draft # agent reads the package source -> skills/v1/ -acumen bench --skill v1 -acumen improve # agent reads v1's TRAIN runs -> skills/v2/ -acumen bench --skill v2 +acumen epoch # one training epoch: bench train -> wiki -> create/improve skill -> bench valid +acumen epoch # again: learns from the last version, produces the next acumen report # report.html + report.csv over every arm on disk acumen ship --skill v2 # wire a `-install-skills` script into the target package ``` +`acumen epoch` is the loop. One invocation is one training epoch: it benchmarks the current +arm on the **train** split, distils each task's runs into a per-task **wiki** +(`wiki//observations.md` + `hypothesis.md`, accumulating one `[version][model]` block per +epoch from `noskill` on up), then creates the first skill (or improves the latest) from that +wiki, and benchmarks the new version on the held-out **valid** split. It is fully resumable — a +crashed epoch continues where it stopped; a completed one starts the next. The granular commands +still exist (`acumen bench`, `acumen wiki`, `acumen improve`) for running a single stage. + That is the shape of the loop, not a script to execute unattended, and not a fixed order: several stages have more than one correct next move, and some stages call for talking to the user rather than running the next command at all. @@ -41,11 +46,10 @@ a command. | Just happened | What to propose next | |---|---| | `acumen init` | Both files are placeholders (`repo: OWNER/REPO`, `REPLACE_ME` answers). Ask the user for what only they can supply — the target repo/path, ref, models, budgets — and fill in `config.yaml` with their answers. Do not guess a target or run the next command. | -| `config.yaml` filled | Offer both ways to get tasks: `acumen tasks` to generate them, or writing `tasks.yaml` by hand. Review generated tasks with the user before benching. | -| config + tasks ready | `acumen bench --no-skill` and `acumen draft` are **both** correct next steps — they are independent, and you need the baseline arm and a skill arm before any comparison means anything. Offer both. | -| a skill version exists | `acumen bench --skill vN`, or bare `acumen bench` to cover the baseline and every version at once (resume means only the unbenched cells cost anything). | -| any `bench` finished | `acumen report` — the per-arm, per-split numbers come from the report; do not judge a version by eyeballing runs or by one arm's pass count. | -| `report` written | Discuss the results with the user and propose next steps: `improve` + re-bench if it has not beaten the baseline or the previous version, `ship` once a version has proven out, or stop. The user decides. | +| `config.yaml` filled | Offer both ways to get tasks: `acumen tasks` to generate them, or writing `tasks.yaml` by hand. Review generated tasks with the user before an epoch. | +| config + tasks ready | `acumen epoch` runs the whole first round (benches the noskill baseline on train+valid, builds the wiki, creates `skills/v1/`, benches it on valid). Propose it, and confirm the budget first — one epoch spends on many benchmark cells plus the meta-agents. | +| an epoch finished | `acumen report` — the per-arm, per-split numbers come from the report; do not judge a version by eyeballing runs. Then discuss with the user whether to run another `acumen epoch`, `ship`, or stop. | +| `report` written | Discuss the results with the user and propose next steps: another `acumen epoch` if the latest version has not beaten the baseline or the previous version, `ship` once a version has proven out, or stop. The user decides. | | user has a skill dir from elsewhere | Copy it into `skills/v1/` (next unused version) inside the acumen project — versions are only ever read from `skills/`; there is no import command and no external path flag. | ## Route by goal @@ -55,9 +59,10 @@ a command. | Start a project for package X | `acumen init`, then fill `config.yaml` with the user | `references/setup.md` | | Get benchmark tasks without writing them | `acumen tasks [--force] [--feedback "…"]` | `references/setup.md` | | Write tasks by hand | edit `tasks.yaml` | `references/setup.md` | -| Get a first skill | `acumen draft [--feedback "…"]` | `references/authoring.md` | -| Measure whether the skill helps | `acumen bench --no-skill` and `acumen bench --skill vN` | `references/benchmark.md` | -| Make the skill better | `acumen improve [--from vN]` then bench the new version | `references/authoring.md` | +| Run one training round (the loop) | `acumen epoch [--feedback "…"]` | `references/authoring.md` | +| Create or improve the skill only | `acumen improve [--from vN]` (reads the wiki) | `references/authoring.md` | +| Distil an arm's train runs into the wiki | `acumen wiki [--no-skill | --skill vN]` | `references/authoring.md` | +| Measure a single arm | `acumen bench --no-skill` / `acumen bench --skill vN` | `references/benchmark.md` | | See results / decide when to stop | `acumen report` | `references/benchmark.md` | | Hand-edit a skill version | copy `skills/vN` → `skills/v(N+1)`, edit, bench it | `references/authoring.md` | | Give package users the skill | `acumen ship --skill vN` | `references/ship.md` | @@ -102,19 +107,20 @@ a command. that is 108. Check with `acumen bench --dry-run` (it plans and exits, spending nothing, over the same arms the real run would) and agree the size with the user before spending. Trim with `models:`, `n_replicates: 1`, `--task ID`, `--split`, or by naming one arm. -3. **`max_turns`/`max_usd` in `config.yaml` cap benchmark agents only.** `draft`, - `improve`, `tasks`, and `ship` are **unbounded** unless you pass `--max-turns`/`--max-usd`. -4. **Skill versions are immutable.** `draft` refuses (exit 2) if any `skills/vN` exists; - `improve` always writes the next unused directory. Never edit a benched version in - place — its hash is recorded in every `result.json`. +3. **`max_turns`/`max_usd` in `config.yaml` cap benchmark agents only.** The meta-agents + (`improve`, `wiki`, `tasks`, `ship`) are **unbounded** unless you pass `--max-turns`/`--max-usd`. +4. **Skill versions are immutable.** `improve` (and `epoch`) always writes the next unused + directory; a resumed epoch skips `improve` when its version already exists. Never edit a + benched version in place — its hash is recorded in every `result.json`. 5. **`SKILL.md` frontmatter `name` must equal `config.skill_name`**, which defaults to the repo's last path component, slugified and lowercased (`.../My_Pkg` → `my_pkg`). A mismatch makes `bench`/`improve`/`ship` fail on load. `description` must be non-empty. -6. **`improve` needs benched train evidence for its parent.** Run - `acumen bench --skill vN` before `acumen improve`, or it errors with nothing to read. -7. **Never leak the test split.** `improve` is structurally and hook-blocked from - `runs/*/test/`; don't defeat that by pasting test answers into `--feedback`. A widening - train/test gap in the report is the overfitting signal you are watching for. +6. **`improve` needs benched train evidence.** `acumen epoch` benches the parent arm on train + first, so it is self-sufficient; standalone `acumen improve` errors if the parent arm has no + train runs — bench it (and run `acumen wiki`) first. +7. **Never leak the valid split.** `improve`/`wiki` are structurally and hook-blocked from + `runs/*/valid/`; don't defeat that by pasting valid answers into `--feedback`. A widening + train/valid gap in the report is the overfitting signal you are watching for. 8. **Task prompts must not name the target package** — the harness already tells the agent which package to use and that it is installed. 9. **Resume is automatic**: a valid run is "complete" when its `result.json` exists and is diff --git a/src/acumen/_skills/data/references/authoring.md b/src/acumen/_skills/data/references/authoring.md index 1817d9a..e0f2e4a 100644 --- a/src/acumen/_skills/data/references/authoring.md +++ b/src/acumen/_skills/data/references/authoring.md @@ -1,4 +1,4 @@ -# Authoring skill versions: draft, improve, hand-edit +# Authoring skill versions: epoch, wiki, improve, hand-edit ## The skill directory contract @@ -17,17 +17,38 @@ skills/v1/ every `result.json` — so editing a version after benching it silently invalidates the comparison. **Versions are immutable; always make a new one.** -## `acumen draft` +## `acumen epoch` — the loop ```bash -acumen draft [--force] [--feedback "…"] [--model M] [--max-turns N] [--max-usd X] +acumen epoch [--feedback "…"] [--model M] [--max-concurrency N] [--replicates N] [--stream] [--log-dir logs] [--auth auto|session|api] ``` -One agent that reads the target's **source** (docstrings, examples, docs) and writes -`skills/v1/`. It works in a staging dir; only a skill that loads and validates is promoted, -so a failed draft leaves no broken version behind. Exits 2 if any version already exists — -pass `--force` to add another, or use `improve` to build on the latest. +One training epoch, end to end and fully resumable: + +1. **Bench the current arm on `train`** (the first epoch also benches the `noskill` baseline + on `valid`, for the report). +2. **Update the wiki** for that arm — one agent per task (see `acumen wiki` below). +3. **Create or improve the skill** into the next version, from the wiki (see `acumen improve`). +4. **Bench the new version on the held-out `valid` split.** + +Re-run it: a crashed epoch resumes at the step it stopped (state is read from disk — a version +present but not valid-benched means "finish that epoch"; a fully benched latest means "start the +next"). A completed epoch's next invocation learns from the version it just produced. + +## `acumen wiki` — the knowledge base + +```bash +acumen wiki [--no-skill | --skill vN] [--feedback n/a] [--model M] [--max-concurrency N] + [--stream] [--log-dir logs] [--auth auto|session|api] +``` + +For each task, one agent reads that arm's **train** runs across models and replicates and +**appends** a terse block to `wiki//observations.md` and `wiki//hypothesis.md`, tagged +`[version][model]` (`noskill` first, then `v1`, `v2`, …). Cumulative and idempotent: an arm +already recorded for a task (tracked in `wiki//.arms`) is skipped. Defaults to the latest +arm; `--no-skill` records the baseline, `--skill vN` a specific version. Entries are kept short on +purpose — the whole wiki is read by the improver every epoch. ## `acumen improve` @@ -36,25 +57,27 @@ acumen improve [--from vN] [--feedback "…"] [--model M] [--max-turns N] [--max [--stream] [--log-dir logs] [--auth auto|session|api] ``` -Reads **how the parent version performed on the train split** — not the package source — -and edits a copy of it into the next version. +Reads the **wiki** (the distilled per-task notes) plus the **filtered** package source, and +either creates the first skill (when no version exists yet, from the `noskill` wiki) or edits a +copy of the parent into the next version. - Defaults to the latest version; `--from vN` picks another. Always writes the next unused - directory. -- Requires `runs/skill_vN/train/**/result.json` to exist: bench the parent first. -- The agent gets a curated copy of train runs (`SUMMARY.md` with failures first, plus each - run's prompt, expected, actual, `script.py`, and transcript). The real `runs/` tree is not - in scope, and a `PreToolUse` hook denies any path resolving under `runs/*/test/`. -- The CLI warns if the new version is byte-identical to its parent (the improver changed - nothing) — that is a signal to give it sharper evidence or `--feedback`. + directory. With no versions it creates `v1`. +- Requires the parent arm's `runs//train/**/result.json` to exist (so run an epoch, or bench + the arm and `acumen wiki`, first). +- The agent reads a staged copy of the wiki + the parent arm's train transcripts (for drill-down). + The real `runs/` tree is denied, and a `PreToolUse` hook denies any path under `runs/*/valid/`. + The source is the *filtered* copy — a skill the package itself ships is stripped and blocked, so + the optimization never re-serves it. +- The CLI warns if the new version is byte-identical to its parent (the improver changed nothing). -## `--feedback` on `tasks` / `draft` / `improve` +## `--feedback` on `tasks` / `improve` / `epoch` Free text injected into the agent's prompt as *subordinate* guidance — it never overrides the isolation or anti-overfit rules. Use it for what the agent cannot infer: package -context, what to emphasise, which functionality to skip. For `draft`/`improve` it is -recorded in `meta.json` and shown in the report. **Never paste test-split answers into -`improve --feedback`** — that defeats the split. +context, what to emphasise, which functionality to skip. For `improve` it is recorded in +`meta.json` and shown in the report. **Never paste valid-split answers into `--feedback`** — +that defeats the split. ## Writing or editing a version by hand @@ -78,12 +101,13 @@ report. What makes a version score well: preprocessing, the function that looks right but isn't, argument shapes/orientation, where output lands, the right order of steps. - **Never name a dataset, parameter value, column, or expected answer from the train - tasks.** That is overfitting, and the test split will catch it. + tasks.** That is overfitting, and the valid split will catch it. - Verify every claim against the installed package before writing it down. ## Watching a long agent run -`tasks`/`draft`/`improve`/`ship` each write `logs/acumen--.jsonl`, +`tasks`/`wiki`/`improve`/`ship` each write `logs/acumen--.jsonl` +(the wiki writes one per task, `acumen-wiki--…`), one compact event per agent message, **flushed as it goes** — read that file to follow progress instead of streaming into your context. Tool results are recorded by status and size, not inlined. `--stream` mirrors the conversation to the terminal; `--log-dir` moves diff --git a/src/acumen/_skills/data/references/benchmark.md b/src/acumen/_skills/data/references/benchmark.md index 56e8a23..057570a 100644 --- a/src/acumen/_skills/data/references/benchmark.md +++ b/src/acumen/_skills/data/references/benchmark.md @@ -3,7 +3,7 @@ ## `acumen bench` ```bash -acumen bench [--no-skill | --skill v1] [--split train|test]... [--task ID]... +acumen bench [--no-skill | --skill v1] [--split train|valid]... [--task ID]... [--replicates N] [--max-concurrency N] [--dry-run] [--no-resume] [--keep-sandboxes] [--refresh-target] [--auth auto|session|api] @@ -87,7 +87,7 @@ exits 2 if the pages cannot be read. Each run records `price_source` (`config` o attributable and one report can mix them; the report flags arms priced on different dates, whose cost gap includes the price change rather than only the skill's effect. -`draft`, `improve`, `tasks`, and `ship` also fetch, but degrade to unpriced rather than failing: +`improve`, `wiki`, `tasks`, and `ship` also fetch, but degrade to unpriced rather than failing: their cost line is progress reporting. Note that Codex's `max_usd` is derived from these rates, so an unpriced Codex run has no enforceable budget cap — bound it with `max_turns`. @@ -145,7 +145,7 @@ acumen report [--runs runs] [--tasks tasks.yaml] [--skills skills] [--out report - Writes `report.html` **and a sidecar `report.csv`** next to it, overwriting both; the report always reflects every run currently on disk across every arm. - Reads only `result.json` files — never transcripts. Fails if `runs/` holds none. -- **The figures show the TEST split** (the held-out measure). The per-run table below them +- **The figures show the VALID split** (the held-out measure). The per-run table below them lists both splits. - Sections: Overview (success rate, tokens, cost, time per arm, bars coloured by model), Per-task breakdown, Runs table (links to each `transcript.html`), and — when `--skills` @@ -157,6 +157,6 @@ acumen report [--runs runs] [--tasks tasks.yaml] [--skills skills] [--out report Run `acumen report` after every bench — it is how a version is judged, and the only view that compares arms and splits side by side. Then take the results to the user: say what improved, what did not, and what you would do next (`improve` and re-bench, more tasks, -`ship`, or stop). Train improving while test does not is overfitting — that gap is exactly +`ship`, or stop). Train improving while valid does not is overfitting — that gap is exactly what the split exists to expose — but the call on when to stop and what to ship is the user's. diff --git a/src/acumen/_skills/data/references/python-api.md b/src/acumen/_skills/data/references/python-api.md index 4f58cda..9ed1ea6 100644 --- a/src/acumen/_skills/data/references/python-api.md +++ b/src/acumen/_skills/data/references/python-api.md @@ -75,20 +75,35 @@ is_complete(d) # a non-empty result.json is what "done" means ## The meta-agents -`draft_skill`, `improve_skill`, `generate_tasks`, `ship_skill` are all coroutines taking -keyword-only args (`cfg=`, `target=`, plus their own roots) and returning a result dataclass -(`DraftResult`, `ImproveResult`, `TaskGenResult`, `ShipResult`) carrying the new `Skill`, -`cost_usd`, `turns`, and log paths. `max_turns`/`max_usd` default to `None` = **unbounded**. -Pass a `LiveLog` as `log=` for the JSONL feed: +`improve_skill`, `update_wiki`, `generate_tasks`, `ship_skill` are all coroutines taking +keyword-only args (`cfg=`, `target=`, plus their own roots) and returning a result +(`ImproveResult`, a list of `TaskWikiResult`, `TaskGenResult`, `ShipResult`) carrying the new +`Skill`/notes, `cost_usd`, `turns`, and log paths. `improve_skill` **creates** the first skill when +`parent_version=None` (reading the `noskill` wiki) and improves otherwise. `max_turns`/`max_usd` +default to `None` = **unbounded**. Pass a `LiveLog` as `log=` for the JSONL feed: ```python -from acumen import LiveLog, draft_skill +from acumen import LiveLog, improve_skill -log = LiveLog.open(Path("logs"), "draft", stream=False) +log = LiveLog.open(Path("logs"), "improve", stream=False) with log: - result = asyncio.run(draft_skill(cfg=cfg, target=target, skills_root=Path("skills"), auth_mode="session", log=log)) + result = asyncio.run( + improve_skill( + cfg=cfg, + target=target, + skills_root=Path("skills"), + runs_root=Path("runs"), + wiki_root=Path("wiki"), + tasks=tasks, + auth_mode="session", + log=log, + ) + ) ``` +`resolve_epoch(skills_root, valid_complete=…)` returns the `EpochPlan` (parent, new version, and +whether it is the first/a resumed epoch) that `acumen epoch` uses to drive the whole loop. + ## Aggregating results ```python diff --git a/src/acumen/_skills/data/references/setup.md b/src/acumen/_skills/data/references/setup.md index 67f9eb8..2543b9d 100644 --- a/src/acumen/_skills/data/references/setup.md +++ b/src/acumen/_skills/data/references/setup.md @@ -21,12 +21,12 @@ Only `repo` is required; delete a line to take its default. | `pip_packages` | `[]` | Packages the target declares nowhere, installed alongside it. PEP 508 specifiers allowed (`numpy<2`). | | `python` | `"3.12"` | Interpreter for the target's venv (quote it — `3.10` unquoted is a float). | | `env_passthrough` | `[]` | Extra env var names agents may keep. The agent env is a clean allowlist (auth/proxy/TLS only); **everything else from your shell is blanked**. A target needing `OMP_NUM_THREADS`, `R_HOME`, a service key, etc. must name it here. | -| `models` | `[claude-opus-5]` | Benchmark models. Claude (`claude*`) and Codex (`gpt-*`, `o1`/`o3`/`o4`, `codex-*`) models may be mixed. Duplicates rejected. `models[0]` is the default for the four `*_model` keys below. | +| `models` | `[claude-opus-5]` | Benchmark models. Claude (`claude*`) and Codex (`gpt-*`, `o1`/`o3`/`o4`, `codex-*`) models may be mixed. Duplicates rejected. `models[0]` is the default for `meta_model`. | | `n_replicates` | `3` | Runs per (model, task, split) cell. | | `max_concurrency` | `4` | Simultaneous benchmark agents. | | `max_turns` | `40` | **Benchmark agents only.** | | `max_usd` | `3.0` | **Benchmark agents only.** | -| `draft_model` / `improve_model` / `tasks_model` / `ship_model` | `models[0]` | Meta-agent models; overridable per command with `--model`. | +| `meta_model` | `models[0]` | Model for the meta-agents (`improve`, `wiki`, `tasks`, `ship`, `check` review); overridable per command with `--model`. | | `skill_name` | repo basename, slugified + lowercased | Must equal the `name:` in the skill's frontmatter. | | `prices` | built-in table | Per-model token rates (USD per million): `{model: {input, output, cached_input?, cache_write?, cache_write_5m?, cache_write_1h?}}`. Overrides or extends acumen's table — needed for a model it doesn't ship a rate for, a gateway, or negotiated rates. See `acumen prices`. | @@ -54,7 +54,7 @@ tasks: prompt: >- One paragraph: the goal, the input, and exactly what to report. answer: ONE_TOKEN - test: + valid: prompt: >- The same analysis on a different input / target. answer: ANOTHER_TOKEN @@ -64,7 +64,7 @@ tasks: # model: claude-sonnet-5 ``` -Both `train` and `test` are required, each with non-empty `prompt` and `answer`. Both +Both `train` and `valid` are required, each with non-empty `prompt` and `answer`. Both splits always run in a pass; only train results ever reach `improve`. ### Writing a task that measures anything @@ -77,7 +77,7 @@ splits always run in a pass; only train results ever reach `improve`. - **State the goal, not the recipe.** No numbered steps, no function/argument names, no description of the data's shape or columns. What is being measured is whether the agent can find the "how" itself; a prompt that spells out the calls measures nothing. -- Train and test must be **the same analysis on different inputs**, with different correct +- Train and valid must be **the same analysis on different inputs**, with different correct answers — otherwise a skill can pass by memorizing one answer. - A good task is one the target package solves and the **no-skill baseline gets wrong** — a task the baseline already passes leaves no room to show a gain. The `--no-skill` arm is diff --git a/src/acumen/agents.py b/src/acumen/agents.py index 2ae4dd8..d051c4d 100644 --- a/src/acumen/agents.py +++ b/src/acumen/agents.py @@ -21,6 +21,7 @@ import shutil import signal import sys +import tempfile import time from collections.abc import Callable, Sequence from contextlib import aclosing, suppress @@ -696,6 +697,54 @@ def _sandbox_failure(lines: Sequence[str]) -> str | None: return None +async def codex_sandbox_probe(env: dict[str, str], *, timeout: float = 30.0) -> str | None: + """Return ``None`` if Codex's command sandbox can start here, else an actionable error string. + + Runs ``codex sandbox -- true`` — no model call, so it is free and deterministic — and reads + its stderr for a namespace/sandbox-init failure. Codex runs every benchmark command inside + this sandbox, so if it cannot start, no Codex run can save an answer; catching it in preflight + turns a paid, per-run ``error_sandbox`` into one free up-front message. + + Conservative on purpose: a probe that cannot run, times out, or exits non-zero for a reason + that is *not* a recognised sandbox failure returns ``None`` (let the run proceed) rather than + block a setup that might work. Only a matched sandbox-init failure is reported. + """ + cli = shutil.which("codex", path=env.get("PATH")) + if cli is None: + return None # a missing CLI is reported by check_agent_cli; nothing to probe here + # Run in a throwaway empty directory with the read-only profile. This tests only the one thing + # that fails on a locked-down host — whether Codex can create its namespace sandbox at all — + # without the workspace profile's git-protection, which mounts a tmpfs over the workspace's + # ``.git`` and errors in a non-repo directory (a false positive unrelated to the real runs, + # which use their own filesystem profile in a temp sandbox dir). + probe_dir = tempfile.mkdtemp(prefix="acumen-codex-probe-") + argv = [cli, "sandbox", "-c", 'default_permissions=":read-only"', "--", "true"] + proc: asyncio.subprocess.Process | None = None + try: + proc = await asyncio.create_subprocess_exec( + *argv, cwd=probe_dir, env=env, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE + ) + _, stderr = await asyncio.wait_for(proc.communicate(), timeout=timeout) + except (OSError, TimeoutError): + if proc is not None: + with suppress(ProcessLookupError): + proc.kill() + return None + finally: + shutil.rmtree(probe_dir, ignore_errors=True) + if proc.returncode == 0: + return None + hit = _sandbox_failure(stderr.decode("utf-8", "replace").splitlines()) + if hit is None: + return None # non-zero for some other reason — not a namespace block + return ( + "codex's command sandbox could not start on this system, so no codex run can execute its " + f"commands or save an answer ({hit}). codex isolates every benchmark command inside an " + "unprivileged namespace sandbox, which this host does not currently permit. Enable " + "unprivileged user namespaces for codex, or run acumen where they are allowed." + ) + + def _is_turn_item(event: dict[str, Any]) -> bool: """Whether ``event`` is one completed model action — the unit Codex turns are counted in.""" if event.get("type") != "item.completed": diff --git a/src/acumen/bench.py b/src/acumen/bench.py index 9e946f4..9137a6d 100644 --- a/src/acumen/bench.py +++ b/src/acumen/bench.py @@ -314,6 +314,13 @@ async def one(item: PlannedRun) -> RunOutcome | None: return outcomes +#: Budget cap for the auth-probe run. Generous on purpose: the probe is bounded by ``max_turns=5`` +#: and only confirms the credential works, but one turn carries the full ``claude_code`` system +#: prompt, whose input cost at a premium model's rate can exceed a tight cap and fail preflight +#: before the model replies. A trivial "reply ok" never approaches this. +PREFLIGHT_MAX_USD = 2.0 + + async def _preflight_model( model: str, *, @@ -323,7 +330,7 @@ async def _preflight_model( env_passthrough: Sequence[str] | None, ) -> str | None: """Return None if the model can authenticate, else a short error string.""" - from acumen.agents import AgentOptions, run_agent + from acumen.agents import AgentOptions, codex_sandbox_probe, run_agent from acumen.sandbox import sandbox provider = provider_for_model(model) @@ -335,14 +342,27 @@ async def _preflight_model( provider=provider, env_passthrough=env_passthrough, ) as box: + # Codex runs every command inside its own namespace sandbox. A trivial auth probe + # writes no files and runs no command, so it would pass even where that sandbox + # cannot start — and then every real run fails with error_sandbox after paying for + # the tokens. Test the sandbox itself first (free, no model call); report and stop + # here if it cannot start. + if provider == "codex": + sandbox_error = await codex_sandbox_probe(box.env) + if sandbox_error is not None: + return sandbox_error result = await run_agent( "Reply with only the word: ok", options=AgentOptions( cwd=box.root, env=box.env, model=model, + # Bounded by turns, not dollars: the probe only confirms the credential works, + # but one turn carries the full claude_code system prompt, whose input cost at a + # premium model's rate (Opus) exceeds a tight cap — so a dollar cap that low + # fails preflight before the model can reply. max_turns=5 already bounds it. max_turns=5, - max_usd=0.10, + max_usd=PREFLIGHT_MAX_USD, discover_skills=False, confine=False, ), diff --git a/src/acumen/check.py b/src/acumen/check.py index bcab39a..15b2293 100644 --- a/src/acumen/check.py +++ b/src/acumen/check.py @@ -18,7 +18,7 @@ throwaway environment an agent gets. Each one still gets a fresh empty working directory, so its ``answer.md`` cannot be another script's and nothing it writes lands in the project. -The scripts hold the ground truth for the **held-out test split**, so nothing may ever read +The scripts hold the ground truth for the **held-out valid split**, so nothing may ever read them into an agent. That holds today because ``bench``, ``draft`` and ``improve`` confine their agents to explicit read roots (:mod:`acumen.guard`) that never include the project directory. """ diff --git a/src/acumen/cli.py b/src/acumen/cli.py index a7ac24b..585b814 100644 --- a/src/acumen/cli.py +++ b/src/acumen/cli.py @@ -29,12 +29,12 @@ summarize_checks, ) from acumen.config import Config, ConfigError, load_config -from acumen.draft import DraftError, draft_skill from acumen.env import DEFAULT_CACHE_ROOT, AuthMode, EnvError, prepare_target, resolve_auth_mode +from acumen.epoch import resolve_epoch from acumen.grade import INVALID_REASONS from acumen.improve import ImproveError, improve_skill from acumen.logs import LiveLog -from acumen.paths import SPLITS, arm_name +from acumen.paths import SPLITS, Split, arm_name from acumen.pricefeed import ( PRICE_SOURCES, PRICE_TIER, @@ -51,9 +51,10 @@ from acumen.runner import RunOutcome, StderrFilter from acumen.scaffold import InitError, is_scaffold_tasks, scaffold from acumen.ship import ShipError, ship_skill -from acumen.skills import Skill, SkillError, available_versions, latest_version, load_skill +from acumen.skills import Skill, SkillError, available_versions, latest_version, load_skill, skill_dir from acumen.taskgen import TaskGenError, generate_tasks from acumen.tasks import Task, TaskError, load_tasks +from acumen.wiki import WikiError, collect_arm_runs, update_wiki def _add_bench_args(parser: argparse.ArgumentParser) -> None: @@ -380,6 +381,100 @@ def _print_skill_loading(outcomes: Sequence[RunOutcome], arm: _Arm, skill_name: print(f"warning: {skill_name} loaded in {loaded} baseline runs", file=sys.stderr) +def _resolve_bench_auth(models: set[str], auth: str) -> dict[AgentProvider, AuthMode]: + """Resolve one auth mode per provider present, checking each CLI and printing the choice.""" + providers = {provider_for_model(model) for model in models} + auth_modes = {provider: resolve_auth_mode(auth, provider=provider) for provider in providers} + for provider in sorted(providers): + check_agent_cli(provider) + _print_auth(auth_modes[provider], provider) + _warn_codex_accounting(provider) + if "session" in auth_modes.values(): + print( + "note: cost_usd for session-billed runs is what they would have cost at API " + "rates, not metered spend; each run records its auth_mode", + file=sys.stderr, + ) + return auth_modes + + +def _execute_arms( + arms: Sequence[_Arm], + *, + cfg: Config, + target, + runs_root: Path, + auth_modes: dict[AgentProvider, AuthMode], + prices: PriceTable, + keep_sandboxes: bool, + progress: _Progress | None = None, +) -> list[RunOutcome]: + """Run each arm's pending runs sequentially, sharing one progress counter across them. + + Arms run one after another: every run in a matrix shares one skill, and a sequential pass + keeps each arm's tally readable while the progress counter spans the whole thing. Raises + :class:`BenchmarkInvalidError` if a harness failure decides the pass. + """ + running = [arm for arm in arms if arm.todo] + todo = [item for arm in running for item in arm.todo] + if not todo: + return [] + progress = progress or _Progress(len(todo)) + collected: list[RunOutcome] = [] + for arm in running: + if len(running) > 1: + print(f"\n=== arm {arm.name}: {len(arm.todo)} runs ===", flush=True) + started = time.monotonic() + outcomes = asyncio.run( + run_matrix( + arm.todo, + target=target, + runs_root=runs_root, + max_concurrency=cfg.max_concurrency, + auth_modes=auth_modes, + skill=arm.skill, + skill_name=cfg.skill_name, + keep_sandbox=keep_sandboxes, + stderr=StderrFilter(), + on_start=progress.on_start, + on_done=progress.on_done, + env_passthrough=cfg.env_passthrough, + prices=prices, + ) + ) + collected.extend(outcomes) + _print_run_summary(outcomes, time.monotonic() - started, label=arm.name if len(running) > 1 else "") + _print_skill_loading(outcomes, arm, cfg.skill_name) + return collected + + +def _build_arms( + specs: Sequence[tuple[str | None, Sequence[Split]]], + *, + cfg: Config, + tasks: Sequence[Task], + runs_root: Path, + skills_root: Path, + resume: bool = True, + task_ids: Sequence[str] | None = None, +) -> list[_Arm]: + """Build arms for an explicit list of ``(version, splits)`` specs (for the epoch orchestrator).""" + arms = [] + for version, splits in specs: + skill = None if version is None else load_skill(skills_root, version, expect_name=cfg.skill_name) + planned = build_matrix(cfg, tasks, skill=version, splits=splits, task_ids=task_ids) + todo = pending(planned, runs_root, resume=resume) + arms.append(_Arm(version=version, skill=skill, planned=planned, todo=todo)) + return arms + + +def _invalid_bench_note() -> None: + print( + "Fix or replenish that credential, then rerun the same command; invalid and cancelled cells remain pending.", + file=sys.stderr, + ) + + def _cmd_bench(args: argparse.Namespace) -> int: cfg = load_config(args.config) tasks = load_tasks(args.tasks) @@ -400,18 +495,7 @@ def _cmd_bench(args: argparse.Namespace) -> int: return 0 # One resolved mode per provider in the matrix, so a mixed pass bills each side correctly. - providers = {provider_for_model(item.model) for item in todo} - auth_modes = {provider: resolve_auth_mode(args.auth, provider=provider) for provider in providers} - for provider in sorted(providers): - check_agent_cli(provider) - _print_auth(auth_modes[provider], provider) - _warn_codex_accounting(provider) - if "session" in auth_modes.values(): - print( - "note: cost_usd for session-billed runs is what they would have cost at API " - "rates, not metered spend; each run records its auth_mode", - file=sys.stderr, - ) + auth_modes = _resolve_bench_auth({item.model for item in todo}, args.auth) # Before the target is built and before any agent runs: an unreachable pricing page # must cost nothing, and a pass must never be priced by a table it cannot date. try: @@ -430,44 +514,23 @@ def _cmd_bench(args: argparse.Namespace) -> int: target = prepare_target(cfg, args.cache, refresh=args.refresh_target) print(f"target ready: {target.fingerprint} @ {target.commit[:8]} (venv {target.venv_dir})", flush=True) - # Arms run one after another: every run in a matrix shares one skill, and a sequential - # pass keeps each arm's tally readable while the progress counter spans the whole thing. running = [arm for arm in arms if arm.todo] print(f"running {len(todo)} runs, up to {cfg.max_concurrency} at a time:", flush=True) progress = _Progress(len(todo)) - collected: list[RunOutcome] = [] try: - for arm in running: - if len(running) > 1: - print(f"\n=== arm {arm.name}: {len(arm.todo)} runs ===", flush=True) - started = time.monotonic() - outcomes = asyncio.run( - run_matrix( - arm.todo, - target=target, - runs_root=args.runs, - max_concurrency=cfg.max_concurrency, - auth_modes=auth_modes, - skill=arm.skill, - skill_name=cfg.skill_name, - keep_sandbox=args.keep_sandboxes, - stderr=StderrFilter(), - on_start=progress.on_start, - on_done=progress.on_done, - env_passthrough=cfg.env_passthrough, - prices=prices, - ) - ) - collected.extend(outcomes) - _print_run_summary(outcomes, time.monotonic() - started, label=arm.name if len(running) > 1 else "") - _print_skill_loading(outcomes, arm, cfg.skill_name) + collected = _execute_arms( + arms, + cfg=cfg, + target=target, + runs_root=args.runs, + auth_modes=auth_modes, + prices=prices, + keep_sandboxes=args.keep_sandboxes, + progress=progress, + ) except BenchmarkInvalidError as err: print(f"\nerror: {err}", file=sys.stderr) - print( - "Fix or replenish that credential, then rerun the same command; invalid and " - "cancelled cells remain pending.", - file=sys.stderr, - ) + _invalid_bench_note() return 2 if len(running) > 1: @@ -476,20 +539,37 @@ def _cmd_bench(args: argparse.Namespace) -> int: return 0 -def _cmd_draft(args: argparse.Namespace) -> int: +def _cmd_improve(args: argparse.Namespace) -> int: cfg = load_config(args.config) + tasks = load_tasks(args.tasks) if args.model: cfg = replace(cfg, meta_model=args.model) - existing = available_versions(args.skills) - if existing and not args.force: + # Parent is an explicit --from, else the latest version, else None (create the first skill + # from the noskill wiki). improve_skill resolves and validates the parent itself. + parent = args.from_version or latest_version(args.skills) + parent_skill = None if parent is None else load_skill(args.skills, parent, expect_name=cfg.skill_name) + # Fail before the costly target prep if there is nothing to learn from: the improver needs the + # parent arm's train runs (and the wiki built from them). + parent_arm = arm_name(parent) + if not collect_arm_runs(args.runs, parent_arm, tasks, split="train"): + hint = ( + "acumen bench --no-skill --split train" + if parent is None + else f"acumen bench --skill {parent} --split train" + ) print( - f"skills already exist ({', '.join(existing)}) — drafting would add " - f"another version. Pass --force to draft anyway, or use `acumen improve` " - f"to build on {existing[-1]}.", + f"no train-split runs found for {parent_arm} under {args.runs / parent_arm / 'train'} — " + f"run `{hint}` and `acumen wiki` first", file=sys.stderr, ) return 2 + if parent_skill is None: + print(f"no skill versions under {args.skills} — creating the first skill from the wiki with {cfg.meta_model}") + else: + print( + f"improving {parent_skill.version} ({parent_skill.name}, {parent_skill.hash[:19]}…) with {cfg.meta_model}" + ) provider = provider_for_model(cfg.meta_model) check_agent_cli(provider) @@ -499,55 +579,116 @@ def _cmd_draft(args: argparse.Namespace) -> int: print(f"preparing target {cfg.repo}@{cfg.ref} ...", flush=True) target = prepare_target(cfg, args.cache, refresh=args.refresh_target) print(f"target ready: {target.fingerprint} @ {target.commit[:8]}", flush=True) - print(f"drafting with {cfg.meta_model} (this reads the package source) ...", flush=True) - log = LiveLog.open(args.log_dir, "draft", stream=args.stream) + log = LiveLog.open(args.log_dir, "improve", stream=args.stream) print(f"log → {log.jsonl_path}", flush=True) with log: result = asyncio.run( - draft_skill( + improve_skill( cfg=cfg, prices=_agent_prices(cfg, model=cfg.meta_model), target=target, skills_root=args.skills, + runs_root=args.runs, + wiki_root=args.wiki, + tasks=tasks, auth_mode=auth_mode, + parent_version=parent, max_turns=args.max_turns, max_usd=args.max_usd, feedback=args.feedback, log=log, ) ) - skill = result.skill - files = sorted(p.relative_to(skill.directory).as_posix() for p in skill.directory.rglob("*") if p.is_file()) - print(f"\nwrote {skill.directory}") - print(f" name: {skill.name}") - print(f" description: {skill.description}") - print(f" hash: {skill.hash}") + new = result.skill + files = sorted(p.relative_to(new.directory).as_posix() for p in new.directory.rglob("*") if p.is_file()) + parent_label = result.parent or "noskill" + print(f"\nwrote {new.directory} (parent {parent_label})") + print(f" name: {new.name}") + print(f" description: {new.description}") + print(f" hash: {new.hash}") print(f" files: {', '.join(files)}") + print(f" evidence: {result.n_train_runs} train runs ({result.n_train_failures} failing)") print(f" cost: {_fmt_cost(result.cost_usd)} over {result.turns} turns") + if parent_skill is not None and new.hash == parent_skill.hash: + print( + "warning: the new version is byte-identical to its parent — the improver changed nothing", + file=sys.stderr, + ) _print_log_result(log) - print(f"\nnext: acumen bench --skill {skill.version}") + print(f"\nnext: acumen bench --skill {new.version} && acumen report") return 0 -def _cmd_improve(args: argparse.Namespace) -> int: +def _wiki_version(args: argparse.Namespace) -> str: + """Resolve the version label a wiki update records, from ``--skill``/``--no-skill``/default.""" + if args.skill is not None: + return args.skill if args.skill.startswith("v") else f"v{args.skill}" + if args.no_skill: + return arm_name(None) + return latest_version(args.skills) or arm_name(None) + + +def _print_wiki_task(result) -> None: + """Print one task's wiki-update outcome as it lands, with any brevity warnings.""" + print(f" wiki [{result.version}] {result.task_id}: {_fmt_cost(result.cost_usd)} over {result.turns} turns") + for warning in result.warnings: + print(f" warning: {warning}", file=sys.stderr) + + +def _run_wiki( + *, + cfg: Config, + tasks: Sequence[Task], + target, + version: str, + runs_root: Path, + wiki_root: Path, + skills_root: Path, + prices: PriceTable, + auth_mode: AuthMode, + log_dir: Path, + stream: bool, +): + """Update the wiki for one arm and print a per-task tally; returns the task results.""" + print(f"updating wiki for [{version}] with {cfg.meta_model} (one agent per task) ...", flush=True) + results = asyncio.run( + update_wiki( + cfg=cfg, + target=target, + runs_root=runs_root, + wiki_root=wiki_root, + skills_root=skills_root, + tasks=tasks, + version=version, + prices=prices, + auth_mode=auth_mode, + max_concurrency=cfg.max_concurrency, + log_dir=log_dir, + stream=stream, + on_task_done=_print_wiki_task, + ) + ) + if not results: + print(f" wiki already had [{version}] for every task — nothing to do") + else: + total = sum(r.cost_usd or 0.0 for r in results) + print(f"wiki: updated {len(results)} task(s) for [{version}] ({_fmt_cost(total)})") + return results + + +def _cmd_wiki(args: argparse.Namespace) -> int: cfg = load_config(args.config) tasks = load_tasks(args.tasks) + if args.max_concurrency: + cfg = replace(cfg, max_concurrency=args.max_concurrency) if args.model: cfg = replace(cfg, meta_model=args.model) - versions = available_versions(args.skills) - if not versions: - print( - f"no skill versions under {args.skills} — run `acumen draft` first, then bench it", - file=sys.stderr, - ) - return 2 - parent = args.from_version or latest_version(args.skills) - # Immutability guard: the improved version is always the next unused directory, - # so an existing version is never in the write path. Say the parent plainly up front. - skill = load_skill(args.skills, parent, expect_name=cfg.skill_name) - print(f"improving {skill.version} ({skill.name}, {skill.hash[:19]}…) with {cfg.meta_model}") + version = _wiki_version(args) + if version != arm_name(None): + # Validate a named skill version exists before the costly target prep. + load_skill(args.skills, version, expect_name=cfg.skill_name) provider = provider_for_model(cfg.meta_model) check_agent_cli(provider) @@ -558,41 +699,154 @@ def _cmd_improve(args: argparse.Namespace) -> int: target = prepare_target(cfg, args.cache, refresh=args.refresh_target) print(f"target ready: {target.fingerprint} @ {target.commit[:8]}", flush=True) - log = LiveLog.open(args.log_dir, "improve", stream=args.stream) - print(f"log → {log.jsonl_path}", flush=True) - with log: - result = asyncio.run( - improve_skill( - cfg=cfg, - prices=_agent_prices(cfg, model=cfg.meta_model), - target=target, - skills_root=args.skills, - runs_root=args.runs, - tasks=tasks, - auth_mode=auth_mode, - parent_version=parent, - max_turns=args.max_turns, - max_usd=args.max_usd, - feedback=args.feedback, - log=log, - ) + _run_wiki( + cfg=cfg, + tasks=tasks, + target=target, + version=version, + runs_root=args.runs, + wiki_root=args.wiki, + skills_root=args.skills, + prices=_agent_prices(cfg, model=cfg.meta_model), + auth_mode=auth_mode, + log_dir=args.log_dir, + stream=args.stream, + ) + print(f"\nwiki written to {args.wiki.resolve()}") + return 0 + + +def _cmd_epoch(args: argparse.Namespace) -> int: + cfg = load_config(args.config) + tasks = load_tasks(args.tasks) + if args.max_concurrency: + cfg = replace(cfg, max_concurrency=args.max_concurrency) + if args.replicates: + cfg = replace(cfg, n_replicates=args.replicates) + if args.model: + cfg = replace(cfg, meta_model=args.model) + + runs_root = args.runs + + def valid_complete(version: str) -> bool: + planned = build_matrix(cfg, tasks, skill=version, splits=["valid"]) + return not pending(planned, runs_root, resume=True) + + plan = resolve_epoch(args.skills, valid_complete=valid_complete) + parent_label = plan.parent_version or arm_name(None) + wiki_version = parent_label + tail = " (resuming)" if plan.resumed else "" + print(f"epoch: learning from [{parent_label}] → producing {plan.new_version}{tail}") + + # Auth for every provider the epoch touches: the benchmark models and the meta model. + auth_modes = _resolve_bench_auth(set(cfg.models) | {cfg.meta_model}, args.auth) + meta_auth = auth_modes[provider_for_model(cfg.meta_model)] + # Benchmark runs freeze their cost, so a pass must establish rates before it spends anything. + try: + prices = _bench_prices(cfg) + except PriceFeedError as err: + print(f"error: {err}", file=sys.stderr) + print("Retry when the pricing pages are reachable, or pin rates in config.yaml.", file=sys.stderr) + return 2 + _warn_unpriced(set(cfg.models) | {cfg.meta_model}, prices) + print(f"preparing target {cfg.repo}@{cfg.ref} ...", flush=True) + target = prepare_target(cfg, args.cache, refresh=args.refresh_target) + print(f"target ready: {target.fingerprint} @ {target.commit[:8]} (venv {target.venv_dir})", flush=True) + + # Step 1 — bench the parent arm on the training signal. The first epoch also benches the + # noskill baseline on valid, so the report has an in-epoch baseline to compare against. + if plan.first: + train_specs: list[tuple[str | None, Sequence[Split]]] = [(None, ["train", "valid"])] + else: + train_specs = [(plan.parent_version, ["train"])] + print(f"\n[1/4] benchmarking [{parent_label}] on the training signal ...", flush=True) + arms = _build_arms(train_specs, cfg=cfg, tasks=tasks, runs_root=runs_root, skills_root=args.skills) + _print_plan(arms) + try: + _execute_arms( + arms, + cfg=cfg, + target=target, + runs_root=runs_root, + auth_modes=auth_modes, + prices=prices, + keep_sandboxes=False, ) - new = result.skill - files = sorted(p.relative_to(new.directory).as_posix() for p in new.directory.rglob("*") if p.is_file()) - print(f"\nwrote {new.directory} (parent {result.parent})") - print(f" name: {new.name}") - print(f" description: {new.description}") - print(f" hash: {new.hash}") - print(f" files: {', '.join(files)}") - print(f" evidence: {result.n_train_runs} train runs ({result.n_train_failures} failing)") - print(f" cost: {_fmt_cost(result.cost_usd)} over {result.turns} turns") - if new.hash == skill.hash: - print( - "warning: the new version is byte-identical to its parent — the improver changed nothing", - file=sys.stderr, + except BenchmarkInvalidError as err: + print(f"\nerror: {err}", file=sys.stderr) + _invalid_bench_note() + return 2 + + # Step 2 — distil that arm into the wiki (idempotent: recorded arms are skipped). + print(f"\n[2/4] updating the wiki for [{wiki_version}] ...", flush=True) + _run_wiki( + cfg=cfg, + tasks=tasks, + target=target, + version=wiki_version, + runs_root=runs_root, + wiki_root=args.wiki, + skills_root=args.skills, + prices=prices, + auth_mode=meta_auth, + log_dir=args.log_dir, + stream=args.stream, + ) + + # Step 3 — create or improve the skill (skipped when the version already exists: a resumed + # epoch that crashed after improve). + if skill_dir(args.skills, plan.new_version).exists(): + print(f"\n[3/4] {plan.new_version} already exists — skipping improve (resumed epoch)") + else: + verb = "creating" if plan.first else "improving" + print(f"\n[3/4] {verb} the skill → {plan.new_version} with {cfg.meta_model} ...", flush=True) + log = LiveLog.open(args.log_dir, "improve", stream=args.stream) + print(f"log → {log.jsonl_path}", flush=True) + with log: + result = asyncio.run( + improve_skill( + cfg=cfg, + prices=prices, + target=target, + skills_root=args.skills, + runs_root=runs_root, + wiki_root=args.wiki, + tasks=tasks, + auth_mode=meta_auth, + parent_version=plan.parent_version, + feedback=args.feedback, + log=log, + ) + ) + new = result.skill + print(f"wrote {new.directory} (parent {result.parent or 'noskill'})") + print(f" description: {new.description}") + print(f" cost: {_fmt_cost(result.cost_usd)} over {result.turns} turns") + _print_log_result(log) + + # Step 4 — bench the new version on the held-out valid signal (completes it if unfinished). + print(f"\n[4/4] benchmarking {plan.new_version} on the held-out valid signal ...", flush=True) + valid_arms = _build_arms( + [(plan.new_version, ["valid"])], cfg=cfg, tasks=tasks, runs_root=runs_root, skills_root=args.skills + ) + _print_plan(valid_arms) + try: + _execute_arms( + valid_arms, + cfg=cfg, + target=target, + runs_root=runs_root, + auth_modes=auth_modes, + prices=prices, + keep_sandboxes=False, ) - _print_log_result(log) - print(f"\nnext: acumen bench --skill {new.version} && acumen report") + except BenchmarkInvalidError as err: + print(f"\nerror: {err}", file=sys.stderr) + _invalid_bench_note() + return 2 + + print(f"\nepoch complete: {plan.new_version} produced and benched on valid.") + print("next: `acumen report` to see it, or `acumen epoch` again for another round") return 0 @@ -668,7 +922,7 @@ def _cmd_tasks(args: argparse.Namespace) -> int: "and split, so it was not kept", file=sys.stderr, ) - print("\nnext: review the tasks, then `acumen check`, `acumen draft` and `acumen bench`") + print("\nnext: review the tasks, then `acumen check`, then `acumen epoch`") return 0 @@ -1082,7 +1336,7 @@ def _cmd_init(args: argparse.Namespace) -> int: written = scaffold(args.directory, force=args.force) for path in written: print(f"wrote {path}") - print("\nnext: edit config.yaml (repo) and tasks.yaml, then `acumen draft`") + print("\nnext: edit config.yaml (repo) and tasks.yaml, then `acumen epoch`") return 0 @@ -1167,25 +1421,18 @@ def build_parser() -> argparse.ArgumentParser: _add_bench_args(bench) bench.set_defaults(func=_cmd_bench) - draft = sub.add_parser("draft", help="draft a skill from the target package's source") - draft.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") - draft.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") - draft.add_argument("--model", help="override config meta_model") - draft.add_argument("--max-turns", type=int, help="cap turns for the drafting agent (default: unbounded)") - draft.add_argument("--max-usd", type=float, help="cap spend for the drafting agent (default: unbounded)") - draft.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") - draft.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") - draft.add_argument("--force", action="store_true", help="draft another version even if some already exist") - _add_auth_arg(draft) - _add_feedback_arg(draft, extra=" (e.g. package context, what the skill should emphasise)") - _add_log_args(draft) - draft.set_defaults(func=_cmd_draft) - - improve = sub.add_parser("improve", help="improve the current skill into a new version from its train results") + improve = sub.add_parser( + "improve", + help="create or improve the skill from the knowledge wiki", + description="Read the knowledge wiki (per-task observations/hypotheses distilled from train " + "runs) and the filtered package source, then create the first skill (when none exist) or " + "improve the latest into the next version. The held-out valid split is never reachable.", + ) improve.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") improve.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") improve.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") improve.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + improve.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") improve.add_argument("--from", dest="from_version", metavar="VERSION", help="version to improve (default: latest)") improve.add_argument("--model", help="override config meta_model") improve.add_argument("--max-turns", type=int, help="cap turns for the improving agent (default: unbounded)") @@ -1195,11 +1442,57 @@ def build_parser() -> argparse.ArgumentParser: _add_auth_arg(improve) _add_feedback_arg( improve, - extra=" (e.g. what to fix or emphasise; do NOT paste test-split answers — that defeats the held-out split)", + extra=" (e.g. what to fix or emphasise; do NOT paste valid-split answers — that defeats the held-out split)", ) _add_log_args(improve) improve.set_defaults(func=_cmd_improve) + wiki = sub.add_parser( + "wiki", + help="distil an arm's train runs into the knowledge wiki (one agent per task)", + description="For each task, read that arm's train-split runs across models and replicates " + "and append a terse [version][model] block to wiki//observations.md and hypothesis.md. " + "Cumulative and idempotent: an arm already recorded for a task is skipped.", + ) + wiki.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") + wiki.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") + wiki.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + wiki.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") + wiki.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") + wiki_arm = wiki.add_mutually_exclusive_group() + wiki_arm.add_argument("--no-skill", action="store_true", help="record the baseline (noskill) arm") + wiki_arm.add_argument("--skill", metavar="VERSION", help="record one skill version, e.g. v1 (default: latest)") + wiki.add_argument("--model", help="override config meta_model") + wiki.add_argument("--max-concurrency", type=int, help="override config max_concurrency") + wiki.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") + wiki.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") + _add_auth_arg(wiki) + _add_log_args(wiki) + wiki.set_defaults(func=_cmd_wiki) + + epoch = sub.add_parser( + "epoch", + help="run one training epoch: bench train, update wiki, improve, bench valid", + description="One training epoch end to end: bench the current arm on the training signal, " + "distil it into the wiki, create/improve the skill, then bench the new version on the " + "held-out valid signal. Fully resumable — re-run to continue a crashed epoch or start the " + "next one.", + ) + epoch.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") + epoch.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") + epoch.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + epoch.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") + epoch.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") + epoch.add_argument("--model", help="override config meta_model") + epoch.add_argument("--max-concurrency", type=int, help="override config max_concurrency") + epoch.add_argument("--replicates", type=int, help="override config n_replicates") + epoch.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") + epoch.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") + _add_auth_arg(epoch) + _add_feedback_arg(epoch, extra=" (passed to the improver; do NOT paste valid-split answers)") + _add_log_args(epoch) + epoch.set_defaults(func=_cmd_epoch) + tasks_cmd = sub.add_parser("tasks", help="autonomously generate a tasks.yaml from the target package") tasks_cmd.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") tasks_cmd.add_argument("--out", type=Path, default=Path("tasks.yaml"), help="tasks.yaml to write") @@ -1325,8 +1618,8 @@ def main(argv: list[str] | None = None) -> int: AgentError, PriceFeedError, SkillError, - DraftError, ImproveError, + WikiError, TaskGenError, ReviewError, ShipError, diff --git a/src/acumen/draft.py b/src/acumen/draft.py deleted file mode 100644 index b77db42..0000000 --- a/src/acumen/draft.py +++ /dev/null @@ -1,235 +0,0 @@ -"""The drafting agent: write ``skills/v1/`` from the target package's source. - -Unlike a benchmark agent, the drafter reads the target's source — it is documenting -the package, so it needs to see it. It is otherwise held to the same isolation: scrubbed -env, throwaway config dir, no user settings or memories. - -What it does *not* see is agent guidance the target already ships. A skill drafted from the -maintainer's own skill is not the independent artifact the benchmark then reports on: the arms -would be comparing that skill, re-served, against no skill at all. So the drafter reads a -filtered copy of the checkout with skills and agent-instruction files stripped -(:func:`acumen.scrub.build_filtered_source`) and is denied the original tree, exactly as -``tasks`` is. The real checkout is never modified. - -The agent writes into a staging directory, not into ``skills/`` directly. Only a skill -that loads and validates is promoted to a version, so a failed or half-finished draft -never leaves a broken ``skills/vN/`` behind — versions are immutable, which means -they must be right the moment they appear. -""" - -from __future__ import annotations - -import shutil -import tempfile -from dataclasses import dataclass -from pathlib import Path - -from acumen.agents import AgentOptions, AgentResult, provider_for_model, run_agent -from acumen.config import Config -from acumen.env import AuthMode, Target, build_agent_env -from acumen.logs import LiveLog -from acumen.prices import PriceTable, price_usage, pricer, resolve_cost -from acumen.procs import label_env, reap -from acumen.prompts import draft_prompt -from acumen.scrub import build_filtered_source, make_skill_guard -from acumen.skills import ( - SKILL_FILE, - Skill, - SkillError, - load_skill, - next_version, - skill_dir, - write_meta, -) - - -class DraftError(RuntimeError): - """Raised when a skill could not be drafted.""" - - -@dataclass(frozen=True) -class DraftResult: - """A drafted skill version and what it cost to produce.""" - - skill: Skill - cost_usd: float | None - turns: int - #: Live log paths for this run, when a :class:`LiveLog` was attached. - log_jsonl: Path | None = None - log_html: Path | None = None - - -def _validate_staged(staging: Path, skill_name: str) -> None: - """Fail loudly if the agent's output isn't a usable skill, before it becomes a version.""" - if not (staging / SKILL_FILE).is_file(): - raise DraftError( - f"the drafting agent did not write {SKILL_FILE} — nothing to promote. Inspect the run log or the prompt." - ) - # load_skill enforces the frontmatter contract (name matches, description present). - try: - load_skill(staging.parent, staging.name, expect_name=skill_name) - except SkillError as err: - raise DraftError(f"the drafted skill is not valid: {err}") from err - - -async def draft_skill( - *, - cfg: Config, - target: Target, - skills_root: Path, - auth_mode: AuthMode = "session", - prices: PriceTable | None = None, - model: str | None = None, - max_turns: int | None = None, - max_usd: float | None = None, - rationale: str = "initial draft", - feedback: str | None = None, - log: LiveLog | None = None, -) -> DraftResult: - """Draft a new skill version from the target package's source. - - Parameters - ---------- - cfg - The pass config; supplies ``skill_name`` and ``meta_model``. - target - The prepared target, supplying the source checkout and interpreter. - skills_root - The ``skills/`` root. The new version is the next unused one. - auth_mode - Which credential the drafting agent authenticates with — ``"session"`` (the Claude - subscription) or ``"api"`` (see :func:`acumen.env.build_agent_env`). - model - Override for the drafting model; defaults to the config's. - max_turns, max_usd - Turn and budget caps for the drafting agent. **Unbounded by default** — the config's - ``max_turns``/``max_usd`` cap benchmark agents only; pass explicit caps to bound the - drafter. - rationale - Recorded in ``meta.json`` as why this version exists. - feedback - Optional maintainer guidance, injected into the draft prompt as a subordinated block - and recorded in ``meta.json`` as provenance. - log - A :class:`LiveLog` to stream the agent's messages to and render an HTML log from. - - Returns - ------- - The drafted skill, loaded and validated. - """ - version = next_version(skills_root) - dest = skill_dir(skills_root, version) - if dest.exists(): - raise DraftError(f"{dest} already exists — skill versions are immutable and never overwritten") - - holder = Path(tempfile.mkdtemp(prefix="acumen-draft-")) - try: - # The staging dir is named for the version so load_skill can validate it in place. - work = holder / "work" - staging = work / version - home = holder / "home" - selected_model = model or cfg.meta_model - table = prices if prices is not None else PriceTable(overrides=cfg.prices) - provider = provider_for_model(selected_model) - config_dir = home / (".claude" if provider == "claude" else ".codex") - for path in (staging, home, config_dir, home / "tmp"): - path.mkdir(parents=True, exist_ok=True) - - # Marks the agent's processes so the teardown below can find what it leaves running. - env = label_env( - build_agent_env( - config_dir=config_dir, - home=home, - extra_path=[target.bin_dir], - auth_mode=auth_mode, - extra_allow=cfg.env_passthrough, - provider=provider, - ), - holder, - ) - - # A copy of the source with the target's own skills and agent guidance stripped. The - # drafter reads this, never the real checkout. - source_copy = build_filtered_source(target.src_dir, holder / "source") - - prompt = draft_prompt( - package=target.pkg_name, - version=target.pkg_version, - src=source_copy, - python=target.python, - out=staging, - skill_name=cfg.skill_name, - feedback=feedback, - ) - options = AgentOptions( - cwd=work, - env=env, - model=selected_model, - # No default turn or budget cap: only bound the agent if the caller asked. The - # config's ``max_turns``/``max_usd`` cap benchmark agents only. - max_turns=max_turns, - max_usd=max_usd, - # Codex reports no billed figure, so a budget cap needs the run's own rate table. - price_usd=pricer(selected_model, table), - # The drafter reads the target source; benchmark agents never do. It points at the - # *filtered* copy, not the real checkout. - read_dirs=(source_copy, target.venv_dir), - write_dirs=(work,), - discover_skills=True, - # Belt-and-braces over the filtered copy: deny any call that reaches an existing - # skill/guidance artifact or the original unfiltered source, wherever pointed. Built - # only for Claude — the hook is an SDK object, and Codex gets ``deny_paths`` below. - claude_hooks={"PreToolUse": [make_skill_guard(target.src_dir)]} if provider == "claude" else None, - # Codex reads the filtered copy and is denied the original checkout. - deny_paths=(target.src_dir.resolve(),), - ) - - result: AgentResult | None = None - agent_error: Exception | None = None - try: - result = await run_agent( - prompt, - options=options, - on_event=log.append if log is not None else None, - ) - except Exception as err: # noqa: BLE001 - a failed draft is an error to report, re-raised below - agent_error = err - finally: - # Render the HTML log while the throwaway config dir still holds the native - # transcript — in a finally so an aborted run (the SDK raises on a cap breach, - # after yielding the result) is still inspectable. - if log is not None: - log.finalize(config_dir=config_dir, work_dir=work, result=result) - - if agent_error is not None: - raise DraftError(f"the drafting agent failed: {type(agent_error).__name__}: {agent_error}") from agent_error - if result is None: - raise DraftError("the drafting agent produced no result message") - if result.is_error: - raise DraftError(f"the drafting agent errored: {result.subtype} {result.errors or ''}".strip()) - - _validate_staged(staging, cfg.skill_name) - - dest.parent.mkdir(parents=True, exist_ok=True) - shutil.copytree(staging, dest) - write_meta(dest, parent=None, rationale=rationale, feedback=feedback) - skill = load_skill(skills_root, version, expect_name=cfg.skill_name) - return DraftResult( - skill=skill, - cost_usd=resolve_cost( - result.total_cost_usd, - price_usage( - result.usage, - model=selected_model, - provider=result.provider, - prices=table, - ), - ).cost_usd, - turns=result.num_turns, - log_jsonl=log.jsonl_path if log is not None else None, - log_html=log.html_path if log is not None and log.html_rendered else None, - ) - finally: - # Kill anything the agent left running before removing the directory it runs in. - reap(holder) - shutil.rmtree(holder, ignore_errors=True) diff --git a/src/acumen/epoch.py b/src/acumen/epoch.py new file mode 100644 index 0000000..8fa64c3 --- /dev/null +++ b/src/acumen/epoch.py @@ -0,0 +1,63 @@ +"""Resolve which training epoch an ``acumen epoch`` invocation is on, from on-disk state. + +An epoch benches the current arm on ``train``, distils it into the wiki, produces the next skill +version, and benches that version on ``valid``. The resolver here decides — purely from what is on +disk — whether a fresh invocation *starts* a new epoch or *resumes* one that crashed partway, so a +crash between "improve wrote the version" and "valid bench finished" re-enters the SAME epoch and +completes it rather than starting another. +""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass + +from acumen.skills import available_versions, next_version + + +@dataclass(frozen=True) +class EpochPlan: + """What one ``acumen epoch`` invocation will do, resolved from state.""" + + #: The arm learned from this epoch: a skill version, or ``None`` for the ``noskill`` baseline. + parent_version: str | None + #: The version this epoch produces, e.g. ``"v1"`` or ``"v2"``. + new_version: str + #: Whether this is the first epoch (``parent_version is None``), which also benches the + #: ``noskill`` baseline on ``valid`` for the report. + first: bool + #: Whether ``new_version`` already exists on disk (a resumed epoch), so ``improve`` is skipped. + resumed: bool + + +def resolve_epoch(skills_root, *, valid_complete: Callable[[str], bool]) -> EpochPlan: + """Decide the epoch to run from the skills on disk and each version's valid-bench state. + + Parameters + ---------- + skills_root + The ``skills/`` root. + valid_complete + Predicate: does this version have a complete ``valid`` bench? Injected so the resolver + stays pure and unit-testable (the caller builds it from ``build_matrix``/``pending``). + + Returns + ------- + The resolved :class:`EpochPlan`. + + Notes + ----- + - No versions yet → first epoch: ``parent=None`` (noskill), ``new=v1``. + - Latest version exists but its valid bench is unfinished → resume that epoch: ``new=latest``, + ``parent`` is the version below it (or ``None`` when latest is ``v1``), ``resumed=True``. + - Otherwise → fresh epoch: ``parent=latest``, ``new=next``. + """ + versions = available_versions(skills_root) + latest = versions[-1] if versions else None + + if latest is not None and not valid_complete(latest): + parent = versions[-2] if len(versions) >= 2 else None + return EpochPlan(parent_version=parent, new_version=latest, first=parent is None, resumed=True) + + new = next_version(skills_root) + return EpochPlan(parent_version=latest, new_version=new, first=latest is None, resumed=False) diff --git a/src/acumen/improve.py b/src/acumen/improve.py index 23ee1b0..3dbadbd 100644 --- a/src/acumen/improve.py +++ b/src/acumen/improve.py @@ -1,26 +1,25 @@ -"""The improving agent: read train evidence, write ``skills/v{n+1}/``. - -Where the drafter reads the package *source* to write the first skill, the improver -reads how the current skill *performed* — on the TRAIN split only — and edits it. Versions -are immutable: ``improve`` always writes a new directory, never mutates an existing one. - -The load-bearing constraint is that the improver must never see test results, or the whole -benchmark is worthless. This is enforced two ways, not one: - -1. **Structurally.** The agent's ``cwd`` is a throwaway work dir and its readable material - is a *curated copy* of train-split runs. The real ``runs/`` tree is not on its ``cwd``, - not in the read-only roots, and not in scope — so there is no test directory to reach in the - first place. -2. **Belt-and-braces.** A ``PreToolUse`` hook denies any tool call whose path resolves under - a ``runs/*/test/`` subtree, so even a call that names an absolute path back into the - project runs tree is refused. See :func:`find_test_access`. - -Prompt-level instruction alone is explicitly *not* sufficient. +"""The improving agent: read the knowledge wiki, write the next ``skills/vN/``. + +The improver no longer reads a raw dump of the parent skill's runs. It reads the **wiki** — the +distilled, per-task ``observations.md``/``hypothesis.md`` notes that :mod:`acumen.wiki` builds from +train-split runs and accumulates across versions — plus the parent skill (in improve mode), the +raw train transcripts (for drill-down), and the *filtered* package source. On the first epoch there +is no parent skill: the wiki holds only the ``noskill`` baseline and the improver **creates** v1. + +Two load-bearing isolations, both structural rather than by prompt: + +1. **No held-out (valid) results.** The improver's readable material is the wiki + staged train + transcripts — the real ``runs/`` tree is denied wholesale, and a ``PreToolUse`` hook refuses any + call that resolves under ``runs/*/valid/`` wherever it is pointed (see :func:`find_valid_access`). +2. **No skill bias from the target.** A package may already ship its own skill/agent guidance; the + improver reads the *filtered* source (:func:`acumen.scrub.build_filtered_source`) with those + artifacts stripped, and the raw checkout denied, so the optimization never re-serves a skill the + package already carries. This matters precisely because acumen's job is to ship skills into + packages. """ from __future__ import annotations -import json import shutil import tempfile from dataclasses import dataclass @@ -36,31 +35,31 @@ from acumen.logs import LiveLog from acumen.paths import ( ANSWER_FILE, - RESULT_FILE, SCRIPT_FILE, TRANSCRIPT_HTML, - TRANSCRIPT_JSONL, arm_name, - parse_run_dir, ) from acumen.prices import PriceTable, price_usage, pricer, resolve_cost from acumen.procs import label_env, reap from acumen.prompts import improve_prompt +from acumen.scrub import build_filtered_source, make_skill_guard from acumen.skills import ( SKILL_FILE, Skill, SkillError, content_files, + latest_version, load_skill, next_version, skill_dir, write_meta, ) from acumen.tasks import Task +from acumen.wiki import WIKI_DIRNAME, collect_arm_runs -#: The subtree component that marks a held-out split. A path under ``runs//test/…`` is +#: The subtree component that marks the held-out split. A path under ``runs//valid/…`` is #: what the improver must never reach. -_TEST_SPLIT = "test" +_VALID_SPLIT = "valid" #: tool_input keys that carry a filesystem path across the default Claude Code toolset. _PATH_KEYS = ("file_path", "path", "notebook_path", "filename") @@ -75,35 +74,13 @@ class ImproveError(RuntimeError): """Raised when a skill could not be improved.""" -@dataclass(frozen=True) -class TrainRun: - """One train-split benchmark run of the parent skill, as evidence for the improver. - - ``success`` and ``skill_loaded`` are independent outcomes with independent fixes: a run - that never loaded the skill says nothing about the skill's body, only about its - ``description``. Both reach the improver so it can tell the two apart. - """ - - task_id: str - model: str - rep: int - prompt: str - expected: str - answer: str | None - reason: str - success: bool - directory: Path - #: Whether the agent actually invoked the skill. ``None`` when the transcript could not - #: be read — undetermined, which is not the same as "did not load". - skill_loaded: bool | None = None - - @dataclass(frozen=True) class ImproveResult: - """A newly improved skill version and what it cost to produce.""" + """A newly created or improved skill version and what it cost to produce.""" skill: Skill - parent: str + #: The parent version, or ``None`` when this is the first (created) skill. + parent: str | None cost_usd: float | None turns: int n_train_runs: int @@ -113,198 +90,64 @@ class ImproveResult: log_html: Path | None = None -# ── Train evidence ───────────────────────────────────────────────────────────────────── - - -def collect_train_runs(runs_root: Path, arm: str, tasks: list[Task]) -> list[TrainRun]: - """Collect the parent arm's train-split runs into evidence for the improver. +# ── Staging ────────────────────────────────────────────────────────────────────────────── - Walks ``runs_root//train`` for ``result.json`` files and pairs each with its task's - train prompt and ground-truth answer. Only train runs are ever collected — the test - subtree is never even visited. - Parameters - ---------- - runs_root - The ``runs/`` root. - arm - The arm whose train runs are the improvement signal, e.g. ``"skill_v1"``. - tasks - The loaded tasks, to recover each run's prompt and expected answer. +def _seed_staging(staging: Path, parent: Skill) -> None: + """Pre-fill the staging dir with the parent skill's content, so the agent edits in place. - Returns - ------- - The train runs, failures first then by task/rep — so a reader sees what went wrong up - front. + Only content files are copied — ``meta.json`` is acumen bookkeeping and is written fresh once + the new version is promoted. """ - train_root = runs_root / arm / "train" - by_id = {task.id: task for task in tasks} - runs: list[TrainRun] = [] - if not train_root.is_dir(): - return runs - for result_path in sorted(train_root.rglob(RESULT_FILE)): - try: - data = json.loads(result_path.read_text()) - except (OSError, ValueError) as err: - raise ImproveError(f"cannot read {result_path}: {err}") from err - if data.get("valid", True) is False: - raise ImproveError( - "cannot improve from an infrastructure-invalid benchmark result: " - f"{result_path} ({data.get('reason')}). Fix the harness — replenish the provider " - "credential, or report the sandbox refusal as a bug if it refused a host — " - "and resume the benchmark first." - ) - key = parse_run_dir(runs_root, result_path.parent) - task = by_id.get(key.task_id) - if task is None: - # A run for a task no longer in tasks.yaml is stale evidence; skip it rather - # than guess a prompt for it. - continue - runs.append( - TrainRun( - task_id=key.task_id, - model=str(data.get("model", key.model)), - rep=key.rep, - prompt=task.train.prompt.strip(), - expected=task.train.answer.strip(), - answer=data.get("answer"), - reason=str(data.get("reason", "")), - success=bool(data.get("success", False)), - directory=result_path.parent, - skill_loaded=_loaded(data.get("skill_loaded")), - ) - ) - runs.sort(key=lambda r: (r.success, r.task_id, r.model, r.rep)) - return runs - - -def _loaded(value: Any) -> bool | None: - """Coerce a ``result.json`` ``skill_loaded`` field, preserving "undetermined" as ``None``.""" - return None if value is None else bool(value) - - -def load_rates(runs: list[TrainRun]) -> dict[str, tuple[int, int, int]]: - """Count how often the skill actually loaded, per model. - - Reported per model rather than pooled because the load rate varies far more across models - than across skill versions — a pooled number invites the improver to chase a model whose - behaviour no wording can reach. + staging.mkdir(parents=True, exist_ok=True) + for src in content_files(parent.directory): + dest = staging / src.relative_to(parent.directory) + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(src, dest) - Parameters - ---------- - runs - The train runs to tally. - Returns - ------- - ``{model: (loaded, total, undetermined)}``, where ``undetermined`` counts runs whose - transcript could not be read and which are therefore excluded from ``loaded``. - """ - rates: dict[str, tuple[int, int, int]] = {} - for run in runs: - loaded, total, unknown = rates.get(run.model, (0, 0, 0)) - rates[run.model] = ( - loaded + (run.skill_loaded is True), - total + 1, - unknown + (run.skill_loaded is None), - ) - return dict(sorted(rates.items())) - - -def _load_mark(loaded: bool | None) -> str: - """The phrase naming a run's load outcome, used identically in SUMMARY.md and run.md.""" - if loaded is None: - return "skill load UNDETERMINED" - return "skill LOADED" if loaded else "skill NOT LOADED" - - -def _load_section(runs: list[TrainRun]) -> list[str]: - """The SUMMARY.md section reporting per-model load rates.""" - rates = load_rates(runs) - lines = [ - "## Did the skill load at all?", - "", - "A run where the skill never loaded is not evidence about the skill's body — the agent", - "never read it. Only the `description` in the frontmatter decides whether a skill loads.", - "", - "| model | loaded | runs | rate |", - "| --- | ---: | ---: | ---: |", - ] - total_loaded = total_runs = total_unknown = 0 - for model, (loaded, total, unknown) in rates.items(): - total_loaded += loaded - total_runs += total - total_unknown += unknown - lines.append(f"| `{model}` | {loaded} | {total} | {loaded / total:.0%} |") - if total_runs: - lines.extend(["", f"**Overall: {total_loaded}/{total_runs} ({total_loaded / total_runs:.0%}).**"]) - if total_unknown: - noun = "run" if total_unknown == 1 else "runs" - lines.append( - f"({total_unknown} {noun} had no readable transcript, so loading is undetermined and is " - "counted as not loaded above.)" - ) - lines.append("") - return lines +def _stage_wiki(wiki_root: Path, dest: Path) -> None: + """Copy the whole wiki into the agent's work dir, so it never reaches back into the project.""" + dest.mkdir(parents=True, exist_ok=True) + if wiki_root.is_dir(): + shutil.copytree(wiki_root, dest, dirs_exist_ok=True) -def _write_material(train_dir: Path, runs: list[TrainRun]) -> None: - """Lay out the curated train evidence the improver reads. +def _stage_transcripts(runs_root: Path, arm: str, tasks: list[Task], dest: Path) -> int: + """Copy the parent arm's train transcripts into the work dir for drill-down. - A ``SUMMARY.md`` digest (per-model load rates, then runs with failures first) plus one - directory per run holding the run's ``script.py``, transcript, and a ``run.md`` stating - prompt/expected/actual/outcome and whether the skill loaded. Copied, not linked, so the - agent never has a path back into the real runs tree. + Returns the number of runs staged. Copies transcript/script/answer per run into + ``dest//__rep_/`` — copies, so there is no path back into the real tree. """ - train_dir.mkdir(parents=True, exist_ok=True) - n_fail = sum(1 for r in runs if not r.success) - noun = "run" if len(runs) == 1 else "runs" - lines = [ - "# Train-split evidence for the current skill", - "", - "Each run below benchmarked the current skill on one train task. Two things were", - "recorded per run, and they fail for different reasons and take different fixes:", - "whether the agent got the answer RIGHT, and whether the agent LOADED the skill at", - "all. Read the load rates first, then the failing runs. Open a run's `run.md`,", - "`script.py`, and `transcript.html` for the detail.", - "", - f"**{len(runs)} {noun}, {n_fail} failing.**", - "", - *_load_section(runs), - "## Runs", - "", - ] + runs = collect_arm_runs(runs_root, arm, tasks, split="train") for run in runs: - slug = f"{run.task_id}__{run.model}__rep_{run.rep}" - mark = "PASS" if run.success else "FAIL" - lines.append( - f"- `{slug}/` — **{mark}** ({run.reason}) — {_load_mark(run.skill_loaded)} — " - f"task `{run.task_id}` on `{run.model}` — expected `{run.expected}`, got `{run.answer!r}`" - ) - run_out = train_dir / slug + run_out = dest / run.task_id / f"{run.model}__rep_{run.rep}" run_out.mkdir(parents=True, exist_ok=True) - (run_out / "run.md").write_text( - f"# {run.task_id} — {mark} ({run.reason})\n\n" - f"- Model: `{run.model}`\n" - f"- Skill: {_load_mark(run.skill_loaded)}\n\n" - f"## Task prompt\n\n{run.prompt}\n\n" - f"## Expected answer\n\n{run.expected}\n\n" - f"## Answer the agent gave\n\n{run.answer!r}\n\n" - f"## Files\n\n- `script.py` — the script the agent wrote\n" - f"- `transcript.html` — the full run transcript\n" - ) - for name in (SCRIPT_FILE, ANSWER_FILE, TRANSCRIPT_HTML, TRANSCRIPT_JSONL): + for name in (TRANSCRIPT_HTML, SCRIPT_FILE, ANSWER_FILE): src = run.directory / name if src.is_file(): shutil.copyfile(src, run_out / name) - (train_dir / "SUMMARY.md").write_text("\n".join(lines) + "\n") + return len(runs) -# ── Test-access guard ────────────────────────────────────────────────────────────────── +def _validate_staged(staging: Path, skill_name: str) -> None: + """Fail loudly if the agent's output isn't a usable skill, before it becomes a version.""" + if not (staging / SKILL_FILE).is_file(): + raise ImproveError( + f"the improving agent left no {SKILL_FILE} in the staging directory — nothing to " + "promote. Inspect the run log or the prompt." + ) + try: + load_skill(staging.parent, staging.name, expect_name=skill_name) + except SkillError as err: + raise ImproveError(f"the improved skill is not valid: {err}") from err + + +# ── Valid-access guard ───────────────────────────────────────────────────────────────────── def _blocks(candidate: str, runs_root: Path) -> bool: - """Return whether ``candidate`` resolves under a ``runs/*/test/`` subtree.""" + """Return whether ``candidate`` resolves under a ``runs/*/valid/`` subtree.""" try: resolved = Path(candidate).expanduser().resolve() except (OSError, RuntimeError, ValueError): @@ -314,17 +157,17 @@ def _blocks(candidate: str, runs_root: Path) -> bool: except ValueError: return False parts = rel.parts - # runs_root//test/... — the split is the second component. - return len(parts) >= 2 and parts[1] == _TEST_SPLIT + # runs_root//valid/... — the split is the second component. + return len(parts) >= 2 and parts[1] == _VALID_SPLIT -def find_test_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path) -> str | None: - """Return the first path in a tool call that reaches held-out test results, else ``None``. +def find_valid_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path) -> str | None: + """Return the first path in a tool call that reaches held-out valid results, else ``None``. - Pure and side-effect free, so the enforcement can be exercised directly without - standing up an agent. Checks the path-bearing tool_input keys, and — for shell tools — - the whitespace/metacharacter-split tokens of the command, since a Bash call can name a - path no structured field would. + Pure and side-effect free, so the enforcement can be exercised directly without standing up an + agent. Checks the path-bearing tool_input keys, and — for shell tools — the whitespace/ + metacharacter-split tokens of the command, since a Bash call can name a path no structured + field would. Parameters ---------- @@ -337,7 +180,7 @@ def find_test_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path Returns ------- - The offending path string, or ``None`` if the call touches no test results. + The offending path string, or ``None`` if the call touches no valid results. """ for key in _PATH_KEYS: value = tool_input.get(key) @@ -352,8 +195,8 @@ def find_test_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path return None -def make_test_guard(runs_root: Path) -> HookMatcher: - """Build the ``PreToolUse`` hook that denies any access to held-out test results. +def make_valid_guard(runs_root: Path) -> HookMatcher: + """Build the ``PreToolUse`` hook that denies any access to held-out valid results. ``matcher=None`` fires the hook for every tool. The hook resolves paths against the real project ``runs/`` root, so it holds regardless of the agent's ``cwd``. @@ -365,7 +208,7 @@ def make_test_guard(runs_root: Path) -> HookMatcher: root = runs_root.resolve() async def guard(input_data: dict[str, Any], tool_use_id: str | None, context: Any) -> dict[str, Any]: - hit = find_test_access( + hit = find_valid_access( input_data.get("tool_name", ""), input_data.get("tool_input", {}) or {}, root, @@ -377,7 +220,7 @@ async def guard(input_data: dict[str, Any], tool_use_id: str | None, context: An "hookEventName": "PreToolUse", "permissionDecision": "deny", "permissionDecisionReason": ( - f"acumen blocks the improver from reading held-out test results ({hit}). " + f"acumen blocks the improver from reading held-out valid results ({hit}). " "Only train-split evidence is available to you." ), } @@ -386,33 +229,7 @@ async def guard(input_data: dict[str, Any], tool_use_id: str | None, context: An return HookMatcher(matcher=None, hooks=[guard]) -# ── Orchestration ────────────────────────────────────────────────────────────────────── - - -def _seed_staging(staging: Path, parent: Skill) -> None: - """Pre-fill the staging dir with the parent skill's content, so the agent edits in place. - - Only content files are copied — ``meta.json`` is acumen bookkeeping and is written - fresh once the new version is promoted. - """ - staging.mkdir(parents=True, exist_ok=True) - for src in content_files(parent.directory): - dest = staging / src.relative_to(parent.directory) - dest.parent.mkdir(parents=True, exist_ok=True) - shutil.copyfile(src, dest) - - -def _validate_staged(staging: Path, skill_name: str) -> None: - """Fail loudly if the agent's output isn't a usable skill, before it becomes a version.""" - if not (staging / SKILL_FILE).is_file(): - raise ImproveError( - f"the improving agent left no {SKILL_FILE} in the staging directory — nothing to " - "promote. Inspect the run log or the prompt." - ) - try: - load_skill(staging.parent, staging.name, expect_name=skill_name) - except SkillError as err: - raise ImproveError(f"the improved skill is not valid: {err}") from err +# ── Orchestration ────────────────────────────────────────────────────────────────────────── async def improve_skill( @@ -421,6 +238,7 @@ async def improve_skill( target: Target, skills_root: Path, runs_root: Path, + wiki_root: Path, tasks: list[Task], auth_mode: AuthMode = "session", prices: PriceTable | None = None, @@ -431,55 +249,67 @@ async def improve_skill( feedback: str | None = None, log: LiveLog | None = None, ) -> ImproveResult: - """Improve the current skill into the next version from its train-split evidence. + """Create or improve a skill from the knowledge wiki. + + When ``parent_version`` is ``None`` and no skill versions exist, the wiki holds only the + ``noskill`` baseline and the agent CREATES v1 from it + the filtered source. Otherwise it + improves ``parent_version`` (default: latest) into the next version, editing a pre-filled copy. Parameters ---------- cfg The pass config; supplies ``skill_name`` and ``meta_model``. target - The prepared target, supplying the interpreter (for verifying claims). Its source - is **not** exposed — the improver works from evidence, not code. + The prepared target. Its source is exposed only as a FILTERED copy (bundled skills + stripped); the interpreter is available for verifying claims. skills_root The ``skills/`` root; the new version is the next unused one. runs_root - The ``runs/`` root, read for train evidence and guarded against test access. + The ``runs/`` root, read for train transcripts and guarded against valid access. + wiki_root + The ``wiki/`` root, the improver's primary distilled signal. tasks - The loaded tasks, to pair runs with their prompts and expected answers. + The loaded tasks, to locate the parent arm's train transcripts. auth_mode - Which credential the improving agent authenticates with — ``"session"`` (the Claude - subscription) or ``"api"`` (see :func:`acumen.env.build_agent_env`). + Which credential the improving agent authenticates with. parent_version - The version to improve; defaults to the latest present. + The version to improve; defaults to the latest present, or ``None`` (create) when none + exist. model Override for the improving model; defaults to the config's. max_turns, max_usd - Turn and budget caps for the improving agent. **Unbounded by default** — the config's - ``max_turns``/``max_usd`` cap benchmark agents only; pass explicit caps to bound the - improver. + Turn and budget caps. Unbounded by default — the config's caps bound benchmark agents only. feedback - Optional maintainer guidance, injected into the improve prompt as a subordinated block - and recorded in ``meta.json`` as provenance. It is prompt text only — it cannot reach - the held-out test split, which stays blocked structurally and by the guard hook. + Optional maintainer guidance, subordinated below the hard rules and recorded in + ``meta.json``. It cannot reach the held-out valid split. log A :class:`LiveLog` to stream the agent's messages to and render an HTML log from. Returns ------- - The improved skill, loaded and validated, with its parent and cost. + The created/improved skill, loaded and validated, with its parent and cost. """ - parent = load_skill(skills_root, parent_version or _latest_or_error(skills_root), expect_name=cfg.skill_name) + # Resolve parent: an explicit version, else the latest present, else None (create the first). + resolved_parent = parent_version if parent_version is not None else latest_version(skills_root) + parent = None if resolved_parent is None else load_skill(skills_root, resolved_parent, expect_name=cfg.skill_name) + parent_ver = parent.version if parent is not None else None + new_version = next_version(skills_root) dest = skill_dir(skills_root, new_version) if dest.exists(): raise ImproveError(f"{dest} already exists — skill versions are immutable and never overwritten") - arm = arm_name(parent.version) - train_runs = collect_train_runs(runs_root, arm, tasks) + parent_arm = arm_name(parent_ver) + train_runs = collect_arm_runs(runs_root, parent_arm, tasks, split="train") if not train_runs: + hint = ( + "acumen bench --no-skill --split train" + if parent_ver is None + else f"acumen bench --skill {parent_ver} --split train" + ) raise ImproveError( - f"no train-split runs found for {parent.version} under {runs_root / arm / 'train'} — " - f"run `acumen bench --skill {parent.version}` first so the improver has evidence to work from" + f"no train-split runs found for {parent_arm} under {runs_root / parent_arm / 'train'} — " + f"run `{hint}` first so the wiki and improver have evidence to work from" ) n_failures = sum(1 for r in train_runs if not r.success) @@ -487,7 +317,8 @@ async def improve_skill( try: work = holder / "work" staging = work / new_version - train_dir = work / "train" + wiki_dir = work / WIKI_DIRNAME + transcripts_dir = work / "transcripts" rationale_path = work / "rationale.md" home = holder / "home" selected_model = model or cfg.meta_model @@ -497,8 +328,14 @@ async def improve_skill( for path in (work, home, config_dir, home / "tmp"): path.mkdir(parents=True, exist_ok=True) - _seed_staging(staging, parent) - _write_material(train_dir, train_runs) + if parent is not None: + _seed_staging(staging, parent) + else: + staging.mkdir(parents=True, exist_ok=True) + _stage_wiki(wiki_root, wiki_dir) + _stage_transcripts(runs_root, parent_arm, tasks, transcripts_dir) + # The filtered source: a package's own shipped skill must not bias the optimization. + source_copy = build_filtered_source(target.src_dir, holder / "source") # Marks the agent's processes so the teardown below can find what it leaves running. env = label_env( @@ -516,35 +353,40 @@ async def improve_skill( prompt = improve_prompt( package=target.pkg_name, version=target.pkg_version, + src=source_copy, python=target.python, skill_dir=staging, - train_dir=train_dir, + wiki_dir=wiki_dir, + transcripts_dir=transcripts_dir, rationale_path=rationale_path, skill_name=cfg.skill_name, - parent_version=parent.version, new_version=new_version, + parent_version=parent_ver, feedback=feedback, ) options = AgentOptions( cwd=work, env=env, model=selected_model, - # No default turn or budget cap: only bound the agent if the caller asked. The - # config's ``max_turns``/``max_usd`` cap benchmark agents only. + # No default turn or budget cap: only bound the agent if the caller asked. max_turns=max_turns, max_usd=max_usd, # Codex reports no billed figure, so a budget cap needs the run's own rate table. price_usd=pricer(selected_model, table), - read_dirs=(target.venv_dir,), + # The filtered source + venv, for verifying claims. Never the raw checkout. + read_dirs=(source_copy, target.venv_dir), write_dirs=(work,), discover_skills=True, # Belt-and-braces over the structural isolation: refuse any call that reaches a - # held-out test result, wherever the agent points it. Built only for Claude — the - # hook is an SDK object, and Codex gets the equivalent from ``deny_paths`` below. - claude_hooks={"PreToolUse": [make_test_guard(runs_root)]} if provider == "claude" else None, - # Codex gets only the staged train evidence and is denied the original - # run tree wholesale, which includes every held-out test artifact. - deny_paths=(runs_root.resolve(),), + # held-out valid result, or an unfiltered source artifact. Built only for Claude. + claude_hooks=( + {"PreToolUse": [make_valid_guard(runs_root), make_skill_guard(target.src_dir)]} + if provider == "claude" + else None + ), + # Codex gets only the staged evidence and filtered source; deny the run tree and the + # original checkout wholesale. + deny_paths=(runs_root.resolve(), target.src_dir.resolve()), ) result: AgentResult | None = None @@ -575,14 +417,14 @@ async def improve_skill( _validate_staged(staging, cfg.skill_name) - rationale = _read_rationale(rationale_path, result, parent.version, new_version) + rationale = _read_rationale(rationale_path, result, parent_ver, new_version) dest.parent.mkdir(parents=True, exist_ok=True) shutil.copytree(staging, dest) - write_meta(dest, parent=parent.version, rationale=rationale, feedback=feedback) + write_meta(dest, parent=parent_ver, rationale=rationale, feedback=feedback) skill = load_skill(skills_root, new_version, expect_name=cfg.skill_name) return ImproveResult( skill=skill, - parent=parent.version, + parent=parent_ver, cost_usd=resolve_cost( result.total_cost_usd, price_usage( @@ -604,20 +446,11 @@ async def improve_skill( shutil.rmtree(holder, ignore_errors=True) -def _latest_or_error(skills_root: Path) -> str: - from acumen.skills import latest_version - - latest = latest_version(skills_root) - if latest is None: - raise ImproveError(f"no skill versions found under {skills_root} — run `acumen draft` first, then bench it") - return latest - - -def _read_rationale(path: Path, result: AgentResult, parent: str, new_version: str) -> str: +def _read_rationale(path: Path, result: AgentResult, parent: str | None, new_version: str) -> str: """Recover the agent's rationale for ``meta.json``. - Prefers the ``rationale.md`` the prompt asks for; falls back to the agent's final - message, then to a plain provenance note, so a version always records why it exists. + Prefers the ``rationale.md`` the prompt asks for; falls back to the agent's final message, then + to a plain provenance note, so a version always records why it exists. """ if path.is_file(): text = path.read_text().strip() @@ -626,4 +459,4 @@ def _read_rationale(path: Path, result: AgentResult, parent: str, new_version: s final = (result.result or "").strip() if final: return final - return f"improved from {parent} to {new_version}" + return f"created {new_version}" if parent is None else f"improved from {parent} to {new_version}" diff --git a/src/acumen/paths.py b/src/acumen/paths.py index 18c4392..904f67b 100644 --- a/src/acumen/paths.py +++ b/src/acumen/paths.py @@ -13,9 +13,11 @@ from pathlib import Path from typing import Literal -Split = Literal["train", "test"] +# ``valid`` is the held-out split — the improver never sees it. ``test`` is reserved for a +# future train/valid/test three-way split; only train/valid exist today. +Split = Literal["train", "valid"] -SPLITS: tuple[Split, ...] = ("train", "test") +SPLITS: tuple[Split, ...] = ("train", "valid") NOSKILL_ARM = "noskill" diff --git a/src/acumen/prompts.py b/src/acumen/prompts.py index 0ba5c20..d05f46c 100644 --- a/src/acumen/prompts.py +++ b/src/acumen/prompts.py @@ -19,12 +19,12 @@ def feedback_block(feedback: str | None) -> str: ```` delimiters and placed (by the templates) *after* the hard rules, with explicit wording that it is guidance and does NOT override anything above it. That subordination is deliberate: the feedback comes from a trusted maintainer, but it must not be - able to talk an agent out of the isolation rules (test-split, skill-bias) or the anti-overfit + able to talk an agent out of the isolation rules (valid-split, skill-bias) or the anti-overfit rules — those are enforced structurally regardless, and the prompt says so. Note: feedback steers *within* each command's methodology; it cannot redefine it. It won't, for instance, stop ``tasks`` from running the package to verify answers, nor let ``improve`` - reach the test split to cheat. + reach the valid split to cheat. Parameters ---------- @@ -102,190 +102,214 @@ def feedback_block(feedback: str | None) -> str: """ -DRAFT_PROMPT = """\ -You are writing an agent skill for the Python package `{package}` (version {version}). +#: Shared guidance for the description trigger and anti-overfit body rules, used by both the +#: create and improve prompts so the two never drift apart. +_SKILL_CRAFT = """\ +# The description is the trigger + +The `description` is the ONE sentence an agent sees before deciding whether to open the skill. +It is the entire loading decision. Its job is to make the skill load exactly when it is +relevant and not otherwise. + +- **State the goals a user would actually phrase**, in their words, not the package's. An agent + matches the description against the task in front of it. Name the outcomes and the kinds of + question the skill answers. +- **Do not overfit it.** Widening the description with the specific datasets, methods, or + phrasings from the tasks is cheating: it buys train load rate and loses on the held-out valid + split. Widen to the CATEGORY of goal, never to the instances you saw. +- **Do not oversell.** A description that claims coverage the body does not deliver loads the + skill on tasks it cannot help with, and costs every one of those runs its tokens for nothing. + +# How to write the body -A skill is documentation written for an agent, not for a human. Its only purpose is to -make an agent that has never used `{package}` succeed at real tasks with it on the first -try. It is not a tutorial, not a README, and not a sales pitch. +- **Organize around goals, not modules.** Structure the skill so a stated goal maps to the right + entry point and the right sequence of steps — do not just mirror the module layout. +- **Write what is not guessable.** The agent already knows Python and can read a traceback. Spend + words on what it would get WRONG by guessing: non-obvious defaults, required preprocessing, the + function that looks right but isn't, where results land, argument shapes, the right order of steps. +- **Generalise — never overfit.** NEVER name a specific dataset, parameter value, column, expected + answer, or task in the skill. It must help on tasks you have not seen; enumerating the cases you + saw is cheating and fails the valid split. +- **Progressive disclosure.** Keep `SKILL.md` short and route depth to `references/*.md`. An agent + pays for every token of `SKILL.md` on every task, including the ones where it is irrelevant. +- **Prefer removing text over adding it.** A correct short code example beats a paragraph. If a + passage changed no outcome, cut it. +- **Verify before you write.** Do not assert a default, a return type, or where an output lands + without confirming it in the installed package or the docs. No hedging — say what to do.""" -The agent arrives with a goal stated in plain English — what a user wants done, not which -function to call. The skill's job is to get it from that goal to a working result: route it -to the right entry point and the right sequence of steps, so it never has to reverse-engineer -that from the module layout. + +IMPROVE_PROMPT = """\ +You are improving an agent skill for the Python package `{package}` (version {version}). + +A skill is documentation written for an agent, not a human. Its only purpose is to make an +agent that has never used `{package}` succeed at real tasks with it on the first try. + +You are producing version {new_version} — an improvement of version {parent_version}. # What you can read -- The package's source is at `{src}`. Read it — the source, the docstrings, the examples, - the docs directory. This is the ground truth about how the package behaves. -- `{package}` is also installed; run `{python}` to check anything you are unsure about. - Verify claims before you write them down. +- `{skill_dir}` — the current skill ({parent_version}). This directory has been pre-filled with a + copy of it. EDIT THESE FILES IN PLACE; what they contain when you finish becomes {new_version}. +- `{wiki_dir}` — the KNOWLEDGE WIKI: one directory per task, each with `observations.md` (what + agents did across models and replicates, how often the skill loaded, and whether loading led to + success) and `hypothesis.md` (why it did or did not work). Entries are tagged `[version][model]` + and accumulate across versions — `noskill` is the baseline, then `v1`, `v2`, … READ THIS FIRST. + It is the distilled signal of what the skill gets right and wrong; the trend across versions + tells you what past changes did. +- `{transcripts_dir}` — the raw TRAIN-split transcripts behind the wiki, for the parent skill. + Drill in here only when the wiki is not specific enough to act on. +- `{src}` — the package source (filtered: any skill or agent-guidance files a package ships have + been stripped and are BLOCKED — write from the API, not from someone else's skill). `{package}` + is installed; run `{python}` to verify a claim before writing it. Do not install packages. +- You have web access if the published docs help. -# Ignore any existing skills or agent instructions — deliberately hidden +# What you are NOT allowed to see -The package may already ship skills or agent-instruction files written for it (`SKILL.md`, -`.agents/skills/`, `.claude/skills/`, `.codex/`, `CLAUDE.md`, `AGENTS.md`, `.cursor/`, -Copilot instructions). These have been stripped from the source above, and any attempt to reach -them — or the original unfiltered checkout — is BLOCKED. This is on purpose: a skill copied from -pre-written guidance is not a skill written from the package, and this one is about to be -measured against no skill at all. Write yours from the API, the source, and the user-facing docs. +You are optimising against the TRAIN split. A separate, held-out VALID split measures whether your +changes generalise rather than memorise these tasks. Any tool call that reaches valid results is +BLOCKED. Do not attempt it — reaching valid data would invalidate the whole benchmark. -# What you must write +# Two different failures, two different fixes -Your working directory is `{out}`. Write: +The wiki records whether the agent LOADED the skill and whether it SUCCEEDED. Separate them before +changing anything — they look identical in a list of failures and have opposite fixes. -1. `{out}/SKILL.md` — required. It must begin with YAML frontmatter, exactly: +1. **The skill never loaded.** The agent never read the body, so nothing in the body caused this + and nothing you write in the body can fix it. The only lever is the `description`. +2. **The skill loaded and the run still failed.** Now the body is on trial. Fix it, using the + cause the wiki (or a transcript) actually shows — not one you imagine. +3. **The skill loaded and the run passed.** Evidence the body works. Do not rewrite it for style. ---- -name: {skill_name} -description: ---- +A skill that is never loaded scores exactly like no skill at all. If the wiki shows load rates low +across the board, that is the biggest available win — and the only lever is the `description`. If +one model loads reliably and another almost never does, that gap is mostly the model's behaviour; +do not contort the sentence chasing it. - The `name` must be exactly `{skill_name}`. - - The `description` is load-bearing and must be HONEST. It is the only part of the skill - an agent sees before deciding whether to open it, and it is the only thing that gets - the skill loaded at the right moment. - State what the skill covers and the goals it applies to — name the outcomes a user would - actually phrase, so the skill loads when one of them comes up. Do not oversell it, and do - not claim coverage the body does not deliver. Use this format: - [What it does] + [When to use it: the goals/triggers it fires on] + [Key capabilities] - -2. `{out}/references/*.md` — optional. Use these for detail that only some tasks need. - -# How to write it - -- **Organize around goals, not modules.** Work out what people actually use `{package}` to - accomplish — read its examples, tutorials, and docs, not just its API — and structure the - skill so a stated goal maps to the right entry point and the right sequence of steps. Do not - just mirror the package's module layout. -- **Write what is not guessable.** An agent already knows Python and can read a - traceback. Spend your words on what it would get WRONG by guessing: non-obvious - defaults, required preprocessing, the function that looks right but isn't, where - results are written, argument shapes and orientation, footguns the API invites, right order - of steps to follow. -- **Generalize within each goal.** Organize around categories of goal, but keep the guidance - under each one general enough to cover any task in that category. Don't enumerate one-off - recipes — that's the failure mode on the other side of module-mirroring. -- **Progressive disclosure.** `SKILL.md` should be short and route to `references/` for - depth. An agent pays for every token of it on every task, including the tasks where it - is irrelevant. If `SKILL.md` is long, you are taxing every run. -- **Be concrete.** A correct short code example beats a paragraph of prose. Show the real - call, with the arguments that matter. -- **Prefer removing text over adding it.** Anything that merely restates the obvious is - worse than nothing: it costs tokens and buries the parts that matter. -- **No hedging.** Say what to do. - -# Verify before you finish - -Do not write claims you have not checked. If you assert a default value, a return type, -or where an output lands, confirm it in the source or by running `{python}`. A skill that -confidently states something false is worse than no skill at all — it will send an agent -in the wrong direction with full confidence. +{craft} + +# What you must write + +1. Edit the skill in place under `{skill_dir}`. When you finish, `{skill_dir}/SKILL.md` must still + begin with YAML frontmatter whose `name` is exactly `{skill_name}`. +2. `{rationale_path}` — one short paragraph stating WHAT you changed and WHY, grounded in the wiki. + Write it OUTSIDE the skill directory; do not put it inside `{skill_dir}`. {feedback} -When you are done, `{out}/SKILL.md` must exist and start with the frontmatter above. +When you are done, `{skill_dir}/SKILL.md` exists and starts with the frontmatter above, and +`{rationale_path}` contains your rationale. """ -IMPROVE_PROMPT = """\ -You are improving an agent skill for the Python package `{package}` (version {version}). - -A skill is documentation written for an agent, not a human. Its only purpose is to make an -agent that has never used `{package}` succeed at real tasks with it on the first try. +CREATE_PROMPT = """\ +You are writing the FIRST agent skill (version {new_version}) for the Python package `{package}` +(version {version}). -You are producing version {new_version} — an improvement of version {parent_version}. +A skill is documentation written for an agent, not a human. Its only purpose is to make an agent +that has never used `{package}` succeed at real tasks with it on the first try. It is not a +tutorial, not a README, not a sales pitch. The agent arrives with a goal in plain English; the +skill routes it to the right entry point and sequence of steps. # What you can read -- `{skill_dir}` — the current skill ({parent_version}). This directory has been pre-filled - with a copy of it. EDIT THESE FILES IN PLACE; what they contain when you finish becomes - version {new_version}. -- `{train_dir}` — evidence from benchmarking the current skill on the TRAIN split. Read - `{train_dir}/SUMMARY.md` first. For each run you will find the task prompt, the expected - answer, the answer the agent actually gave, whether it passed, WHETHER THE AGENT LOADED THE - SKILL AT ALL, which model ran it, the `script.py` it wrote, and its full transcript. This is - your only signal about what the skill gets right and what it gets wrong. -- `{package}` is installed; run `{python}` (also `python` on your PATH) to verify any - claim about the API before you write it down. Do not install or upgrade packages. +- `{wiki_dir}` — the KNOWLEDGE WIKI: one directory per task, each with `observations.md` (what + agents did WITHOUT any skill — the `noskill` baseline — across models and replicates, and how + often they got it right) and `hypothesis.md` (why they succeeded or failed). READ THIS FIRST: it + tells you exactly where an unaided agent goes wrong, which is precisely what your skill must fix. +- `{transcripts_dir}` — the raw TRAIN-split transcripts behind the wiki. Drill in when the wiki is + not specific enough. +- `{src}` — the package source (filtered: any skill or agent-guidance files a package ships have + been stripped and are BLOCKED — write from the API, the source, and the user-facing docs, not + from someone else's skill). `{package}` is installed; run `{python}` to verify claims. Do not + install packages. - You have web access if the published docs help. # What you are NOT allowed to see -You are optimising against a TRAIN split. A separate, held-out TEST split is what measures -whether your changes actually generalise rather than memorising these particular tasks. You -must not see the test split, and any tool call that reaches test results will be BLOCKED. -Do not attempt it — reaching test data would invalidate the whole benchmark. +The wiki and transcripts are from the TRAIN split only. A held-out VALID split measures whether +your skill generalises. Any tool call that reaches valid results is BLOCKED — do not attempt it. # What you must write -1. Edit the skill in place under `{skill_dir}`. When you finish, `{skill_dir}/SKILL.md` - must still begin with YAML frontmatter whose `name` is exactly `{skill_name}` and whose - `description` is an honest one-sentence statement of what the skill covers and when to - use it. The `description` is the ONLY thing that decides whether the skill loads — see - below. - -2. `{rationale_path}` — one short paragraph stating WHAT you changed and WHY, grounded in - the train evidence. Write it here, OUTSIDE the skill directory. Do not put the rationale - inside `{skill_dir}`. - -# Two different failures, two different fixes +Your staging directory is `{skill_dir}`. Write: -Every run records whether the agent LOADED the skill and whether it SUCCEEDED. Separate them -before you change anything — they look identical in a list of failures and have opposite fixes. +1. `{skill_dir}/SKILL.md` — required. It must begin with YAML frontmatter, exactly: -1. **The skill never loaded.** The agent never read a word of the body, so nothing in the body - caused this and nothing you write in the body can fix it. The only lever is the - `description`. Editing the body in response to these runs is wasted work. -2. **The skill loaded and the run still failed.** Now the body is on trial: it steered the - agent wrong, or it was silent where it should have spoken. Fix the body, using the cause - you can see in that run's transcript — not one you imagine. -3. **The skill loaded and the run passed.** Evidence the body works. Do not rewrite it for - style. +--- +name: {skill_name} +description: +--- -A skill that is never loaded scores exactly like no skill at all, however good its body is. If -the load rate in `SUMMARY.md` is low, that is the biggest available win — and the only thing -you can do about it is the `description`. + The `name` must be exactly `{skill_name}`. The `description` is load-bearing and must be HONEST — + it is the only thing an agent sees before deciding whether to open the skill. -# The description is the trigger +2. `{skill_dir}/references/*.md` — optional, for detail only some tasks need. -The `description` is the ONE sentence an agent sees before deciding whether to open the skill. -It is the entire loading decision. Its job is to make the skill load exactly when it is -relevant and not otherwise. - -- **Read the load rates per model.** They are in `SUMMARY.md`, broken down by model. If one - model loads the skill reliably and another almost never does, that gap is mostly the model's - behaviour, not your wording — do not contort the sentence chasing it. Act on a rate that is - low across the board. -- **State the goals a user would actually phrase**, in their words, not the package's. An - agent matches the description against the task in front of it. Name the outcomes and the - kinds of question the skill answers. -- **Do not overfit it to the train tasks.** Widening the description with the specific - datasets, methods, or phrasings you just read in the train runs is the same cheating as - putting them in the body: it buys train load rate and loses on the test split. Widen to - the CATEGORY of goal, never to the instances you saw. -- **Do not oversell.** A description that claims coverage the body does not deliver loads the - skill on tasks it cannot help with, and costs every one of those runs its tokens for nothing. +3. `{rationale_path}` — one short paragraph on what the skill covers and why, grounded in the + baseline wiki. Write it OUTSIDE `{skill_dir}`. -# How to improve the body - -- **Fix what the evidence shows is broken.** Work from the runs where the skill loaded and - the agent still failed. Address the cause visible in the transcript, not one you imagine. -- **Generalise — never overfit.** NEVER name a specific dataset, parameter value, column, - expected answer, or task from the train runs in the skill. The skill must help on tasks - you have not seen. Guidance that enumerates these particular cases is cheating and will - fail the test split. -- **Prefer removing text over adding it.** A shorter skill that an agent reads and follows - beats a longer one it skims. Every token is paid on every task, including the ones where - the skill is irrelevant. If a passage did not change any outcome, cut it. -- **Verify before you write.** Do not assert a default, a return type, or where an output - lands without confirming it in the installed package or the docs. -- **No hedging.** Say what to do. +{craft} {feedback} When you are done, `{skill_dir}/SKILL.md` exists and starts with the frontmatter above, and `{rationale_path}` contains your rationale. """ +WIKI_PROMPT = """\ +You are a benchmark analyst keeping a running knowledge wiki about how well an agent skill for the +Python package `{package}` helps on ONE task. Your job this round: read what happened when the +`[{version}]` arm was benchmarked on this task across several models and replicates, and record it +BRIEFLY into two files. + +`[{version}]` is the skill version under review. `noskill` means no skill was installed (the +baseline). {skill_note} + +# What you are given + +- `{evidence_dir}/INDEX.md` — every run of `[{version}]` on task `{task_id}`, one line each: pass or + fail, whether the skill LOADED, the model, the expected answer, and the answer given. +- `{evidence_dir}/__rep_/` — that run's `transcript.html` (what the agent actually did), + its `answer.md`, and its `script.py`. Open a few to understand the pattern; you do NOT need to + read every one. +- The package source is at `{src}` and it is installed — run `{python}` if a fact helps you explain + an outcome. Optional; do not go down a rabbit hole. +{skill_body_note} +# What you must write — APPEND, do not rewrite + +Two files already exist and may contain entries from earlier arms. LEAVE those untouched and ADD +your new entries at the end. `noskill` is the first arm; later arms are `v1`, `v2`, … + +1. `{observations_path}` — add ONE line per model, in EXACTLY this format: + + - [{version}][]: + + Say the general idea, not a play-by-play. For `noskill`, "loaded" does not apply — just say what + the agents did and how often they got it right. + +2. `{hypothesis_path}` — add ONE line per model, in EXACTLY this format: + + - [{version}][]: + +# BE BRIEF — this is the whole point + +This wiki is read in full by the skill improver every round and grows forever. A verbose wiki is +WORSE than a short one. Hard rules: + +- One to two lines per entry. No transcript quotes, no step-by-step, no per-run breakdown, no + restating the task. Compress across replicates into the pattern. +- The hypothesis is one or two short sentences. No essays, no hedging. + +GOOD observation: + - [{version}][claude-opus-5]: Loaded 3/3; agents used the right entry point and passed every run. +GOOD hypothesis: + - [{version}][claude-opus-5]: The skill named the correct function and output location, which is + the step agents otherwise guess wrong. +BAD (too verbose): a paragraph recounting each replicate's tool calls and reasoning. + +When you are done, `{observations_path}` and `{hypothesis_path}` each contain your new +`[{version}]` line(s) appended after whatever was already there — and nothing else changed. +""" + + TASKGEN_PROMPT = """\ You are writing a benchmark of real analysis tasks for a Python package (`{package}`). Each task states a GOAL a user has, in plain language, plus the single answer a correct analysis @@ -351,9 +375,9 @@ def feedback_block(feedback: str | None) -> str: - GOOD (a lazy goal): "Using the pbmc3k data, find which transcription factor is most active in the monocytes. Give only the factor's symbol." -# Train and test variants +# Train and valid variants -Give each task a train and a test variant of the SAME goal, differing only in the input or the +Give each task a train and a valid variant of the SAME goal, differing only in the input or the target it asks about (a different cell type, group, condition, or dataset). Two instances of one analysis with two different correct answers — so a skill cannot pass by memorising one answer. @@ -369,7 +393,7 @@ def feedback_block(feedback: str | None) -> str: The script you run to obtain an answer is NOT scratch. Save one per split to `{scripts_dir}/-.py`, using the SAME `id` you gave the task in `{out}` — so the task -`bulk` needs `{scripts_dir}/bulk-train.py` and `{scripts_dir}/bulk-test.py`. These are what +`bulk` needs `{scripts_dir}/bulk-train.py` and `{scripts_dir}/bulk-valid.py`. These are what `acumen check` reruns later to confirm the answer still holds, so each one must: - Be SELF-CONTAINED and runnable from ANY empty working directory: `