diff --git a/README.md b/README.md index 9dac23f..b2788b4 100644 --- a/README.md +++ b/README.md @@ -10,263 +10,52 @@ [badge-tests]: https://img.shields.io/github/actions/workflow/status/scverse/acumen/test.yaml?branch=main [badge-docs]: https://app.readthedocs.org/projects/acumen/badge/ -Agentic skills, tool instructions writen in plain text, allow agents to use tools more succesfuly and efficient. -However most python packages do not ship skills with them because developers have no easy way to build and benchmark skills for their tools. -Acumen closes this gap. Point it at a Python package and a few evaluation tasks, and it drafts a skill, benchmarks it and improves it across a train/test split so the gains are generalizable. +An agentic skill is a short, plain-text set of instructions that helps a coding agent use a tool +correctly. Most Python packages ship none — maintainers have no easy way to write one, or to tell +whether it actually helps. acumen closes that loop: point it at a package and a few tasks, and it +learns a skill in **training epochs**, each one benchmarking the current skill, distilling what went +right and wrong into a knowledge wiki, and rewriting the skill — always measured against a no-skill +baseline on a held-out split, so the gains generalize instead of memorizing answers. +## Installation -Many good tools are unusable by coding agents because their maintainers have no way to write -a skill for them — or, having written one, no way to tell whether it helps. acumen closes -that loop: point it at a Python package and a few tasks, and it drafts a skill, benchmarks it -against a no-skill baseline, and improves it across a train/test split so the gains are real -generalization, not memorized answers. - -- **`acumen check`** — rerun the script behind each task's answer, then have an agent judge - whether the prompt actually asks for what that script and answer produce. Both before a - benchmark pass pays to find out. -- **`acumen draft`** — write `skills/v1` from the package's own source. -- **`acumen bench`** — score a skill against a no-skill baseline, in a scrubbed sandbox where - the skill is the only difference between arms. Agent guidance the target ships itself is - removed from the venv first, so the baseline really is skill-free. -- **`acumen improve`** — refine the skill from its train results, then benchmark again. -- **`acumen report`** — aggregate every run into one self-contained `report.html`: success - rate per version, train vs. test. Bars are coloured by model, with a grey bar pooling all - of them; pass `--palette claude-opus-5=#3b7ea1` (repeatable) to recolour any of them. +Python 3.12+ is required. Install the backend you actually run — both are optional, and either +alone is a complete install: -You decide when to stop. Every version is benchmarked on both splits, and only train results -reach the improver — so a widening train/test gap is a visible sign a skill is overfitting -rather than genuinely helping. +| you run | install | also needs | +|---|---|---| +| Claude only | `pip install acumen[claude]` | an Anthropic key or a `claude` login | +| Codex only | `pip install acumen` | the `codex` CLI on `PATH`, plus a Codex login or key | +| both | `pip install acumen[all]` | both of the above | ## Quickstart ```bash -# 1. Scaffold a starter config.yaml and tasks.yaml -acumen init - -# 2. Fill in config.yaml (repo). Write tasks.yaml by hand, or generate it: -acumen tasks # mine the package for real analyses -> tasks.yaml + tasks/ -acumen check # is the ground truth right, and does each prompt ask for it? +acumen init # scaffold config.yaml + tasks.yaml +# edit config.yaml: point `repo` at your package (a GitHub URL or local path) -# 3. Then run the loop: -acumen bench --no-skill # the baseline arm -acumen draft # generate skills/v1 from the package source, or write by hand -acumen bench --skill v1 # benchmark the skill against the baseline -acumen improve # generate skills/v2 from v1's train results, or write by hand -acumen bench --skill v2 -acumen bench # or: every arm at once (baseline + each skills/vN) -acumen report # aggregate every run into report.html +acumen tasks # mine the package for real tasks + ground-truth reproducers +acumen check # verify each task's answer reproduces before you spend -# 4. Once a version proves out, ship it into the package itself: -acumen ship --skill v2 # add a -install-skills console script (PR, or local edit) -``` - -`acumen ship` packages the chosen skill version into the target: the package gains a -`-install-skills` command that installs the skill into the skills directory of whichever -agent the user names — `--agent {claude,codex,agents,claude-science}`, or an explicit `--dest` — -so the package's own users get the guidance with one command, wherever they run their agent. The -same bundle installs verbatim into every framework. - -## Checking the ground truth - -A task is only worth benchmarking if its recorded answer is actually correct. A wrong answer makes -every model fail that task: real money spent, and the failure reads in the report as the model's -fault rather than the task's. `acumen check` catches the two ways that happens. +acumen epoch # one training round: bench train → wiki → create/improve skill → bench held-out +acumen epoch # run again for the next round (v2, v3, …) -**Does the answer still come out of running the code?** Each task keeps a **reproducer** at -`tasks/-.py`, a self-contained script that redoes the analysis in the target venv and -writes its answer to `answer.md` — the same contract a benchmark run has, graded the same way. -`acumen tasks` writes them as it generates the tasks; `acumen check` reruns them: - -```bash -acumen check # every task, both splits -acumen check --task bulk --split train # one cell, while you fix it -acumen check --jobs 8 --timeout 600 # or: --keep to inspect what a script wrote +acumen report # aggregate every run into a self-contained report.html +acumen ship --skill v2 # ship the chosen skill into your package ``` -You get one row per task and split — reproduced, wrong answer, script error, timed out, or no -script at all — then the summary statistics: how much of the task set has a reproducer, how much -of it reproduces, and how many tasks reproduce on both splits. Before running anything it checks -that the package imports in the venv at all, since that one failure would otherwise be reported -once per task. - -**Does the prompt actually ask for what the script and answer produce?** Reproducing an answer -proves the code and the answer agree. It says nothing about the prompt, and a prompt describing -something else fails every agent that reads it correctly. A real example: - -> Find the 3 most deactivated PROGENy pathways in Megakaryocytes … Report only the pathway names -> sorted by score **(ascending)**. - -The script sorted descending, the recorded answer was descending, the reproducer check said `ok` — -and every agent that honoured the prompt produced the reverse order and was graded wrong. So after -the scripts run, one agent reads every split's prompt, recorded answer and reproducer together and -adds a `review` column of `ok` or `mismatch`, with one line naming the contradiction and one naming -the fix. It never edits `tasks.yaml`: which of the three artifacts to repair is your call. - -``` -task split status review detail -scell train ok ok MAPK;Estrogen;TGFb -scell test ok mismatch Trail;JAK-STAT;Estrogen - -1 split the review flagged - scell/test prompt says ascending; script and answer are descending - fix: say descending in the prompt, or reverse the answer -``` - -The review is on by default and picks its model from `meta_model`; it is the one phase that costs -money, so `acumen check --no-review` runs the reproducers alone and spends nothing — what you want -while iterating on a script. `check` takes the same `--auth`, `--stream` and `--log-dir` flags as -the other agentic commands, and `--max-turns`/`--max-usd` bound the reviewer. - -Either phase failing exits non-zero, so `acumen check` works as a gate before a pass. - -A task that needs no code to answer (a licence, a supported species, a documented default) sets -`needs_script: false`; its reproducer column reads `n/a` rather than counting as a gap, and its -prompt and answer are still reviewed. - -The reproducers hold the answers to the held-out test split, so nothing must feed them to an agent -under test. They are safe where they are: `bench`, `draft`, and `improve` confine their agents to -explicit read roots that never include your project directory, and the reviewer reads a staged copy -with no path back to `tasks.yaml`. - -`acumen tasks`, `acumen draft`, and `acumen improve` each accept `--feedback "…"` to steer the -agent with context it can't infer — which functionality to skip when generating tasks, what a -skill should emphasise or fix. The guidance is added to the prompt without overriding the -train/test isolation, and for `draft`/`improve` it is recorded in the version's `meta.json` and -shown in the report. (Don't paste held-out test answers into `improve` feedback — that would -defeat the split.) - -Claude and Codex can run side by side. Put both model families in `models` to compare them -in one matrix; model IDs beginning with `claude` use Claude Code, while `gpt-*`, `o1`, -`o3`, `o4`, and `codex-*` use Codex: - -```yaml -models: - - claude-opus-5 - - claude-sonnet-5 - - claude-haiku-4-5-20251001 - - gpt-5.6-sol - - gpt-5.6-terra - - gpt-5.6-luna -``` - -This spans each provider's quality/cost range; it is not a claim that the tiers are -one-to-one equivalents. - -Neither backend is required. Claude is an optional dependency and Codex is an external CLI, -so install only the one you run — `pip install acumen[claude]`, or plain `acumen` plus the -`codex` CLI on `PATH`. Selecting a model whose backend is missing fails immediately, with the -install command, before acumen prepares a target or spends anything. +`acumen epoch` is the loop. Each round benchmarks the current arm on the training tasks, records +what happened per task in `wiki/`, creates or improves the skill from that wiki, and benchmarks the +new version on the held-out validation tasks. Re-run it to keep going; you decide when to stop. -Claude API runs use `ANTHROPIC_API_KEY`; Codex API runs use `CODEX_API_KEY` (or -`OPENAI_API_KEY`). The meta-agent commands also accept a Codex model through the -`meta_model` config key or `--model`. +The stages also run on their own — `acumen bench`, `acumen wiki`, and `acumen improve`. -Every agentic command — `bench` included — takes `--auth {auto,session,api}` and defaults to -the provider's logged-in subscription, falling back to its API key. Both billing modes report -tokens, so Acumen can calculate the same API-rate estimate for either. Under `session`, that -estimate is what the run *would* have cost at API rates, not money billed — so each run records -its `auth_mode` alongside the figure. - -If the selected subscription runs out of usage or the API account runs out of credit, Acumen -invalidates the pass instead of scoring that as an agent failure: it prints the provider error, -cancels remaining cells for that provider, lets other providers finish all running and queued -cells, and exits non-zero. Replenish the credential and rerun the same command; automatic resume -retries the invalid and cancelled cells. Reports and `improve` refuse invalid quota/credit -evidence. - -`max_turns` and `max_usd` apply to both providers, but they are not equally strict for Codex, -which has no cap of its own — acumen enforces both against its event stream: - -- **`max_turns` bounds the run.** One `codex exec` is a single Codex turn however much work - happens inside it, so turns are counted in completed model actions (a message, a command, a - file change, a tool or search call) and the agent is stopped at the cap. -- **`max_usd` cannot.** Codex reports usage once, when the turn ends, so a breach is only - visible after the money is spent. The run is recorded as a budget failure — the same outcome - Claude gives it — but bound Codex spend with `max_turns`. acumen prints this before the pass. - -**Every cost acumen shows is inferred from tokens.** Each run records its breakdown (fresh -input, cache reads, cache writes, and output) and Acumen prices it with the rate table stored -in `result.json`. That gives Claude and Codex one comparable basis and prevents an old -benchmark from being silently re-priced, so it is what `cost_usd` holds and what every figure, -table, CSV column and console line reports. Where a backend supplies a dollar figure of its own -it is recorded beside it as `provider_cost_usd` (`recorded_cost_usd` in the report's sidecar -CSV), with the gap between the two, but nothing is plotted or tallied from it: Claude's SDK -total covers nested subagents that the run's own usage block does not, so a console reading it -would disagree with the report it summarises. A model no layer prices stays unpriced even when -the provider reported dollars, since one run on a basis the rest of the pass is not on is worse -than a visible gap. - -**Rates are read from the providers' pricing pages, never shipped with the package.** Prices -move, and each run's cost is frozen into its `result.json` and never recomputed, so a table -compiled into a release would store numbers that were already wrong. `bench` resolves rates -before it spends anything and **fails the pass** if the pages cannot be read: cost is a headline -metric, and a benchmark that cannot establish rates has not earned the numbers it would print. -`draft`, `improve`, `tasks`, `check`, and `ship` fetch too but degrade to unpriced instead — their -cost line is progress reporting, not stored evidence. - -Alongside the rates themselves each run records `price_source` (`config` or `fetched`) and -`price_rates_as_of`, so a pass run in August and another in October stay individually -attributable and one report can cover both without restating either. When arms in a report were -priced on different dates, the report says so: the cost gap between them includes the price -change, not only the skill's effect. - -```bash -acumen prices # the rates in use today, and where each came from -acumen prices --refresh # check pinned rates against what the providers publish -``` - -Pin rates with a `prices:` block in `config.yaml` to price a model the providers don't publish, -to price a gateway, or to record negotiated rates — pins outrank a live fetch, since only you -know what you are billed. They are also the only rates that can drift unnoticed, which is what -`--refresh` checks; it prints a diff for you to accept and never rewrites anything, because -picking the wrong tier or context band would silently misprice future runs. A model no layer -prices records its tokens and leaves report cost unavailable — never zero, which would read as -free. - -> One consequence worth knowing: Codex's `max_usd` cap is enforced from these same rates, so an -> unpriced model under Codex has no enforceable budget cap. Bound those runs with `max_turns`, -> or pin the rates. - -`draft`, `improve`, `tasks`, `ship`, and `check`'s review phase each drive an autonomous agent. -Every run writes a live `logs/acumen--.jsonl` (one event per step, flushed as it -goes — so you can watch progress by reading the file) and a rendered `.html` transcript. Add -`--stream` to mirror the conversation to the terminal, or `--log-dir` to change where the logs -land. +`acumen ship` gives your package a `-install-skills` command so its users can install the +skill into whichever agent they use (`--agent {claude,codex,agents,claude-science}`, or `--dest`). ## Getting started -Please refer to the [documentation][], -in particular, the [API documentation][]. - -## Installation - -You need to have Python 3.12 or newer installed on your system. -If you don't have Python installed, we recommend installing [uv][]. - -Install the backend you actually run — both are optional, and either alone is a complete -install: - -| you run | install | also needs | -|---|---|---| -| Claude only | `pip install acumen[claude]` | an Anthropic key or a `claude` login | -| Codex only | `pip install acumen` | the `codex` CLI on `PATH`, plus a Codex login or key | -| both | `pip install acumen[all]` | both of the above | - - - -And to install the acumen skill that ships with the package into your agent's skills directory, -run `acumen-install-skills --agent {claude,codex,agents,claude-science}` (or `--dest ` to -choose the directory yourself): - -```bash -acumen-install-skills --agent claude -``` +Please refer to the [documentation][], in particular the [API documentation][]. ## Release notes @@ -274,7 +63,7 @@ See the [changelog][]. ## Contact -For questions and help requests, you can reach out in the [scverse discourse][]. +For questions and help requests, reach out on the [scverse discourse][]. If you found a bug, please use the [issue tracker][]. ## Citation @@ -288,4 +77,3 @@ If you found a bug, please use the [issue tracker][]. [documentation]: https://acumen.readthedocs.io [changelog]: https://acumen.readthedocs.io/page/changelog.html [api documentation]: https://acumen.readthedocs.io/page/api.html -[pypi]: https://pypi.org/project/acumen diff --git a/pyproject.toml b/pyproject.toml index 3da0ae9..e4be2fc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -44,7 +44,6 @@ urls.Documentation = "https://acumen.readthedocs.io/" urls.Homepage = "https://github.com/scverse/acumen" urls.Source = "https://github.com/scverse/acumen" scripts.acumen = "acumen.cli:main" -scripts.acumen-install-skills = "acumen._skills.install:main" [dependency-groups] dev = [ diff --git a/src/acumen/__init__.py b/src/acumen/__init__.py index 542ce72..ee500ec 100644 --- a/src/acumen/__init__.py +++ b/src/acumen/__init__.py @@ -29,7 +29,6 @@ summarize_checks, ) from acumen.config import Config, ConfigError, load_config, parse_config -from acumen.draft import DraftError, DraftResult, draft_skill from acumen.env import ( AuthMode, EnvError, @@ -41,15 +40,14 @@ scrubbed_env, session_auth_available, ) +from acumen.epoch import EpochPlan, resolve_epoch from acumen.grade import Grade, Reason, grade_answer, grade_run from acumen.improve import ( ImproveError, ImproveResult, - TrainRun, - collect_train_runs, - find_test_access, + find_valid_access, improve_skill, - make_test_guard, + make_valid_guard, ) from acumen.logs import LiveLog from acumen.paths import RunKey, Split, arm_name, is_complete, parse_run_dir, run_dir, skill_from_arm @@ -104,6 +102,14 @@ harvest_scripts, ) from acumen.tasks import Task, TaskError, TaskSplit, load_tasks, parse_tasks +from acumen.training import ( + EpochRow, + best_version, + build_training_rows, + epochs_since_best, + patience_exhausted, + write_training_csv, +) from acumen.trajectory import Trajectory, render_trajectory from acumen.transcript import ( build_trajectory, @@ -112,8 +118,32 @@ render_codex_transcript, render_transcript, ) +from acumen.wiki import ( + RunRecord, + TaskWikiResult, + WikiError, + collect_arm_runs, + recorded_arms, + update_wiki, +) __all__ = [ + "EpochRow", + "build_training_rows", + "write_training_csv", + "patience_exhausted", + "best_version", + "epochs_since_best", + "EpochPlan", + "resolve_epoch", + "find_valid_access", + "make_valid_guard", + "RunRecord", + "TaskWikiResult", + "WikiError", + "collect_arm_runs", + "recorded_arms", + "update_wiki", "AuthMode", "AgentError", "AgentOptions", @@ -126,8 +156,6 @@ "CheckSummary", "Config", "ConfigError", - "DraftError", - "DraftResult", "EnvError", "Grade", "Harvest", @@ -160,7 +188,6 @@ "TaskGenResult", "TaskSplit", "Trajectory", - "TrainRun", "__version__", "api_auth_available", "arm_metrics", @@ -176,11 +203,8 @@ "build_report", "check_task_split", "check_tasks", - "collect_train_runs", - "draft_skill", "dump_tasks", "find_skill_access", - "find_test_access", "generate_tasks", "grade_answer", "grade_run", @@ -191,7 +215,6 @@ "installer_exists", "is_complete", "make_skill_guard", - "make_test_guard", "latest_version", "load_config", "locate_transcript", diff --git a/src/acumen/_skills/__init__.py b/src/acumen/_skills/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/src/acumen/_skills/data/SKILL.md b/src/acumen/_skills/data/SKILL.md deleted file mode 100644 index 0a7fbf3..0000000 --- a/src/acumen/_skills/data/SKILL.md +++ /dev/null @@ -1,127 +0,0 @@ ---- -name: acumen -description: Use for any question or task involving the Python package `acumen` (its CLI or its API) — setting up a benchmark project against a target package, writing or generating benchmark tasks, drafting/improving/hand-editing an agent Skill, running and interpreting skill-vs-baseline benchmark passes, shipping a skill into the target package, diagnosing a run, and choosing what to do next in that loop — open it before answering or running anything, because acumen's defaults, guardrails and correct next step are not guessable from the command names. ---- - -# acumen - -One project directory targets one Python package. Everything is driven by the `acumen` -CLI (a thin shell over an importable API — see `references/python-api.md`). Commands are -run from the project dir and default to `config.yaml`, `tasks.yaml`, `skills/`, `runs/`, -`logs/`. - -```bash -acumen init # scaffold config.yaml + tasks.yaml (placeholders the user must fill) -acumen tasks # optional: mine the package for tasks, run them for ground truth -acumen bench --no-skill # baseline arm (bare `acumen bench` runs every arm there is) -acumen draft # agent reads the package source -> skills/v1/ -acumen bench --skill v1 -acumen improve # agent reads v1's TRAIN runs -> skills/v2/ -acumen bench --skill v2 -acumen report # report.html + report.csv over every arm on disk -acumen ship --skill v2 # wire a `-install-skills` script into the target package -``` - -That is the shape of the loop, not a script to execute unattended, and not a fixed order: -several stages have more than one correct next move, and some stages call for talking to -the user rather than running the next command at all. - -## The loop is collaborative — propose, discuss, then run - -`bench` bills the Anthropic API for every cell and the other commands drive long -autonomous agents, so the user decides each step and when to stop. At every stage: state -what you would do next and why, and let the user choose. Never chain commands -unattended, and never decide on your own that a skill is good enough. - -**Before running anything, get from the user what acumen cannot infer** — the target, the -models, the budgets, whether tasks and results look right, whether to ship. Guessing those -is the most common way to waste a pass. The step after a command is often a question, not -a command. - -| Just happened | What to propose next | -|---|---| -| `acumen init` | Both files are placeholders (`repo: OWNER/REPO`, `REPLACE_ME` answers). Ask the user for what only they can supply — the target repo/path, ref, models, budgets — and fill in `config.yaml` with their answers. Do not guess a target or run the next command. | -| `config.yaml` filled | Offer both ways to get tasks: `acumen tasks` to generate them, or writing `tasks.yaml` by hand. Review generated tasks with the user before benching. | -| config + tasks ready | `acumen bench --no-skill` and `acumen draft` are **both** correct next steps — they are independent, and you need the baseline arm and a skill arm before any comparison means anything. Offer both. | -| a skill version exists | `acumen bench --skill vN`, or bare `acumen bench` to cover the baseline and every version at once (resume means only the unbenched cells cost anything). | -| any `bench` finished | `acumen report` — the per-arm, per-split numbers come from the report; do not judge a version by eyeballing runs or by one arm's pass count. | -| `report` written | Discuss the results with the user and propose next steps: `improve` + re-bench if it has not beaten the baseline or the previous version, `ship` once a version has proven out, or stop. The user decides. | -| user has a skill dir from elsewhere | Copy it into `skills/v1/` (next unused version) inside the acumen project — versions are only ever read from `skills/`; there is no import command and no external path flag. | - -## Route by goal - -| Goal | Command | Depth | -|---|---|---| -| Start a project for package X | `acumen init`, then fill `config.yaml` with the user | `references/setup.md` | -| Get benchmark tasks without writing them | `acumen tasks [--force] [--feedback "…"]` | `references/setup.md` | -| Write tasks by hand | edit `tasks.yaml` | `references/setup.md` | -| Get a first skill | `acumen draft [--feedback "…"]` | `references/authoring.md` | -| Measure whether the skill helps | `acumen bench --no-skill` and `acumen bench --skill vN` | `references/benchmark.md` | -| Make the skill better | `acumen improve [--from vN]` then bench the new version | `references/authoring.md` | -| See results / decide when to stop | `acumen report` | `references/benchmark.md` | -| Hand-edit a skill version | copy `skills/vN` → `skills/v(N+1)`, edit, bench it | `references/authoring.md` | -| Give package users the skill | `acumen ship --skill vN` | `references/ship.md` | -| Script any of this in Python | `from acumen import …` | `references/python-api.md` | -| A run failed / the skill did nothing | inspect `runs/…/result.json`, `logs/*.jsonl` | `references/benchmark.md` | - -## Preconditions - -- Python ≥ 3.12, and **`uv` on PATH** — acumen builds the target's venv with it. -- The target (`repo`: a GitHub URL or a local path) must be pip-installable and declare - `[project].name` in `pyproject.toml`. A local `repo` path is resolved **relative to - `config.yaml`**. -- Credentials: every agentic command takes `--auth {auto,session,api}` and defaults to the - selected provider's logged-in subscription, falling back to its API key. Claude uses - `ANTHROPIC_API_KEY` (or Anthropic provider credentials) / a `claude` login; Codex uses - `CODEX_API_KEY` or `OPENAI_API_KEY` / a `codex login`. `bench` included: cost comes from - token counts, which both billing modes report, so a subscription run prices as accurately - as a metered one — but under `session` `cost_usd` is what the run *would* have cost at API - rates, not metered spend. Each run records its `auth_mode`. - Both backends are optional: `pip install acumen[claude]` for Claude, plain `acumen` plus - the `codex` CLI for Codex. A model whose backend is missing fails preflight with the - install command. -- **Codex caps are not equally strict.** `codex exec` has no cap of its own, so acumen - enforces both from its event stream. `max_turns` bounds the run (counted in completed - model actions, since one `codex exec` is a single Codex turn). `max_usd` cannot: Codex - reports usage only when a turn ends, so an over-budget run is recorded as a `budget` - failure but the spend already happened — bound Codex with `max_turns`. -- **`cost_usd` is derived from token counts**, not from the provider — one arithmetic - path for both. Rates are frozen into each run; see them with `acumen prices`, re-check - them with `acumen prices --refresh`, override or extend via `prices:` in `config.yaml`. -- The target is cloned + installed into a venv cached under `~/.cache/acumen`, keyed by - (repo, ref). Use `--refresh-target` after changing the target's own source. - -## What guessing gets wrong - -1. **Grading is exact string match on `answer.md` after `strip()`, case-sensitive.** - Nothing is normalized. `**TOKEN**` against `TOKEN` fails (recorded as `format_error`, - not `wrong_answer`). Task answers must be one short unambiguous token. -2. **A pass is models × tasks × splits × replicates, times the arms it covers.** The - scaffolded config lists 6 models and `n_replicates: 3`, so *one* task costs 36 agent runs - *per arm* — and bare `acumen bench` runs every arm on disk, so with two skill versions - that is 108. Check with `acumen bench --dry-run` (it plans and exits, spending nothing, - over the same arms the real run would) and agree the size with the user before spending. - Trim with `models:`, `n_replicates: 1`, `--task ID`, `--split`, or by naming one arm. -3. **`max_turns`/`max_usd` in `config.yaml` cap benchmark agents only.** `draft`, - `improve`, `tasks`, and `ship` are **unbounded** unless you pass `--max-turns`/`--max-usd`. -4. **Skill versions are immutable.** `draft` refuses (exit 2) if any `skills/vN` exists; - `improve` always writes the next unused directory. Never edit a benched version in - place — its hash is recorded in every `result.json`. -5. **`SKILL.md` frontmatter `name` must equal `config.skill_name`**, which defaults to the - repo's last path component, slugified and lowercased (`.../My_Pkg` → `my_pkg`). A - mismatch makes `bench`/`improve`/`ship` fail on load. `description` must be non-empty. -6. **`improve` needs benched train evidence for its parent.** Run - `acumen bench --skill vN` before `acumen improve`, or it errors with nothing to read. -7. **Never leak the test split.** `improve` is structurally and hook-blocked from - `runs/*/test/`; don't defeat that by pasting test answers into `--feedback`. A widening - train/test gap in the report is the overfitting signal you are watching for. -8. **Task prompts must not name the target package** — the harness already tells the agent - which package to use and that it is installed. -9. **Resume is automatic**: a valid run is "complete" when its `result.json` exists and is - non-empty, and completed runs are skipped. `--no-resume` re-runs them. Provider usage/credit - exhaustion writes a diagnostic `valid: false` result, cancels only that provider's remaining - cells while other providers finish, and remains pending; replenish the credential and rerun - the same command. Renaming a task `id` orphans its old runs (the id is a path component). -10. **Confirm the skill actually loaded.** `bench` prints `skill loaded in N/M runs` from - per-run `skill_loaded` evidence, and warns when a skill arm never fired the Skill tool - (that arm measures nothing) or when the baseline did. diff --git a/src/acumen/_skills/data/references/authoring.md b/src/acumen/_skills/data/references/authoring.md deleted file mode 100644 index 1817d9a..0000000 --- a/src/acumen/_skills/data/references/authoring.md +++ /dev/null @@ -1,92 +0,0 @@ -# Authoring skill versions: draft, improve, hand-edit - -## The skill directory contract - -``` -skills/v1/ - SKILL.md # required, YAML frontmatter with `name` and `description` - references/*.md # optional - meta.json # acumen bookkeeping — NOT part of the skill -``` - -- `name` must equal `config.skill_name` exactly; `description` must be non-empty. Both are - enforced on load by `bench`, `improve`, and `ship`. -- `meta.json` (`version`, `parent`, `rationale`, `hash`, optional `feedback`) is written by - acumen. It is excluded from the content hash and is never copied to a consuming agent. -- The content hash covers every other file's relative path and bytes, and is recorded in - every `result.json` — so editing a version after benching it silently invalidates the - comparison. **Versions are immutable; always make a new one.** - -## `acumen draft` - -```bash -acumen draft [--force] [--feedback "…"] [--model M] [--max-turns N] [--max-usd X] - [--stream] [--log-dir logs] [--auth auto|session|api] -``` - -One agent that reads the target's **source** (docstrings, examples, docs) and writes -`skills/v1/`. It works in a staging dir; only a skill that loads and validates is promoted, -so a failed draft leaves no broken version behind. Exits 2 if any version already exists — -pass `--force` to add another, or use `improve` to build on the latest. - -## `acumen improve` - -```bash -acumen improve [--from vN] [--feedback "…"] [--model M] [--max-turns N] [--max-usd X] - [--stream] [--log-dir logs] [--auth auto|session|api] -``` - -Reads **how the parent version performed on the train split** — not the package source — -and edits a copy of it into the next version. - -- Defaults to the latest version; `--from vN` picks another. Always writes the next unused - directory. -- Requires `runs/skill_vN/train/**/result.json` to exist: bench the parent first. -- The agent gets a curated copy of train runs (`SUMMARY.md` with failures first, plus each - run's prompt, expected, actual, `script.py`, and transcript). The real `runs/` tree is not - in scope, and a `PreToolUse` hook denies any path resolving under `runs/*/test/`. -- The CLI warns if the new version is byte-identical to its parent (the improver changed - nothing) — that is a signal to give it sharper evidence or `--feedback`. - -## `--feedback` on `tasks` / `draft` / `improve` - -Free text injected into the agent's prompt as *subordinate* guidance — it never overrides -the isolation or anti-overfit rules. Use it for what the agent cannot infer: package -context, what to emphasise, which functionality to skip. For `draft`/`improve` it is -recorded in `meta.json` and shown in the report. **Never paste test-split answers into -`improve --feedback`** — that defeats the split. - -## Writing or editing a version by hand - -Perfectly supported; the agents are a convenience, not a requirement. A skill authored -outside the project is benched the same way: copy the directory in as the next unused -`skills/vN` — `bench` only ever loads versions from the project's `skills/` root. - -```bash -cp -r skills/v2 skills/v3 # then edit skills/v3/, and delete its inherited meta.json -acumen bench --skill v3 -``` - -A hand-made version with no `meta.json` benches fine; it just has no rationale/diff in the -report. What makes a version score well: - -- The `description` decides whether the skill loads at all — name the goals a user would - actually phrase, honestly. An unloaded skill is a wasted arm. -- Keep `SKILL.md` short and push depth into `references/`; every token is paid on every - task, including ones where the skill is irrelevant. -- Spend words on what an agent would get wrong by guessing: non-obvious defaults, required - preprocessing, the function that looks right but isn't, argument shapes/orientation, - where output lands, the right order of steps. -- **Never name a dataset, parameter value, column, or expected answer from the train - tasks.** That is overfitting, and the test split will catch it. -- Verify every claim against the installed package before writing it down. - -## Watching a long agent run - -`tasks`/`draft`/`improve`/`ship` each write `logs/acumen--.jsonl`, -one compact event per agent message, **flushed as it goes** — read that file to follow -progress instead of streaming into your context. Tool results are recorded by status and -size, not inlined. `--stream` mirrors the conversation to the terminal; `--log-dir` moves -the logs. At the end, the run is mapped into acumen's harness-neutral trajectory model and a -rendered `.html` transcript plus a portable `.trajectory.json` land beside the jsonl — the same -model and renderer for every provider, so the reports look alike. diff --git a/src/acumen/_skills/data/references/benchmark.md b/src/acumen/_skills/data/references/benchmark.md deleted file mode 100644 index 56e8a23..0000000 --- a/src/acumen/_skills/data/references/benchmark.md +++ /dev/null @@ -1,162 +0,0 @@ -# Benchmarking and reading results - -## `acumen bench` - -```bash -acumen bench [--no-skill | --skill v1] [--split train|test]... [--task ID]... - [--replicates N] [--max-concurrency N] [--dry-run] [--no-resume] - [--keep-sandboxes] [--refresh-target] - [--auth auto|session|api] - [--config config.yaml] [--tasks tasks.yaml] [--runs runs] [--skills skills] - [--cache ~/.cache/acumen] -``` - -- Neither `--no-skill` nor `--skill` → **every arm the project has**: the baseline plus each - version in `skills/`, benched one arm after another against one prepared target. That is - the whole comparison in one command, and also the largest thing `bench` can spend, so - check it with `--dry-run` first. Name an arm (`--no-skill` / `--skill vN`, mutually - exclusive) to run just that one. -- Arms run sequentially; within an arm, runs go `max_concurrency` at a time. Each arm prints - its own tally, then a combined one. Completed runs are skipped per arm as usual, so adding - `skills/v3` and rerunning `acumen bench` benches only v3. -- A version in `skills/` that fails to load stops the pass at planning, before target prep - and before any spend, rather than being silently dropped from the comparison. -- `--split` and `--task` are repeatable; omitted means all. They apply to every arm. -- `--dry-run` prints the planned matrix and exits **before** target prep and before any - agent runs — free, and the right way to check the size of a pass. It covers the same arms - the real run would, with per-arm counts and a total. -- `--replicates` / `--max-concurrency` override the config for this pass only. -- `--auth auto` (default) prefers a stored account/subscription login, then falls back to - metered API credentials. Use `--auth session` or `--auth api` to require one explicitly. - Each run records the resolved mode; under `session`, `inferred_cost_usd` is the equivalent - API-rate cost computed from tokens, not money charged to the subscription. -- Ctrl-C is safe: completed runs are preserved and the next invocation resumes. -- Provider account/session usage exhaustion or an API account without credit invalidates the - pass rather than counting against the agent. Acumen prints the provider error, cancels the - remaining cells for that provider, lets every other provider finish its running and queued - cells, exits non-zero, and records the triggering cell as `valid: false` for diagnosis. - Reports and `improve` refuse that evidence; after replenishing the credential, rerun the same - command and automatic resume retries invalid and cancelled cells. - -**Arm parity is the whole point.** Both arms get an identical prompt, tools, caps, and -environment; the only difference is that a skill arm copies the skill's content files into -`/.claude/skills//` for Claude or -`/.agents/skills//` for Codex, where project discovery finds it. -`meta.json` is never copied. Each run gets a fresh empty sandbox with the target venv on -PATH, a throwaway `HOME` and `CLAUDE_CONFIG_DIR` — no repo source, no user settings, no -`CLAUDE.md` memories, no visibility of other runs. - -## The run tree - -``` -runs/{arm}/{split}/{model}/{task_id}/rep_{n}/ -``` - -`arm` is `noskill` or `skill_v1`, `skill_v2`, …; `model` is slugified. Each completed leaf -holds five files: - -| File | What | -|---|---| -| `answer.md` | What the agent wrote — the only thing graded | -| `script.py` | The agent's reproduction script (absent if it ran no code) | -| `transcript.jsonl` | The provider's own transcript — SDK-native for Claude, the `codex exec` event stream for Codex | -| `transcript.html` | Rendered transcript, written for both providers | -| `result.json` | The unit of record — written **last** | - -`result.json` presence + non-zero size is what marks a run complete, which is what makes -resume safe. Useful fields: `success`, `reason`, `answer`, `expected`, `model`, `turns`, -`cost_usd`, `input_tokens`, `output_tokens`, `duration_s`, `skill_hash`, `skill_name`, -`skill_loaded`, `pkg_version`, `commit`, `session_id`, `subtype`, `valid`, `error`. - -**`cost_usd` is inferred from tokens, and equals `inferred_cost_usd`.** Each run also records -`provider_cost_usd` (Claude's SDK total, API-equivalent rather than necessarily an invoice -charge under session authentication; Codex reports no dollars), `cost_source`, and the provider -figure's absolute/relative distance from the inferred one when both exist. Inference is the -canonical basis because both providers report tokens and the rates are frozen into the result, -so the figure is reproducible and comparable across providers. Token classes and the rates in -`price_rates` remain available for reproduction, including Claude's separate five-minute and -one-hour cache writes plus the legacy aggregate. A run without rates leaves `cost_usd` and -`inferred_cost_usd` null and `cost_available` false even when the provider supplied a value: -that figure would put one run on a basis no other run in the pass is on. - -**No rates ship with the package.** They are read from the providers' pricing pages, because a -compiled-in table is wrong from whatever date prices next move, and each run's cost is frozen -when written rather than corrected later. `bench` resolves rates before it spends anything and -exits 2 if the pages cannot be read. Each run records `price_source` (`config` or `fetched`) and -`price_rates_as_of` next to the rates themselves, so passes months apart stay individually -attributable and one report can mix them; the report flags arms priced on different dates, whose -cost gap includes the price change rather than only the skill's effect. - -`draft`, `improve`, `tasks`, and `ship` also fetch, but degrade to unpriced rather than failing: -their cost line is progress reporting. Note that Codex's `max_usd` is derived from these rates, -so an unpriced Codex run has no enforceable budget cap — bound it with `max_turns`. - -Inspect today's rates with `acumen prices`. Set `prices:` in `config.yaml` to price a model the -providers don't publish or to pin negotiated rates, which outrank a fetch; `acumen prices ---refresh` reports pins that have drifted from the published price. - -Reports, the run table and the console all show that inferred value, so Claude and Codex stay -on the same pricing basis even when Claude supplies an SDK estimate. The sidecar CSV carries -`cost_usd` (the figure the charts use), `inferred_cost_usd`, and the provider's own figure as -`recorded_cost_usd`. Unpriced runs are dropped from cost charts and comparisons, and the report -opens with a warning naming the models that had no rates. - -`codex exec` has no cap of its own, so acumen enforces both from its event stream, at -different resolutions. `max_turns` really does stop the run — counted in completed model -actions (message, command, file change, tool/search call), because one `codex exec` is a -single Codex turn however much work happens inside it. `max_usd` only marks the outcome: -Codex reports usage once, when the turn ends, so the breach is visible after the spend, not -before. The run is recorded `budget` either way; the CLI warns before the pass. A turn-capped -Codex run is stopped mid-turn and so records no usage — a failure with zero tokens. - -## Reasons - -| `reason` | Meaning | -|---|---| -| `ok` | Exact match — the only success | -| `wrong_answer` | Content differs | -| `format_error` | Content would match but formatting broke the exact match (bold, quotes, "Answer:", code fence, trailing period) | -| `no_answer_file` | The agent never wrote `answer.md` | -| `budget` / `max_turns` | Cap breached — a **failure regardless** of what `answer.md` contains | -| `error` | The agent crashed or produced no result | -| `provider_exhausted` | Provider usage/credit ran out; infrastructure-invalid, never agent evidence | - -A run of `format_error`s means the harness's answer-format instruction is losing to the -task prompt; a run of `no_answer_file` usually means the task is too big for `max_turns`. - -## Diagnosing a disappointing arm - -1. `skill loaded in 0/N runs` on a skill arm → the arm measured nothing. The skill's - `description` is what decides whether it loads; make it name the goals the task prompts - actually phrase. -2. The Skill tool firing in *baseline* runs is also flagged — that baseline is not clean. -3. Open `runs/.../transcript.html` for a failing run, or re-run one cell with - `--task ID --split train --replicates 1 --keep-sandboxes` and inspect the sandbox. -4. Compare `answer` vs `expected` in `result.json` before blaming the skill — a systematic - near-miss is a task-authoring problem, not a skill problem. - -## `acumen report` - -```bash -acumen report [--runs runs] [--tasks tasks.yaml] [--skills skills] [--out report.html] - [--palette claude-opus-5=#3b7ea1]... -``` - -- Writes `report.html` **and a sidecar `report.csv`** next to it, overwriting both; the - report always reflects every run currently on disk across every arm. -- Reads only `result.json` files — never transcripts. Fails if `runs/` holds none. -- **The figures show the TEST split** (the held-out measure). The per-run table below them - lists both splits. -- Sections: Overview (success rate, tokens, cost, time per arm, bars coloured by model), - Per-task breakdown, Runs table (links to each `transcript.html`), and — when `--skills` - resolves — Skill versions with each version's rationale and diff against its parent. -- `--tasks` / `--skills` are optional; missing ones only drop their sections (it says so). -- `--palette MODEL=COLOUR` is repeatable and accepts comma-separated pairs. A key that - matches no model in the data, or an unparseable colour, is rejected up front. - -Run `acumen report` after every bench — it is how a version is judged, and the only view -that compares arms and splits side by side. Then take the results to the user: say what -improved, what did not, and what you would do next (`improve` and re-bench, more tasks, -`ship`, or stop). Train improving while test does not is overfitting — that gap is exactly -what the split exists to expose — but the call on when to stop and what to ship is the -user's. diff --git a/src/acumen/_skills/data/references/python-api.md b/src/acumen/_skills/data/references/python-api.md deleted file mode 100644 index 4f58cda..0000000 --- a/src/acumen/_skills/data/references/python-api.md +++ /dev/null @@ -1,112 +0,0 @@ -# Python API - -Everything is re-exported from the top level: `from acumen import build_report, load_config, …`. -The CLI (`acumen.cli:main`) is a thin shell over it, so anything the CLI does is scriptable. -`main(argv)` returns an exit code and converts the library's error types into `error: …` on -stderr — call the functions directly if you want the exceptions. - -## Loading config and tasks - -```python -from pathlib import Path -from acumen import load_config, load_tasks, parse_config, parse_tasks - -cfg = load_config(Path("config.yaml")) # validates; resolves a local repo relative to the file -tasks = load_tasks(Path("tasks.yaml")) # list[Task]; task.split("train") -> TaskSplit(prompt, answer) -``` - -`Config` is a frozen dataclass — override with `dataclasses.replace(cfg, n_replicates=1)`. -`parse_config` / `parse_tasks` take already-parsed dicts, for building them in memory. - -## Preparing the target - -```python -from acumen import prepare_target -from acumen.env import DEFAULT_CACHE_ROOT - -target = prepare_target(cfg, DEFAULT_CACHE_ROOT, refresh=False) -target.python # venv interpreter with the package installed -target.bin_dir # what goes on an agent's PATH -target.fingerprint # " ", as recorded in result.json -``` - -Needs `uv` on PATH. Cached by (repo, ref) under `~/.cache/acumen`. - -## Planning and running a pass - -```python -import asyncio -from acumen import build_matrix, pending, run_matrix, load_skill, summarize - -skill = load_skill(Path("skills"), "v1", expect_name=cfg.skill_name) # None for the baseline -planned = build_matrix(cfg, tasks, skill="v1", splits=("train",), task_ids=["t1"]) -todo = pending(planned, Path("runs"), resume=True) -outcomes = asyncio.run( - run_matrix( - todo, - target=target, - runs_root=Path("runs"), - max_concurrency=cfg.max_concurrency, - auth_mode="api", - skill=skill, - on_start=lambda p: None, - on_done=lambda o: None, - env_passthrough=cfg.env_passthrough, - ) -) -summarize(outcomes) # {"ok": 5, "wrong_answer": 1, ...} -``` - -`run_matrix` and `run_once` are coroutines. `run_once` records agent crashes as failed runs -rather than raising, so one bad run never kills a pass. `run_once` raises `ValueError` if -`key.arm` and the `skill` argument disagree — the arm is the source of truth. - -`build_matrix` (and `--dry-run`) is pure planning: no agents, no network, no cost. - -## Grading and paths - -```python -from acumen import grade_answer, grade_run, run_dir, parse_run_dir, is_complete, RunKey, arm_name - -grade_answer("**SPI1**", "SPI1") # Grade(success=False, reason='format_error', answer='**SPI1**') -run_dir(Path("runs"), RunKey(arm=arm_name("v1"), split="test", model="claude-opus-5", task_id="t1", rep=1)) -is_complete(d) # a non-empty result.json is what "done" means -``` - -## The meta-agents - -`draft_skill`, `improve_skill`, `generate_tasks`, `ship_skill` are all coroutines taking -keyword-only args (`cfg=`, `target=`, plus their own roots) and returning a result dataclass -(`DraftResult`, `ImproveResult`, `TaskGenResult`, `ShipResult`) carrying the new `Skill`, -`cost_usd`, `turns`, and log paths. `max_turns`/`max_usd` default to `None` = **unbounded**. -Pass a `LiveLog` as `log=` for the JSONL feed: - -```python -from acumen import LiveLog, draft_skill - -log = LiveLog.open(Path("logs"), "draft", stream=False) -with log: - result = asyncio.run(draft_skill(cfg=cfg, target=target, skills_root=Path("skills"), auth_mode="session", log=log)) -``` - -## Aggregating results - -```python -from acumen import load_results, arm_metrics, build_report - -df = load_results(Path("runs")) # one row per result.json, + total_tokens, arm_label -arm_metrics(df[df["split"] == "test"]) # per-arm rate, stderr, tokens, cost, time, n -report = build_report(Path("runs"), Path("report.html"), tasks, skills_root=Path("skills")) -report.n_runs, report.results # the DataFrame behind the HTML -``` - -`build_report` writes `report.html` **and `report.csv`** (same stem) and returns `Report`. -Filter `df` to a split yourself before `arm_metrics` — it does not. - -## Introspection helpers - -`available_versions`, `latest_version`, `next_version`, `skill_hash`, `skill_content` -(for diffing), `installer_exists(src_dir)`, `collect_train_runs(runs_root, arm, tasks)`, -and the two pure guards `find_test_access` / `find_skill_access` (testable without an agent). -`scrubbed_env` / `build_agent_env` / `sandbox` / `install_skill` let you reproduce a run's -exact isolated environment by hand. diff --git a/src/acumen/_skills/data/references/setup.md b/src/acumen/_skills/data/references/setup.md deleted file mode 100644 index 67f9eb8..0000000 --- a/src/acumen/_skills/data/references/setup.md +++ /dev/null @@ -1,109 +0,0 @@ -# Project setup: `config.yaml` and `tasks.yaml` - -`acumen init [--dir DIR] [--force]` writes both files as annotated placeholders. It refuses -to clobber either unless `--force`. Both loaders are **strict: unknown keys are rejected**. - -Nothing else can run until `config.yaml` is filled in: the scaffold ships -`repo: https://github.com/OWNER/REPO` and a `REPLACE_ME` example task. Ask the user for the -values only they know (target repo/path, ref, extras, which models, budgets) instead of -inventing them. - -## config.yaml - -Only `repo` is required; delete a line to take its default. - -| Key | Default | Notes | -|---|---|---| -| `repo` | — | GitHub URL (`https://`, `git@`, `ssh://`, `git://`) or a local path. A local path is resolved relative to `config.yaml` and must exist. | -| `ref` | `main` | Branch/tag/commit; ignored for local paths. | -| `extras` | `[]` | Extras the target publishes in `[project.optional-dependencies]`, e.g. `[test]`. | -| `dependency_groups` | `[]` | PEP 735 groups from the target's `[dependency-groups]`, e.g. `[full]`. | -| `pip_packages` | `[]` | Packages the target declares nowhere, installed alongside it. PEP 508 specifiers allowed (`numpy<2`). | -| `python` | `"3.12"` | Interpreter for the target's venv (quote it — `3.10` unquoted is a float). | -| `env_passthrough` | `[]` | Extra env var names agents may keep. The agent env is a clean allowlist (auth/proxy/TLS only); **everything else from your shell is blanked**. A target needing `OMP_NUM_THREADS`, `R_HOME`, a service key, etc. must name it here. | -| `models` | `[claude-opus-5]` | Benchmark models. Claude (`claude*`) and Codex (`gpt-*`, `o1`/`o3`/`o4`, `codex-*`) models may be mixed. Duplicates rejected. `models[0]` is the default for the four `*_model` keys below. | -| `n_replicates` | `3` | Runs per (model, task, split) cell. | -| `max_concurrency` | `4` | Simultaneous benchmark agents. | -| `max_turns` | `40` | **Benchmark agents only.** | -| `max_usd` | `3.0` | **Benchmark agents only.** | -| `draft_model` / `improve_model` / `tasks_model` / `ship_model` | `models[0]` | Meta-agent models; overridable per command with `--model`. | -| `skill_name` | repo basename, slugified + lowercased | Must equal the `name:` in the skill's frontmatter. | -| `prices` | built-in table | Per-model token rates (USD per million): `{model: {input, output, cached_input?, cache_write?, cache_write_5m?, cache_write_1h?}}`. Overrides or extends acumen's table — needed for a model it doesn't ship a rate for, a gateway, or negotiated rates. See `acumen prices`. | - -**`extras` vs `dependency_groups` — pick the wrong one and the packages are missing.** Extras -are published in package metadata, so they are what `pip install pkg[name]` resolves. PEP 735 -groups live only in the source tree and are invisible to anyone installing from PyPI. Rule of -thumb: if `pip install [name]` from PyPI would not work, `name` is a group. Check the -target's `pyproject.toml` rather than guessing — many scverse packages declare **no** extras -and put their optional stack in a `full` group. Asking for the wrong kind is now a hard error -naming the right key, but only after the checkout, so read the file first. - -Changing `extras`, `dependency_groups` or `pip_packages` changes the cache key, so the venv -rebuilds on the next command — no `--refresh-target` needed. - -The scaffold's `models:` line lists three Claude and three Codex models. With -`n_replicates: 3`, one task expands to 36 runs per arm, so cut the matrix down before -the first real pass. - -## tasks.yaml - -```yaml -tasks: - - id: some_analysis # unique, filesystem-safe: [A-Za-z0-9._-]; used as a path component - train: - prompt: >- - One paragraph: the goal, the input, and exactly what to report. - answer: ONE_TOKEN - test: - prompt: >- - The same analysis on a different input / target. - answer: ANOTHER_TOKEN - # optional per-task overrides — the only other allowed keys: - # max_turns: 60 - # max_usd: 5.0 - # model: claude-sonnet-5 -``` - -Both `train` and `test` are required, each with non-empty `prompt` and `answer`. Both -splits always run in a pass; only train results ever reach `improve`. - -### Writing a task that measures anything - -- **Answers are graded by exact string match after `strip()`, case-sensitive.** Keep the - answer to one token: a name, a category, a count, a number at a stated precision. End the - prompt by stating exactly what to report and in what form. -- **Do not name the target package**, and do not mention a version — the harness preamble - already tells the agent the package is installed and which interpreter to use. -- **State the goal, not the recipe.** No numbered steps, no function/argument names, no - description of the data's shape or columns. What is being measured is whether the agent - can find the "how" itself; a prompt that spells out the calls measures nothing. -- Train and test must be **the same analysis on different inputs**, with different correct - answers — otherwise a skill can pass by memorizing one answer. -- A good task is one the target package solves and the **no-skill baseline gets wrong** — - a task the baseline already passes leaves no room to show a gain. The `--no-skill` arm is - what tells you; it can be run before or after the first skill exists. -- Renaming an `id` orphans every existing run under it. - -## `acumen tasks` — generate tasks.yaml - -```bash -acumen tasks [--out tasks.yaml] [--force] [--feedback "skip the plotting API"] \ - [--model M] [--max-turns N] [--max-usd X] [--stream] [--log-dir logs] -``` - -One autonomous agent that reads the package source **and executes it in the target venv**: -every ground-truth answer comes from a script it actually ran, not from docs. It enumerates -the package's tutorials/vignettes and writes at least one task per tutorial, so expect a -sizeable file. Unbounded in turns and cost by default. - -- Refuses to overwrite an existing `--out` without `--force` (the check runs *before* the - costly target prep). -- **There is no append mode.** The generator never reads your existing tasks file, so - appending would silently duplicate. To combine, generate to a separate path and merge by - hand. -- Existing skills and agent-instruction files (`SKILL.md`, `.agents/`, `.claude/`, `.codex/`, `CLAUDE.md`, - `AGENTS.md`, `.cursor/`, root `skills/`) are stripped from the source copy it reads and - blocked by a hook, so pre-written guidance cannot bias which tasks it picks. -- The output is validated through the strict tasks loader before it is written — it can - never emit a file the rest of the pipeline would reject. -- **Review what it produces.** The answers are only as good as the scripts it ran. diff --git a/src/acumen/_skills/data/references/ship.md b/src/acumen/_skills/data/references/ship.md deleted file mode 100644 index 0a4bd51..0000000 --- a/src/acumen/_skills/data/references/ship.md +++ /dev/null @@ -1,53 +0,0 @@ -# `acumen ship` — put a proven skill inside the package - -```bash -acumen ship --skill v2 [--force] [--model M] [--max-turns N] [--max-usd X] - [--stream] [--log-dir logs] [--auth auto|session|api] -``` - -`--skill VERSION` is **required** — there is no implicit "latest" or "best". Run it only -after the report shows that version earning its place *and* the user has agreed to ship it: -it writes to the target's real checkout and can open a PR. - -## What it produces - -An agent modifies the target checkout so the package gains a `-install-skills` -console script. The package's users then run, e.g.: - -```bash -scanpy-install-skills --agent claude # or codex | agents | claude-science -scanpy-install-skills --dest ./somewhere # explicit destination -scanpy-install-skills --print-path # where the bundled skill lives -``` - -There is **no default framework** — `--agent` or `--dest` is required. The same -`SKILL.md` + `references/` bundle installs verbatim into every framework; there is no -per-framework conversion. Destinations are `/skills/`, with roots -`~/.claude` (`CLAUDE_CONFIG_DIR`), `~/.codex` (`CODEX_HOME`), `~/.agents`, and the active -org's directory for `claude-science`. - -Inside the package the agent creates `/_skills/` with `install.py` (a canonical -template it must copy verbatim) and `data/` holding the skill files, then wires the console -script and — the step that silently fails — the **build-backend packaging** so those -non-`.py` data files actually ship in the wheel. It verifies by building a wheel, installing -it into a fresh (non-editable) venv, and confirming `SKILL.md` lands. - -## Delivery depends on `repo` - -- **GitHub URL** → the agent branches, commits, pushes, and opens a PR with `gh` (it does - not merge). Needs write access; a rejected push is reported, not worked around. -- **Local path** → the change is written straight into your working tree, no branch, no - commit. Review with `git diff`. - -## Non-obvious behaviour - -- **The ship agent is deliberately NOT isolated.** Unlike every other acumen agent it runs - in your real environment: real `HOME`, real network, real git/`gh` credentials, real `uv`, - with `permission_mode="bypassPermissions"`. Only the model credential is constrained (by - `--auth`). Run it when you are ready for it to touch your checkout and remote. -- It refuses to run if the target already declares a `*-install-skills` console script or - has a `_skills/install.py` — pass `--force` to ship anyway. -- It ships exactly one version, copied verbatim; it never authors or edits skill text, and - it writes no tests. -- Unbounded turns/cost by default. Its final summary (PR URL, packaging changes, - build-verify result) is printed and returned. diff --git a/src/acumen/_skills/install.py b/src/acumen/_skills/install.py deleted file mode 100644 index 1ea990a..0000000 --- a/src/acumen/_skills/install.py +++ /dev/null @@ -1,175 +0,0 @@ -"""Install the bundled ``acumen`` skill into an agentic framework's skills directory. - -Console script (wired as ``-install-skills`` in ``pyproject.toml``): copy the skill that -ships inside this package into a chosen framework's skills directory, so an agent can load it. -The same ``SKILL.md`` + ``references/`` bundle is a cross-framework standard, so it installs -verbatim — no per-framework conversion. - -Frameworks (``--agent``): - -- ``claude`` -> ``~/.claude/skills`` (honours ``CLAUDE_CONFIG_DIR``) -- ``codex`` -> ``~/.codex/skills`` (honours ``CODEX_HOME``) -- ``agents`` -> ``~/.agents/skills`` -- ``claude-science`` -> the active org's skills dir, resolved from - ``~/.claude-science/active-org.json`` - -``--dest`` overrides all of them. There is **no default framework**: pass ``--agent`` or -``--dest``. The skill files live in the ``data/`` directory next to this module and are read via -``importlib.resources``, so this works from an installed wheel, not just an editable checkout. -""" - -from __future__ import annotations - -import argparse -import json -import os -import shutil -import sys -from importlib import resources -from pathlib import Path - -#: The skill name; it installs to ``//``. -SKILL_NAME = "acumen" - -#: Framework -> (env var that overrides the config root, default config root). The skills -#: directory is ``/skills``. ``claude-science`` is resolved separately. -_AGENT_ROOTS = { - "codex": ("CODEX_HOME", "~/.codex"), - "claude": ("CLAUDE_CONFIG_DIR", "~/.claude"), - "agents": (None, "~/.agents"), -} - -#: The frameworks ``--agent`` accepts. -AGENTS = (*sorted(_AGENT_ROOTS), "claude-science") - - -def source_dir() -> Path: - """Return the package-owned skill directory (the bundle that gets copied).""" - source = Path(str(resources.files(__package__).joinpath("data"))) - if not (source / "SKILL.md").is_file(): - raise RuntimeError(f"packaged skill data is missing: {source}") - return source - - -def _claude_science_skills_dir() -> Path: - """Resolve the active Claude Science org's skills directory.""" - root = Path("~/.claude-science").expanduser() - active_org_path = root / "active-org.json" - try: - active_org = json.loads(active_org_path.read_text(encoding="utf-8")) - except (OSError, json.JSONDecodeError) as error: - raise ValueError( - f"cannot resolve the Claude Science active organization from {active_org_path}; pass --dest instead" - ) from error - org_uuid = active_org.get("org_uuid") if isinstance(active_org, dict) else None - if not isinstance(org_uuid, str) or not org_uuid or Path(org_uuid).name != org_uuid or org_uuid in {".", ".."}: - raise ValueError(f"invalid Claude Science org_uuid in {active_org_path}; pass --dest instead") - return root / "orgs" / org_uuid / "skills" - - -def _skills_dir(agent: str) -> Path: - """Return the skills directory (parent of the install dir) for a framework.""" - if agent == "claude-science": - return _claude_science_skills_dir() - variable, fallback = _AGENT_ROOTS[agent] - root = os.environ.get(variable) if variable is not None else None - return Path(root or fallback).expanduser() / "skills" - - -def resolve_dest(agent: str | None, dest: Path | None) -> Path: - """Resolve the install destination from ``--agent`` / ``--dest``. - - ``--dest`` wins. With neither, raise ``ValueError`` — there is no default framework. - """ - if dest is not None: - return dest.expanduser() - if agent is None: - raise ValueError("pass --agent {" + ",".join(AGENTS) + "} or --dest to choose where to install") - return _skills_dir(agent) / SKILL_NAME - - -def _snapshot(root: Path) -> dict[str, bytes]: - """Map each file under ``root`` to its bytes, for exact tree comparison.""" - return {str(item.relative_to(root)): item.read_bytes() for item in root.rglob("*") if item.is_file()} - - -def _matches(source: Path, target: Path) -> bool: - """Report whether ``target`` is a byte-for-byte copy of ``source``.""" - return target.is_dir() and _snapshot(source) == _snapshot(target) - - -def main(argv: list[str] | None = None) -> int: - """Install the bundled skill; entry point for the ``-install-skills`` script.""" - parser = argparse.ArgumentParser( - description=f"Install the {SKILL_NAME} skill bundled with this package into an agent's skills directory.", - ) - parser.add_argument( - "--agent", - choices=AGENTS, - default=None, - help="framework to install into (no default — pass this or --dest)", - ) - parser.add_argument( - "--dest", - type=Path, - default=None, - help="exact skills directory to install into (overrides --agent)", - ) - parser.add_argument( - "--force", - action="store_true", - help="overwrite an existing installation that differs from the bundled skill", - ) - action = parser.add_mutually_exclusive_group() - action.add_argument( - "--check", - action="store_true", - help="report whether the installed skill matches the bundled one; do not install", - ) - action.add_argument( - "--print-path", - action="store_true", - help="print the bundled skill's location inside the package and exit", - ) - args = parser.parse_args(argv) - - try: - source = source_dir() - if args.print_path: - print(source) - return 0 - if args.check: - target = resolve_dest(args.agent, args.dest) - if not target.exists(): - print(f"{SKILL_NAME} skill is not installed at {target}", file=sys.stderr) - return 1 - if _matches(source, target): - print(f"{SKILL_NAME} skill at {target} matches the bundled copy") - return 0 - print(f"{SKILL_NAME} skill at {target} differs from the bundled copy", file=sys.stderr) - return 1 - - dest = resolve_dest(args.agent, args.dest) - if dest.exists(): - if _matches(source, dest): - print(f"{SKILL_NAME} skill already up to date at {dest}") - return 0 - if not args.force: - print(f"{dest} already exists and differs; pass --force to overwrite", file=sys.stderr) - return 1 - if dest.is_dir(): - shutil.rmtree(dest) - else: - dest.unlink() - dest.parent.mkdir(parents=True, exist_ok=True) - shutil.copytree(source, dest) - except (OSError, RuntimeError, ValueError) as error: - print(f"error: {error}", file=sys.stderr) - return 1 - - print(f"installed the {SKILL_NAME} skill to {dest}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/src/acumen/agents.py b/src/acumen/agents.py index 2ae4dd8..7ea8271 100644 --- a/src/acumen/agents.py +++ b/src/acumen/agents.py @@ -17,10 +17,12 @@ import asyncio import importlib.util import json +import re import shlex import shutil import signal import sys +import tempfile import time from collections.abc import Callable, Sequence from contextlib import aclosing, suppress @@ -119,6 +121,13 @@ class AgentOptions: #: agent. The shipper sets it False: it is the one agent that runs in the operator's real #: environment on purpose, and it needs the git and ``gh`` credentials that live there. confine: bool = True + #: When set, install a ``PreToolUse`` source guard (:func:`acumen.scrub.make_source_guard` on + #: Claude, its Codex equivalent) that denies fetching the target's own repository or source + #: distribution. ``block_repo`` is the target's remote URL (``None`` for a local target, whose + #: source is not fetchable anyway); ``block_pkg`` is the installed package name. Set only for + #: benchmark runs; meta-agents and ``ship`` leave both ``None``. + block_repo: str | None = None + block_pkg: str | None = None stderr: Callable[[str], None] | None = None #: Prices one provider usage block in USD. Claude enforces ``max_usd`` itself against the #: figure it bills; Codex reports tokens and no dollar amount, so enforcing a budget there @@ -247,21 +256,28 @@ def _claude_hooks(options: AgentOptions) -> dict[str, Any]: alongside containment. """ hooks: dict[str, Any] = {key: list(value) for key, value in (options.claude_hooks or {}).items()} - if not options.confine: - return hooks - - from acumen.guard import containment_hook - - reads, _ = _access_roots(options) - agent_home = options.env.get("HOME") - guard = containment_hook( - reads, - options.deny_paths, - cwd=options.cwd, - home=Path(agent_home) if agent_home else None, - ) - hooks.setdefault("PreToolUse", []) - hooks["PreToolUse"] = [guard, *hooks["PreToolUse"]] + pre = list(hooks.get("PreToolUse", [])) + + if options.block_repo or options.block_pkg: + from acumen.scrub import make_source_guard + + pre.insert(0, make_source_guard(options.block_repo, options.block_pkg)) + + if options.confine: + from acumen.guard import containment_hook + + reads, _ = _access_roots(options) + agent_home = options.env.get("HOME") + guard = containment_hook( + reads, + options.deny_paths, + cwd=options.cwd, + home=Path(agent_home) if agent_home else None, + ) + pre.insert(0, guard) + + if pre: + hooks["PreToolUse"] = pre return hooks @@ -520,7 +536,7 @@ def _codex_command(options: AgentOptions, prompt: str) -> list[str]: "--model", options.model, ] - if options.deny_paths: + if options.deny_paths or options.block_repo or options.block_pkg: command.append("--dangerously-bypass-hook-trust") reads, writes = _access_roots(options) # The Linux command sandbox re-executes the Codex binary through bubblewrap. @@ -531,6 +547,10 @@ def _codex_command(options: AgentOptions, prompt: str) -> list[str]: # evaluates the command path it is given, so allowing /usr/bin does not make # an invocation through the /bin -> /usr/bin symlink readable. reads.extend(path for path in (Path("/bin"), Path("/lib"), Path("/lib64")) if path.exists()) + if options.block_repo or options.block_pkg: + # The source guard imports acumen.scrub; make the package importable inside the sandbox + # regardless of install layout (an editable install lives outside the venv prefix). + reads.append(_acumen_package_root()) reads = list(dict.fromkeys(reads)) command.extend(("-c", 'default_permissions="acumen"')) filesystem = {":root": "deny", ":minimal": "read"} @@ -556,9 +576,26 @@ def _codex_command(options: AgentOptions, prompt: str) -> list[str]: return command -def _codex_guard_source(denied: Sequence[Path]) -> str: - """Return a standalone Codex PreToolUse guard for absolute denied roots.""" +def _acumen_package_root() -> Path: + """Directory containing the importable ``acumen`` package (its parent on ``sys.path``).""" + import acumen + + return Path(acumen.__file__).resolve().parent.parent + + +def _codex_guard_source(denied: Sequence[Path], *, repo: str | None = None, pkg_name: str | None = None) -> str: + """Return a standalone Codex PreToolUse guard. + + Denies two things, either optional: any path resolving under ``denied`` (filesystem + containment), and — via :func:`acumen.scrub.find_source_fetch`, imported at run time because + the guard executes under acumen's own interpreter — any attempt to fetch the target's own + source repository/distribution (``repo``/``pkg_name``). Mirrors the Claude-side + :func:`acumen.scrub.make_source_guard` so both providers enforce the same rule. + """ roots = repr([str(path.resolve()) for path in denied]) + repo_repr = repr(repo) + pkg_repr = repr(pkg_name) + acumen_root_repr = repr(str(_acumen_package_root())) return f"""\ import json import shlex @@ -566,6 +603,8 @@ def _codex_guard_source(denied: Sequence[Path]) -> str: from pathlib import Path ROOTS = [Path(value) for value in {roots}] +REPO = {repo_repr} +PKG = {pkg_repr} def blocked(value, cwd): @@ -606,30 +645,50 @@ def strings(value): yield from strings(item) +def deny(reason): + print(json.dumps({{ + "hookSpecificOutput": {{ + "hookEventName": "PreToolUse", + "permissionDecision": "deny", + "permissionDecisionReason": reason, + }} + }})) + raise SystemExit(0) + + payload = json.load(sys.stdin) cwd = Path(payload.get("cwd") or ".").resolve() -for value in strings(payload.get("tool_input") or {{}}): +tool_input = payload.get("tool_input") or {{}} +for value in strings(tool_input): hit = blocked(value, cwd) if hit is not None: - print(json.dumps({{ - "hookSpecificOutput": {{ - "hookEventName": "PreToolUse", - "permissionDecision": "deny", - "permissionDecisionReason": "acumen blocks access to isolated benchmark data: " + hit, - }} - }})) - raise SystemExit(0) + deny("acumen blocks access to isolated benchmark data: " + hit) + +if REPO or PKG: + if {acumen_root_repr} not in sys.path: + sys.path.insert(0, {acumen_root_repr}) + try: + from acumen.scrub import find_source_fetch + except Exception: + find_source_fetch = None + if find_source_fetch is not None: + hit = find_source_fetch(payload.get("tool_name", ""), tool_input, repo=REPO, pkg_name=PKG) + if hit is not None: + deny( + "acumen benchmarks against the installed package only; fetching the target's " + "source repository or distribution is not permitted: " + str(hit) + ) """ def _install_codex_guard(options: AgentOptions) -> None: """Install a trusted, run-local Codex guard when isolation needs a deny boundary.""" - if not options.deny_paths: + if not options.deny_paths and not (options.block_repo or options.block_pkg): return codex_home = Path(options.env["CODEX_HOME"]) script = codex_home / "acumen_guard.py" script.parent.mkdir(parents=True, exist_ok=True) - script.write_text(_codex_guard_source(options.deny_paths)) + script.write_text(_codex_guard_source(options.deny_paths, repo=options.block_repo, pkg_name=options.block_pkg)) hooks_dir = options.cwd / ".codex" hooks_dir.mkdir(parents=True, exist_ok=True) hooks = { @@ -696,6 +755,54 @@ def _sandbox_failure(lines: Sequence[str]) -> str | None: return None +async def codex_sandbox_probe(env: dict[str, str], *, timeout: float = 30.0) -> str | None: + """Return ``None`` if Codex's command sandbox can start here, else an actionable error string. + + Runs ``codex sandbox -- true`` — no model call, so it is free and deterministic — and reads + its stderr for a namespace/sandbox-init failure. Codex runs every benchmark command inside + this sandbox, so if it cannot start, no Codex run can save an answer; catching it in preflight + turns a paid, per-run ``error_sandbox`` into one free up-front message. + + Conservative on purpose: a probe that cannot run, times out, or exits non-zero for a reason + that is *not* a recognised sandbox failure returns ``None`` (let the run proceed) rather than + block a setup that might work. Only a matched sandbox-init failure is reported. + """ + cli = shutil.which("codex", path=env.get("PATH")) + if cli is None: + return None # a missing CLI is reported by check_agent_cli; nothing to probe here + # Run in a throwaway empty directory with the read-only profile. This tests only the one thing + # that fails on a locked-down host — whether Codex can create its namespace sandbox at all — + # without the workspace profile's git-protection, which mounts a tmpfs over the workspace's + # ``.git`` and errors in a non-repo directory (a false positive unrelated to the real runs, + # which use their own filesystem profile in a temp sandbox dir). + probe_dir = tempfile.mkdtemp(prefix="acumen-codex-probe-") + argv = [cli, "sandbox", "-c", 'default_permissions=":read-only"', "--", "true"] + proc: asyncio.subprocess.Process | None = None + try: + proc = await asyncio.create_subprocess_exec( + *argv, cwd=probe_dir, env=env, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE + ) + _, stderr = await asyncio.wait_for(proc.communicate(), timeout=timeout) + except (OSError, TimeoutError): + if proc is not None: + with suppress(ProcessLookupError): + proc.kill() + return None + finally: + shutil.rmtree(probe_dir, ignore_errors=True) + if proc.returncode == 0: + return None + hit = _sandbox_failure(stderr.decode("utf-8", "replace").splitlines()) + if hit is None: + return None # non-zero for some other reason — not a namespace block + return ( + "codex's command sandbox could not start on this system, so no codex run can execute its " + f"commands or save an answer ({hit}). codex isolates every benchmark command inside an " + "unprivileged namespace sandbox, which this host does not currently permit. Enable " + "unprivileged user namespaces for codex, or run acumen where they are allowed." + ) + + def _is_turn_item(event: dict[str, Any]) -> bool: """Whether ``event`` is one completed model action — the unit Codex turns are counted in.""" if event.get("type") != "item.completed": @@ -918,6 +1025,21 @@ def _codex_terminal( ) +# ``codex exec`` prints this to stderr whenever its stdin is not a TTY (acumen wires it to +# /dev/null), then reads immediate EOF and appends an empty ```` block. It is benign +# but reads as a confusing prompt to anyone watching the run, so drop it before it surfaces. +_CODEX_STDIN_NOTICE = "Reading additional input from stdin" + +# Codex's own ``tracing`` logs, shaped ``Z codex_: ``. +# They report Codex-internal events (rollout writes after a session ends, commands the sandbox +# denies under ``approval_policy="never"``, the model's malformed ``apply_patch`` attempts) that +# are captured in the run transcript anyway and never signal an acumen fault — so they are noise +# on the console, and worse they corrupt the ``\r`` progress bars. Kept in the noise sink for +# sandbox-failure detection, but never echoed. Real bwrap/kernel sandbox errors are bare (no +# ``codex_`` prefix), so they still print and are still detected. +_CODEX_TRACE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T[\d:.]+Z\s+(?:ERROR|WARN|INFO|DEBUG|TRACE)\s+codex") + + async def _drain_stderr( stream: asyncio.StreamReader, callback: Callable[[str], None] | None, @@ -925,8 +1047,12 @@ async def _drain_stderr( ) -> None: while line := await stream.readline(): text = line.decode(errors="replace").rstrip("\r\n") + if _CODEX_STDIN_NOTICE in text: + continue if sink is not None: sink.append(text) + if _CODEX_TRACE_RE.match(text): + continue if callback is not None: callback(text) else: diff --git a/src/acumen/bench.py b/src/acumen/bench.py index 9e946f4..2c4d050 100644 --- a/src/acumen/bench.py +++ b/src/acumen/bench.py @@ -16,6 +16,12 @@ from acumen.skills import Skill from acumen.tasks import Task +#: Invalid reasons that stop the rest of a provider's cells for this pass: the credential is +#: empty (``provider_exhausted``) or the network is down (``connection_error``). Both would fail +#: every remaining cell the same way, so the pass raises ``BenchmarkInvalidError`` and the cells +#: stay pending for the next run to resume. ``sandbox_blocked`` is handled separately (pass-scoped). +_PROVIDER_SCOPED_STOP = frozenset({"provider_exhausted", "connection_error"}) + @dataclass(frozen=True) class PlannedRun: @@ -275,7 +281,7 @@ async def one(item: PlannedRun) -> RunOutcome | None: if peer is not asyncio.current_task() and not peer.done(): peer.cancel() return outcome - if outcome.reason == "provider_exhausted" and provider not in exhausted: + if outcome.reason in _PROVIDER_SCOPED_STOP and provider not in exhausted: exhausted[provider] = outcome current = asyncio.current_task() # Stop both queued and in-flight siblings for this provider. Tasks for the @@ -314,6 +320,13 @@ async def one(item: PlannedRun) -> RunOutcome | None: return outcomes +#: Budget cap for the auth-probe run. Generous on purpose: the probe is bounded by ``max_turns=5`` +#: and only confirms the credential works, but one turn carries the full ``claude_code`` system +#: prompt, whose input cost at a premium model's rate can exceed a tight cap and fail preflight +#: before the model replies. A trivial "reply ok" never approaches this. +PREFLIGHT_MAX_USD = 2.0 + + async def _preflight_model( model: str, *, @@ -323,7 +336,7 @@ async def _preflight_model( env_passthrough: Sequence[str] | None, ) -> str | None: """Return None if the model can authenticate, else a short error string.""" - from acumen.agents import AgentOptions, run_agent + from acumen.agents import AgentOptions, codex_sandbox_probe, run_agent from acumen.sandbox import sandbox provider = provider_for_model(model) @@ -335,14 +348,27 @@ async def _preflight_model( provider=provider, env_passthrough=env_passthrough, ) as box: + # Codex runs every command inside its own namespace sandbox. A trivial auth probe + # writes no files and runs no command, so it would pass even where that sandbox + # cannot start — and then every real run fails with error_sandbox after paying for + # the tokens. Test the sandbox itself first (free, no model call); report and stop + # here if it cannot start. + if provider == "codex": + sandbox_error = await codex_sandbox_probe(box.env) + if sandbox_error is not None: + return sandbox_error result = await run_agent( "Reply with only the word: ok", options=AgentOptions( cwd=box.root, env=box.env, model=model, + # Bounded by turns, not dollars: the probe only confirms the credential works, + # but one turn carries the full claude_code system prompt, whose input cost at a + # premium model's rate (Opus) exceeds a tight cap — so a dollar cap that low + # fails preflight before the model can reply. max_turns=5 already bounds it. max_turns=5, - max_usd=0.10, + max_usd=PREFLIGHT_MAX_USD, discover_skills=False, confine=False, ), diff --git a/src/acumen/check.py b/src/acumen/check.py index bcab39a..15b2293 100644 --- a/src/acumen/check.py +++ b/src/acumen/check.py @@ -18,7 +18,7 @@ throwaway environment an agent gets. Each one still gets a fresh empty working directory, so its ``answer.md`` cannot be another script's and nothing it writes lands in the project. -The scripts hold the ground truth for the **held-out test split**, so nothing may ever read +The scripts hold the ground truth for the **held-out valid split**, so nothing may ever read them into an agent. That holds today because ``bench``, ``draft`` and ``improve`` confine their agents to explicit read roots (:mod:`acumen.guard`) that never include the project directory. """ diff --git a/src/acumen/cli.py b/src/acumen/cli.py index a7ac24b..7d816fa 100644 --- a/src/acumen/cli.py +++ b/src/acumen/cli.py @@ -4,11 +4,15 @@ import argparse import asyncio +import itertools import json +import math import sys import tempfile +import threading import time -from collections.abc import Sequence +from collections.abc import Callable, Sequence +from contextlib import nullcontext from dataclasses import dataclass, replace from datetime import date from pathlib import Path @@ -29,12 +33,12 @@ summarize_checks, ) from acumen.config import Config, ConfigError, load_config -from acumen.draft import DraftError, draft_skill from acumen.env import DEFAULT_CACHE_ROOT, AuthMode, EnvError, prepare_target, resolve_auth_mode +from acumen.epoch import EpochPlan, completed_epochs, resolve_epoch from acumen.grade import INVALID_REASONS from acumen.improve import ImproveError, improve_skill from acumen.logs import LiveLog -from acumen.paths import SPLITS, arm_name +from acumen.paths import SPLITS, Split, arm_name from acumen.pricefeed import ( PRICE_SOURCES, PRICE_TIER, @@ -51,9 +55,19 @@ from acumen.runner import RunOutcome, StderrFilter from acumen.scaffold import InitError, is_scaffold_tasks, scaffold from acumen.ship import ShipError, ship_skill -from acumen.skills import Skill, SkillError, available_versions, latest_version, load_skill +from acumen.skills import Skill, SkillError, available_versions, latest_version, load_skill, skill_dir from acumen.taskgen import TaskGenError, generate_tasks from acumen.tasks import Task, TaskError, load_tasks +from acumen.training import ( + EpochRow, + best_version, + build_training_rows, + epochs_since_best, + is_perfect, + patience_exhausted, + write_training_csv, +) +from acumen.wiki import WikiError, collect_arm_runs, update_wiki def _add_bench_args(parser: argparse.ArgumentParser) -> None: @@ -285,6 +299,187 @@ def _fmt_cost(value: float | None) -> str: return f"${value:.2f}" if value is not None else "cost n/a" +def _fmt_rate(value: float | None) -> str: + """Format a 0..1 success rate as a percentage, or ``n/a`` when unknown.""" + if value is None or (isinstance(value, float) and math.isnan(value)): + return "n/a" + return f"{value:.0%}" + + +def _epoch_bar( + done: int, + total: int, + row: EpochRow | None, + *, + best: str | None, + patience: int, + since_best: int, + fixed: bool, +) -> str: + """A tqdm-style one-liner summarising an epoch: bar, version, train/valid success, patience.""" + width = 14 + filled = round(width * done / total) if total else width + bar = "█" * filled + "░" * (width - filled) + version = row.version if row is not None else "?" + train = _fmt_rate(row.train_success) if row is not None else "n/a" + valid = _fmt_rate(row.valid_success) if row is not None else "n/a" + parts = [f"Epoch {done}/{total} |{bar}| {version}", f"train {train}", f"valid {valid}"] + if row is not None and not (isinstance(row.valid_cost, float) and math.isnan(row.valid_cost)): + parts.append(f"${row.valid_cost:.2f}/run") + if not fixed and best is not None: + parts.append(f"(best {best}, patience {min(since_best, patience)}/{patience})") + return " ".join(parts) + + +# Progress rendering for `epoch`/`fit`. Three modes, resolved once per command: +# "bars" — tqdm-style single line rewritten in place with \r (a live terminal) +# "plain" — throttled fresh lines, no \r (output redirected to a file / CI) +# "verbose" — today's per-run scrolling logs, via _Progress (opt-in with --verbose) +_BAR_WIDTH = 14 +_SPINNER_FRAMES = "⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏" + + +def _progress_mode(args: argparse.Namespace) -> str: + """Pick the progress style: explicit --verbose wins, else live bars on a TTY, else plain.""" + if getattr(args, "verbose", False): + return "verbose" + if sys.stdout.isatty() and not getattr(args, "stream", False): + return "bars" + return "plain" + + +def _epoch_header(mode: str, n: int, total: int) -> None: + """Print the marker that starts an epoch — a heavy banner in verbose, a compact line else.""" + if mode == "verbose": + print(f"\n{'═' * 78}\nEPOCH {n}/{total}\n{'═' * 78}", flush=True) + else: + print(f"\nEpoch {n}/{total}", flush=True) + + +def _progress_bar(done: int, total: int, width: int = _BAR_WIDTH) -> str: + filled = width if not total else max(0, min(width, round(width * done / total))) + return "█" * filled + "░" * (width - filled) + + +class _PhaseBar: + r"""A tqdm-style bar for one bench or wiki phase of an epoch. + + Exposes the ``on_start``/``on_done(RunOutcome)`` callbacks ``run_matrix`` expects (bench) and + ``on_wiki_task(TaskWikiResult)`` for the wiki pass. In ``bars`` mode it rewrites one line with + ``\r``; in ``plain`` mode it prints throttled fresh lines so redirected logs stay readable. + (``verbose`` keeps :class:`_Progress` and never builds this, so only those two modes reach here.) + """ + + _THROTTLE_S = 5.0 + + def __init__(self, label: str, total: int, mode: str, *, track_success: bool = True) -> None: + self.label = label + self.total = total + self.mode = mode + self.track_success = track_success + self.done = 0 + self.passed = 0 + self.cost = 0.0 + self._t0 = time.monotonic() + self._last_print = 0.0 + if mode == "bars": + print("\r" + self._line(), end="", flush=True) + + @property + def _elapsed(self) -> float: + return time.monotonic() - self._t0 + + def set_total(self, total: int) -> None: + """Set the denominator once it is known (the wiki pass reports it via ``on_plan``).""" + self.total = total + if self.mode == "bars": + print("\r" + self._line(), end="", flush=True) + + def on_start(self, item) -> None: + """Present so it can be handed to ``run_matrix``; the bar only renders on completion.""" + + def on_done(self, outcome: RunOutcome) -> None: + payload = outcome.payload + priced = payload.get("cost_available", True) and payload.get("cost_usd") is not None + self._surface_error(outcome) + self._tick(success=outcome.success, cost=float(payload["cost_usd"]) if priced else None) + + def on_wiki_task(self, result) -> None: + self._tick(success=None, cost=result.cost_usd) + + def _tick(self, *, success: bool | None, cost: float | None) -> None: + self.done += 1 + if success: + self.passed += 1 + if cost is not None: + self.cost += cost + if self.mode == "bars": + print("\r" + self._line(), end="", flush=True) + elif self.done < self.total and self._elapsed - self._last_print >= self._THROTTLE_S: + self._last_print = self._elapsed + print(" " + self._line(), flush=True) + + def _line(self) -> str: + pct = round(100 * self.done / self.total) if self.total else 100 + parts = [f"{self.label:>14} {pct:>3}%|{_progress_bar(self.done, self.total)}| {self.done}/{self.total}"] + parts.append(f"[{_fmt_secs(self._elapsed)}]") + if self.track_success: + parts.append(f"success {_fmt_rate(self.passed / self.done if self.done else None)} (mean)") + parts.append(f"{_fmt_cost(self.cost)} total") + return " ".join(parts) + + def _surface_error(self, outcome: RunOutcome) -> None: + """Let genuine harness failures through even in bars mode; ordinary test fails just lower the rate.""" + detail = outcome.payload.get("error") + if outcome.reason == "provider_exhausted": + msg = f"provider usage/credit exhausted: {detail or 'no provider detail'}" + elif outcome.reason == "sandbox_blocked": + msg = f"agent sandbox refused an outbound host (harness bug): {detail or 'no sandbox detail'}" + else: + return + print(f"{chr(10) if self.mode == 'bars' else ''}error: {msg}", file=sys.stderr, flush=True) + + def finish(self) -> None: + """Leave the completed bar on screen (bars) or print the final line (plain).""" + print(("\r" if self.mode == "bars" else " ") + self._line(), flush=True) + + +class _Spinner: + """An indeterminate elapsed-time spinner for the single-agent improve step (no sub-progress). + + In ``bars`` mode a daemon thread rewrites the line while ``improve_skill`` blocks the main + thread; other modes just print a start line. Use as a context manager around the blocking call. + """ + + def __init__(self, label: str, mode: str) -> None: + self.label = label + self.mode = mode + self._t0 = time.monotonic() + self._stop = threading.Event() + self._thread: threading.Thread | None = None + + def __enter__(self) -> _Spinner: + if self.mode == "bars": + self._thread = threading.Thread(target=self._spin, daemon=True) + self._thread.start() + else: + print(f" {self.label} ...", flush=True) + return self + + def _spin(self) -> None: + for frame in itertools.cycle(_SPINNER_FRAMES): + if self._stop.is_set(): + break + print(f"\r {self.label} {frame} {_fmt_secs(time.monotonic() - self._t0)}", end="", flush=True) + self._stop.wait(0.1) + + def __exit__(self, *exc: object) -> None: + self._stop.set() + if self._thread is not None: + self._thread.join() + print("\r\033[K", end="", flush=True) # clear the spinner line for the done line + + @dataclass(frozen=True) class _Arm: """One arm of a pass: its version, its loaded skill, and its matrix.""" @@ -366,11 +561,16 @@ def _print_run_summary(outcomes: Sequence[RunOutcome], elapsed: float, *, label: print(f"\n{prefix}{passed}/{len(outcomes)} passed in {_fmt_secs(elapsed)} ({cost_summary}, {breakdown})") -def _print_skill_loading(outcomes: Sequence[RunOutcome], arm: _Arm, skill_name: str) -> None: - """Say whether the skill reached the agent — the comparison means nothing otherwise.""" +def _print_skill_loading(outcomes: Sequence[RunOutcome], arm: _Arm, skill_name: str, *, quiet: bool = False) -> None: + """Say whether the skill reached the agent — the comparison means nothing otherwise. + + ``quiet`` drops the routine "loaded in N/M" line (the bars keep phase output compact) but + still raises the warnings, which signal a broken comparison and must never be swallowed. + """ loaded = sum(1 for o in outcomes if o.payload.get("skill_loaded")) if arm.skill is not None: - print(f"skill loaded in {loaded}/{len(outcomes)} runs") + if not quiet: + print(f"skill loaded in {loaded}/{len(outcomes)} runs") if loaded == 0: print( f"warning: {arm.name} never loaded the skill — that arm is not measuring the skill", @@ -380,6 +580,104 @@ def _print_skill_loading(outcomes: Sequence[RunOutcome], arm: _Arm, skill_name: print(f"warning: {skill_name} loaded in {loaded} baseline runs", file=sys.stderr) +def _resolve_bench_auth(models: set[str], auth: str) -> dict[AgentProvider, AuthMode]: + """Resolve one auth mode per provider present, checking each CLI and printing the choice.""" + providers = {provider_for_model(model) for model in models} + auth_modes = {provider: resolve_auth_mode(auth, provider=provider) for provider in providers} + for provider in sorted(providers): + check_agent_cli(provider) + _print_auth(auth_modes[provider], provider) + _warn_codex_accounting(provider) + if "session" in auth_modes.values(): + print( + "note: cost_usd for session-billed runs is what they would have cost at API " + "rates, not metered spend; each run records its auth_mode", + file=sys.stderr, + ) + return auth_modes + + +def _execute_arms( + arms: Sequence[_Arm], + *, + cfg: Config, + target, + runs_root: Path, + auth_modes: dict[AgentProvider, AuthMode], + prices: PriceTable, + keep_sandboxes: bool, + progress: _Progress | _PhaseBar | None = None, + quiet: bool = False, +) -> list[RunOutcome]: + """Run each arm's pending runs sequentially, sharing one progress counter across them. + + Arms run one after another: every run in a matrix shares one skill, and a sequential pass + keeps each arm's tally readable while the progress counter spans the whole thing. ``quiet`` + (epoch/fit bar mode) suppresses the per-arm banner and tally so the phase bar is the only + output; harness warnings still surface. Raises :class:`BenchmarkInvalidError` if a harness + failure decides the pass. + """ + running = [arm for arm in arms if arm.todo] + todo = [item for arm in running for item in arm.todo] + if not todo: + return [] + progress = progress or _Progress(len(todo)) + collected: list[RunOutcome] = [] + for arm in running: + if len(running) > 1 and not quiet: + print(f"\n=== arm {arm.name}: {len(arm.todo)} runs ===", flush=True) + started = time.monotonic() + outcomes = asyncio.run( + run_matrix( + arm.todo, + target=target, + runs_root=runs_root, + max_concurrency=cfg.max_concurrency, + auth_modes=auth_modes, + skill=arm.skill, + skill_name=cfg.skill_name, + keep_sandbox=keep_sandboxes, + stderr=StderrFilter(), + on_start=progress.on_start, + on_done=progress.on_done, + env_passthrough=cfg.env_passthrough, + prices=prices, + ) + ) + collected.extend(outcomes) + if not quiet: + _print_run_summary(outcomes, time.monotonic() - started, label=arm.name if len(running) > 1 else "") + _print_skill_loading(outcomes, arm, cfg.skill_name, quiet=quiet) + return collected + + +def _build_arms( + specs: Sequence[tuple[str | None, Sequence[Split]]], + *, + cfg: Config, + tasks: Sequence[Task], + runs_root: Path, + skills_root: Path, + resume: bool = True, + task_ids: Sequence[str] | None = None, +) -> list[_Arm]: + """Build arms for an explicit list of ``(version, splits)`` specs (for the epoch orchestrator).""" + arms = [] + for version, splits in specs: + skill = None if version is None else load_skill(skills_root, version, expect_name=cfg.skill_name) + planned = build_matrix(cfg, tasks, skill=version, splits=splits, task_ids=task_ids) + todo = pending(planned, runs_root, resume=resume) + arms.append(_Arm(version=version, skill=skill, planned=planned, todo=todo)) + return arms + + +def _invalid_bench_note() -> None: + print( + "Fix or replenish that credential, then rerun the same command; invalid and cancelled cells remain pending.", + file=sys.stderr, + ) + + def _cmd_bench(args: argparse.Namespace) -> int: cfg = load_config(args.config) tasks = load_tasks(args.tasks) @@ -400,18 +698,7 @@ def _cmd_bench(args: argparse.Namespace) -> int: return 0 # One resolved mode per provider in the matrix, so a mixed pass bills each side correctly. - providers = {provider_for_model(item.model) for item in todo} - auth_modes = {provider: resolve_auth_mode(args.auth, provider=provider) for provider in providers} - for provider in sorted(providers): - check_agent_cli(provider) - _print_auth(auth_modes[provider], provider) - _warn_codex_accounting(provider) - if "session" in auth_modes.values(): - print( - "note: cost_usd for session-billed runs is what they would have cost at API " - "rates, not metered spend; each run records its auth_mode", - file=sys.stderr, - ) + auth_modes = _resolve_bench_auth({item.model for item in todo}, args.auth) # Before the target is built and before any agent runs: an unreachable pricing page # must cost nothing, and a pass must never be priced by a table it cannot date. try: @@ -430,44 +717,23 @@ def _cmd_bench(args: argparse.Namespace) -> int: target = prepare_target(cfg, args.cache, refresh=args.refresh_target) print(f"target ready: {target.fingerprint} @ {target.commit[:8]} (venv {target.venv_dir})", flush=True) - # Arms run one after another: every run in a matrix shares one skill, and a sequential - # pass keeps each arm's tally readable while the progress counter spans the whole thing. running = [arm for arm in arms if arm.todo] print(f"running {len(todo)} runs, up to {cfg.max_concurrency} at a time:", flush=True) progress = _Progress(len(todo)) - collected: list[RunOutcome] = [] try: - for arm in running: - if len(running) > 1: - print(f"\n=== arm {arm.name}: {len(arm.todo)} runs ===", flush=True) - started = time.monotonic() - outcomes = asyncio.run( - run_matrix( - arm.todo, - target=target, - runs_root=args.runs, - max_concurrency=cfg.max_concurrency, - auth_modes=auth_modes, - skill=arm.skill, - skill_name=cfg.skill_name, - keep_sandbox=args.keep_sandboxes, - stderr=StderrFilter(), - on_start=progress.on_start, - on_done=progress.on_done, - env_passthrough=cfg.env_passthrough, - prices=prices, - ) - ) - collected.extend(outcomes) - _print_run_summary(outcomes, time.monotonic() - started, label=arm.name if len(running) > 1 else "") - _print_skill_loading(outcomes, arm, cfg.skill_name) + collected = _execute_arms( + arms, + cfg=cfg, + target=target, + runs_root=args.runs, + auth_modes=auth_modes, + prices=prices, + keep_sandboxes=args.keep_sandboxes, + progress=progress, + ) except BenchmarkInvalidError as err: print(f"\nerror: {err}", file=sys.stderr) - print( - "Fix or replenish that credential, then rerun the same command; invalid and " - "cancelled cells remain pending.", - file=sys.stderr, - ) + _invalid_bench_note() return 2 if len(running) > 1: @@ -476,20 +742,37 @@ def _cmd_bench(args: argparse.Namespace) -> int: return 0 -def _cmd_draft(args: argparse.Namespace) -> int: +def _cmd_improve(args: argparse.Namespace) -> int: cfg = load_config(args.config) + tasks = load_tasks(args.tasks) if args.model: cfg = replace(cfg, meta_model=args.model) - existing = available_versions(args.skills) - if existing and not args.force: + # Parent is an explicit --from, else the latest version, else None (create the first skill + # from the noskill wiki). improve_skill resolves and validates the parent itself. + parent = args.from_version or latest_version(args.skills) + parent_skill = None if parent is None else load_skill(args.skills, parent, expect_name=cfg.skill_name) + # Fail before the costly target prep if there is nothing to learn from: the improver needs the + # parent arm's train runs (and the wiki built from them). + parent_arm = arm_name(parent) + if not collect_arm_runs(args.runs, parent_arm, tasks, split="train"): + hint = ( + "acumen bench --no-skill --split train" + if parent is None + else f"acumen bench --skill {parent} --split train" + ) print( - f"skills already exist ({', '.join(existing)}) — drafting would add " - f"another version. Pass --force to draft anyway, or use `acumen improve` " - f"to build on {existing[-1]}.", + f"no train-split runs found for {parent_arm} under {args.runs / parent_arm / 'train'} — " + f"run `{hint}` and `acumen wiki` first", file=sys.stderr, ) return 2 + if parent_skill is None: + print(f"no skill versions under {args.skills} — creating the first skill from the wiki with {cfg.meta_model}") + else: + print( + f"improving {parent_skill.version} ({parent_skill.name}, {parent_skill.hash[:19]}…) with {cfg.meta_model}" + ) provider = provider_for_model(cfg.meta_model) check_agent_cli(provider) @@ -499,55 +782,127 @@ def _cmd_draft(args: argparse.Namespace) -> int: print(f"preparing target {cfg.repo}@{cfg.ref} ...", flush=True) target = prepare_target(cfg, args.cache, refresh=args.refresh_target) print(f"target ready: {target.fingerprint} @ {target.commit[:8]}", flush=True) - print(f"drafting with {cfg.meta_model} (this reads the package source) ...", flush=True) - log = LiveLog.open(args.log_dir, "draft", stream=args.stream) + log = LiveLog.open(args.log_dir, "improve", stream=args.stream) print(f"log → {log.jsonl_path}", flush=True) with log: result = asyncio.run( - draft_skill( + improve_skill( cfg=cfg, prices=_agent_prices(cfg, model=cfg.meta_model), target=target, skills_root=args.skills, + runs_root=args.runs, + wiki_root=args.wiki, + tasks=tasks, auth_mode=auth_mode, + parent_version=parent, max_turns=args.max_turns, max_usd=args.max_usd, feedback=args.feedback, log=log, ) ) - skill = result.skill - files = sorted(p.relative_to(skill.directory).as_posix() for p in skill.directory.rglob("*") if p.is_file()) - print(f"\nwrote {skill.directory}") - print(f" name: {skill.name}") - print(f" description: {skill.description}") - print(f" hash: {skill.hash}") + new = result.skill + files = sorted(p.relative_to(new.directory).as_posix() for p in new.directory.rglob("*") if p.is_file()) + parent_label = result.parent or "noskill" + print(f"\nwrote {new.directory} (parent {parent_label})") + print(f" name: {new.name}") + print(f" description: {new.description}") + print(f" hash: {new.hash}") print(f" files: {', '.join(files)}") + print(f" evidence: {result.n_train_runs} train runs ({result.n_train_failures} failing)") print(f" cost: {_fmt_cost(result.cost_usd)} over {result.turns} turns") + if parent_skill is not None and new.hash == parent_skill.hash: + print( + "warning: the new version is byte-identical to its parent — the improver changed nothing", + file=sys.stderr, + ) _print_log_result(log) - print(f"\nnext: acumen bench --skill {skill.version}") + print(f"\nnext: acumen bench --skill {new.version} && acumen report") return 0 -def _cmd_improve(args: argparse.Namespace) -> int: +def _wiki_version(args: argparse.Namespace) -> str: + """Resolve the version label a wiki update records, from ``--skill``/``--no-skill``/default.""" + if args.skill is not None: + return args.skill if args.skill.startswith("v") else f"v{args.skill}" + if args.no_skill: + return arm_name(None) + return latest_version(args.skills) or arm_name(None) + + +def _print_wiki_task(result) -> None: + """Print one task's wiki-update outcome as it lands, with any brevity warnings.""" + print(f" wiki [{result.version}] {result.task_id}: {_fmt_cost(result.cost_usd)} over {result.turns} turns") + for warning in result.warnings: + print(f" warning: {warning}", file=sys.stderr) + + +def _run_wiki( + *, + cfg: Config, + tasks: Sequence[Task], + target, + version: str, + runs_root: Path, + wiki_root: Path, + skills_root: Path, + prices: PriceTable, + auth_mode: AuthMode, + log_dir: Path, + stream: bool, + mode: str = "verbose", +): + """Update the wiki for one arm and report progress; returns the task results. + + ``mode`` (``verbose``/``bars``/``plain``) picks the reporting style — a per-task tally in + verbose, a single progress bar otherwise. The bar's denominator is the count of tasks the + pass will actually run, reported once via ``update_wiki``'s ``on_plan`` callback. + """ + bar = None if mode == "verbose" else _PhaseBar("wiki", 0, mode, track_success=False) + if mode == "verbose": + print(f"updating wiki for [{version}] with {cfg.meta_model} (one agent per task) ...", flush=True) + results = asyncio.run( + update_wiki( + cfg=cfg, + target=target, + runs_root=runs_root, + wiki_root=wiki_root, + skills_root=skills_root, + tasks=tasks, + version=version, + prices=prices, + auth_mode=auth_mode, + max_concurrency=cfg.max_concurrency, + log_dir=log_dir, + stream=stream, + on_plan=(bar.set_total if bar is not None else None), + on_task_done=(_print_wiki_task if bar is None else bar.on_wiki_task), + ) + ) + if bar is not None: + bar.finish() + elif not results: + print(f" wiki already had [{version}] for every task — nothing to do") + else: + total = sum(r.cost_usd or 0.0 for r in results) + print(f"wiki: updated {len(results)} task(s) for [{version}] ({_fmt_cost(total)})") + return results + + +def _cmd_wiki(args: argparse.Namespace) -> int: cfg = load_config(args.config) tasks = load_tasks(args.tasks) + if args.max_concurrency: + cfg = replace(cfg, max_concurrency=args.max_concurrency) if args.model: cfg = replace(cfg, meta_model=args.model) - versions = available_versions(args.skills) - if not versions: - print( - f"no skill versions under {args.skills} — run `acumen draft` first, then bench it", - file=sys.stderr, - ) - return 2 - parent = args.from_version or latest_version(args.skills) - # Immutability guard: the improved version is always the next unused directory, - # so an existing version is never in the write path. Say the parent plainly up front. - skill = load_skill(args.skills, parent, expect_name=cfg.skill_name) - print(f"improving {skill.version} ({skill.name}, {skill.hash[:19]}…) with {cfg.meta_model}") + version = _wiki_version(args) + if version != arm_name(None): + # Validate a named skill version exists before the costly target prep. + load_skill(args.skills, version, expect_name=cfg.skill_name) provider = provider_for_model(cfg.meta_model) check_agent_cli(provider) @@ -558,41 +913,327 @@ def _cmd_improve(args: argparse.Namespace) -> int: target = prepare_target(cfg, args.cache, refresh=args.refresh_target) print(f"target ready: {target.fingerprint} @ {target.commit[:8]}", flush=True) - log = LiveLog.open(args.log_dir, "improve", stream=args.stream) - print(f"log → {log.jsonl_path}", flush=True) - with log: - result = asyncio.run( - improve_skill( + _run_wiki( + cfg=cfg, + tasks=tasks, + target=target, + version=version, + runs_root=args.runs, + wiki_root=args.wiki, + skills_root=args.skills, + prices=_agent_prices(cfg, model=cfg.meta_model), + auth_mode=auth_mode, + log_dir=args.log_dir, + stream=args.stream, + ) + print(f"\nwiki written to {args.wiki.resolve()}") + return 0 + + +def _prepare_pass(cfg: Config, args: argparse.Namespace): + """Resolve auth, freeze prices, and build the target once for a pass that spends on benches. + + Shared by ``epoch`` and ``fit`` so a multi-epoch run prepares the target and rates once rather + than per epoch. Raises :class:`PriceFeedError` if rates cannot be established (benchmark cost is + frozen into every result, so a pass must not run without them). + """ + auth_modes = _resolve_bench_auth(set(cfg.models) | {cfg.meta_model}, args.auth) + meta_auth = auth_modes[provider_for_model(cfg.meta_model)] + prices = _bench_prices(cfg) + _warn_unpriced(set(cfg.models) | {cfg.meta_model}, prices) + print(f"preparing target {cfg.repo}@{cfg.ref} ...", flush=True) + target = prepare_target(cfg, args.cache, refresh=args.refresh_target) + print(f"target ready: {target.fingerprint} @ {target.commit[:8]} (venv {target.venv_dir})", flush=True) + return auth_modes, meta_auth, prices, target + + +def _valid_complete_fn(cfg: Config, tasks: list[Task], runs_root: Path) -> Callable[[str], bool]: + """Build the "does this version have a complete ``valid`` bench?" predicate. + + Shared by epoch resolution (``resolve_epoch``/``completed_epochs``) so the two never + disagree on what "done" means. + """ + + def valid_complete(version: str) -> bool: + planned = build_matrix(cfg, tasks, skill=version, splits=["valid"]) + return not pending(planned, runs_root, resume=True) + + return valid_complete + + +def _run_one_epoch( + args: argparse.Namespace, + *, + cfg: Config, + tasks: list[Task], + target, + prices: PriceTable, + auth_modes: dict[AgentProvider, AuthMode], + meta_auth: AuthMode, + mode: str = "verbose", +) -> EpochPlan: + """Run one training epoch on an already-prepared pass, returning the resolved plan. + + The four steps: bench the parent arm on train (the first epoch also benches the noskill + baseline on valid), distil that arm into the wiki, create or improve the skill (skipped when + the version already exists — a resumed epoch), then bench the new version on valid. + + Auth, prices, and the target are passed in, not re-derived, so ``fit`` runs many epochs against + one prepared target. Raises :class:`BenchmarkInvalidError` if a harness failure decides a bench; + the caller reports it. + """ + runs_root = args.runs + + valid_complete = _valid_complete_fn(cfg, tasks, runs_root) + plan = resolve_epoch(args.skills, valid_complete=valid_complete) + parent_label = plan.parent_version or arm_name(None) + tail = " (resuming)" if plan.resumed else "" + print(f"epoch: learning from [{parent_label}] → producing {plan.new_version}{tail}") + + def bench_phase(specs, *, label: str, verbose_banner: str) -> None: + """Run one bench step: a numbered banner + plan + tally in verbose, a phase bar otherwise.""" + if mode == "verbose": + print(verbose_banner, flush=True) + bench_arms = _build_arms(specs, cfg=cfg, tasks=tasks, runs_root=runs_root, skills_root=args.skills) + if mode == "verbose": + _print_plan(bench_arms) + bar = None if mode == "verbose" else _PhaseBar(label, sum(len(a.todo) for a in bench_arms), mode) + _execute_arms( + bench_arms, + cfg=cfg, + target=target, + runs_root=runs_root, + auth_modes=auth_modes, + prices=prices, + keep_sandboxes=False, + progress=bar, + quiet=mode != "verbose", + ) + if bar is not None: + bar.finish() + + # Step 1 — bench the parent arm on the training signal. The first epoch also benches the + # noskill baseline on valid, so the report has an in-epoch baseline to compare against. + if plan.first: + train_specs: list[tuple[str | None, Sequence[Split]]] = [(None, ["train", "valid"])] + else: + train_specs = [(plan.parent_version, ["train"])] + bench_phase( + train_specs, + label="bench baseline" if plan.first else "bench train", + verbose_banner=f"\n[1/4] benchmarking [{parent_label}] on the training signal ...", + ) + + # Step 2 — distil that arm into the wiki (idempotent: recorded arms are skipped). + if mode == "verbose": + print(f"\n[2/4] updating the wiki for [{parent_label}] ...", flush=True) + _run_wiki( + cfg=cfg, + tasks=tasks, + target=target, + version=parent_label, + runs_root=runs_root, + wiki_root=args.wiki, + skills_root=args.skills, + prices=prices, + auth_mode=meta_auth, + log_dir=args.log_dir, + stream=args.stream, + mode=mode, + ) + + # Step 3 — create or improve the skill (skipped when the version already exists: a resumed + # epoch that crashed after improve). + if skill_dir(args.skills, plan.new_version).exists(): + resumed_msg = f"{plan.new_version} already exists — skipping improve (resumed epoch)" + print(f"\n[3/4] {resumed_msg}" if mode == "verbose" else f" {resumed_msg}", flush=True) + else: + verb = "creating" if plan.first else "improving" + log = LiveLog.open(args.log_dir, "improve", stream=args.stream) + if mode == "verbose": + print(f"\n[3/4] {verb} the skill → {plan.new_version} with {cfg.meta_model} ...", flush=True) + print(f"log → {log.jsonl_path}", flush=True) + spinner = nullcontext() if mode == "verbose" else _Spinner(f"inferring skill → {plan.new_version}", mode) + improve_started = time.monotonic() + with log, spinner: + result = asyncio.run( + improve_skill( + cfg=cfg, + prices=prices, + target=target, + skills_root=args.skills, + runs_root=runs_root, + wiki_root=args.wiki, + tasks=tasks, + auth_mode=meta_auth, + parent_version=plan.parent_version, + feedback=args.feedback, + log=log, + ) + ) + improve_elapsed = _fmt_secs(time.monotonic() - improve_started) + new = result.skill + if mode == "verbose": + print(f"wrote {new.directory} (parent {result.parent or 'noskill'})") + print(f" description: {new.description}") + print(f" cost: {_fmt_cost(result.cost_usd)} over {result.turns} turns in {improve_elapsed}") + _print_log_result(log) + else: + print( + f" inferring skill → {new.version} done " + f"({improve_elapsed}, {_fmt_cost(result.cost_usd)}, {result.turns} turns)", + flush=True, + ) + + # Step 4 — bench the new version on the held-out valid signal (completes it if unfinished). + bench_phase( + [(plan.new_version, ["valid"])], + label="bench valid", + verbose_banner=f"\n[4/4] benchmarking {plan.new_version} on the held-out valid signal ...", + ) + return plan + + +def _cmd_epoch(args: argparse.Namespace) -> int: + cfg = load_config(args.config) + tasks = load_tasks(args.tasks) + if args.max_concurrency: + cfg = replace(cfg, max_concurrency=args.max_concurrency) + if args.replicates: + cfg = replace(cfg, n_replicates=args.replicates) + if args.model: + cfg = replace(cfg, meta_model=args.model) + + try: + auth_modes, meta_auth, prices, target = _prepare_pass(cfg, args) + except PriceFeedError as err: + print(f"error: {err}", file=sys.stderr) + print("Retry when the pricing pages are reachable, or pin rates in config.yaml.", file=sys.stderr) + return 2 + + try: + plan = _run_one_epoch( + args, + cfg=cfg, + tasks=tasks, + target=target, + prices=prices, + auth_modes=auth_modes, + meta_auth=meta_auth, + mode=_progress_mode(args), + ) + except BenchmarkInvalidError as err: + print(f"\nerror: {err}", file=sys.stderr) + _invalid_bench_note() + return 2 + + print(f"\nepoch complete: {plan.new_version} produced and benched on valid.") + print("next: `acumen report` to see it, or `acumen epoch` again for another round") + return 0 + + +def _cmd_fit(args: argparse.Namespace) -> int: + cfg = load_config(args.config) + tasks = load_tasks(args.tasks) + if args.max_concurrency: + cfg = replace(cfg, max_concurrency=args.max_concurrency) + if args.replicates: + cfg = replace(cfg, n_replicates=args.replicates) + if args.model: + cfg = replace(cfg, meta_model=args.model) + + fixed = args.epochs is not None + # A GLOBAL target across invocations, not a per-run count: --epochs/--max-epochs is the total + # number of epochs to end up with. It is taken from this invocation's flag and not persisted, + # so re-run with the same flag to keep continuing toward the same target. + total = args.epochs if fixed else args.max_epochs + if total < 1: + print("error: nothing to run — --epochs/--max-epochs must be >= 1", file=sys.stderr) + return 2 + + try: + auth_modes, meta_auth, prices, target = _prepare_pass(cfg, args) + except PriceFeedError as err: + print(f"error: {err}", file=sys.stderr) + print("Retry when the pricing pages are reachable, or pin rates in config.yaml.", file=sys.stderr) + return 2 + + # How many epochs are already fully done on disk — the loop resumes from the next one. + completed = completed_epochs(args.skills, valid_complete=_valid_complete_fn(cfg, tasks, args.runs)) + remaining = max(0, total - completed) + prior = [row for row in build_training_rows(args.runs, cfg) if row.epoch <= completed] if completed else [] + stopping = "early stopping disabled" if fixed else f"early stopping with patience {args.patience}" + if completed: + print(f"fit: target {total} epoch(s) total ({stopping}); {completed} done, {remaining} to go") + # Show the progress of the epochs already finished (skip any in-progress/partial version). + prior_best = best_version(prior) + print(f"\nresuming: {completed} epoch(s) already complete (target {total})") + for index, row in enumerate(prior): + since = epochs_since_best([r.valid_success for r in prior[: index + 1]]) + print( + _epoch_bar( + row.epoch, total, row, best=prior_best, patience=args.patience, since_best=since, fixed=fixed + ) + ) + else: + print(f"fit: target {total} epoch(s) total ({stopping})") + + # A finished epoch that already hit 100% means there is nothing left to gain — do not resume + # into more epochs. Mirrors the in-loop is_perfect stop, which never sees a prior epoch. + already_perfect = next((row for row in prior if is_perfect(row.valid_success)), None) + if already_perfect is not None: + print(f"\nearly stop: validation success already reached 100% at {already_perfect.version}.") + elif remaining == 0: + print(f"\nnothing to do: already at {completed} epoch(s) (target {total}).") + + mode = _progress_mode(args) + epochs = range(completed + 1, total + 1) if already_perfect is None else range(0) + for epoch_no in epochs: + _epoch_header(mode, epoch_no, total) + try: + plan = _run_one_epoch( + args, cfg=cfg, - prices=_agent_prices(cfg, model=cfg.meta_model), - target=target, - skills_root=args.skills, - runs_root=args.runs, tasks=tasks, - auth_mode=auth_mode, - parent_version=parent, - max_turns=args.max_turns, - max_usd=args.max_usd, - feedback=args.feedback, - log=log, + target=target, + prices=prices, + auth_modes=auth_modes, + meta_auth=meta_auth, + mode=mode, ) - ) - new = result.skill - files = sorted(p.relative_to(new.directory).as_posix() for p in new.directory.rglob("*") if p.is_file()) - print(f"\nwrote {new.directory} (parent {result.parent})") - print(f" name: {new.name}") - print(f" description: {new.description}") - print(f" hash: {new.hash}") - print(f" files: {', '.join(files)}") - print(f" evidence: {result.n_train_runs} train runs ({result.n_train_failures} failing)") - print(f" cost: {_fmt_cost(result.cost_usd)} over {result.turns} turns") - if new.hash == skill.hash: - print( - "warning: the new version is byte-identical to its parent — the improver changed nothing", - file=sys.stderr, - ) - _print_log_result(log) - print(f"\nnext: acumen bench --skill {new.version} && acumen report") + except BenchmarkInvalidError as err: + print(f"\nerror: {err}", file=sys.stderr) + _invalid_bench_note() + return 2 + + # Rebuild the whole training curve from runs/ each epoch, so the CSV is always consistent + # with what exists and a resumed fit produces the same table. + rows = build_training_rows(args.runs, cfg) + write_training_csv(rows, args.out) + current = next((row for row in rows if row.version == plan.new_version), None) + valids = [row.valid_success for row in rows] + best = best_version(rows) + since = epochs_since_best(valids) + print(_epoch_bar(epoch_no, total, current, best=best, patience=args.patience, since_best=since, fixed=fixed)) + + # A perfect validation score leaves nothing to gain — stop even under a fixed --epochs. + if current is not None and is_perfect(current.valid_success): + print(f"\nearly stop: validation success reached 100% at {current.version}.") + break + + if not fixed and patience_exhausted(valids, args.patience): + print(f"\nearly stop: validation mean success did not improve in {args.patience} epoch(s).") + break + + rows = build_training_rows(args.runs, cfg) + best = best_version(rows) + if best is not None: + best_row = next(row for row in rows if row.version == best) + print(f"\nfit complete: best version {best} (valid {_fmt_rate(best_row.valid_success)}).") + else: + print("\nfit complete.") + print(f"training curve → {args.out.resolve()}") + print("next: `acumen report` for the full breakdown") return 0 @@ -668,7 +1309,7 @@ def _cmd_tasks(args: argparse.Namespace) -> int: "and split, so it was not kept", file=sys.stderr, ) - print("\nnext: review the tasks, then `acumen check`, `acumen draft` and `acumen bench`") + print("\nnext: review the tasks, then `acumen check`, then `acumen epoch`") return 0 @@ -1082,7 +1723,7 @@ def _cmd_init(args: argparse.Namespace) -> int: written = scaffold(args.directory, force=args.force) for path in written: print(f"wrote {path}") - print("\nnext: edit config.yaml (repo) and tasks.yaml, then `acumen draft`") + print("\nnext: edit config.yaml (repo) and tasks.yaml, then `acumen epoch`") return 0 @@ -1167,25 +1808,18 @@ def build_parser() -> argparse.ArgumentParser: _add_bench_args(bench) bench.set_defaults(func=_cmd_bench) - draft = sub.add_parser("draft", help="draft a skill from the target package's source") - draft.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") - draft.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") - draft.add_argument("--model", help="override config meta_model") - draft.add_argument("--max-turns", type=int, help="cap turns for the drafting agent (default: unbounded)") - draft.add_argument("--max-usd", type=float, help="cap spend for the drafting agent (default: unbounded)") - draft.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") - draft.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") - draft.add_argument("--force", action="store_true", help="draft another version even if some already exist") - _add_auth_arg(draft) - _add_feedback_arg(draft, extra=" (e.g. package context, what the skill should emphasise)") - _add_log_args(draft) - draft.set_defaults(func=_cmd_draft) - - improve = sub.add_parser("improve", help="improve the current skill into a new version from its train results") + improve = sub.add_parser( + "improve", + help="create or improve the skill from the knowledge wiki", + description="Read the knowledge wiki (per-task observations/hypotheses distilled from train " + "runs) and the filtered package source, then create the first skill (when none exist) or " + "improve the latest into the next version. The held-out valid split is never reachable.", + ) improve.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") improve.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") improve.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") improve.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + improve.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") improve.add_argument("--from", dest="from_version", metavar="VERSION", help="version to improve (default: latest)") improve.add_argument("--model", help="override config meta_model") improve.add_argument("--max-turns", type=int, help="cap turns for the improving agent (default: unbounded)") @@ -1195,11 +1829,92 @@ def build_parser() -> argparse.ArgumentParser: _add_auth_arg(improve) _add_feedback_arg( improve, - extra=" (e.g. what to fix or emphasise; do NOT paste test-split answers — that defeats the held-out split)", + extra=" (e.g. what to fix or emphasise; do NOT paste valid-split answers — that defeats the held-out split)", ) _add_log_args(improve) improve.set_defaults(func=_cmd_improve) + wiki = sub.add_parser( + "wiki", + help="distil an arm's train runs into the knowledge wiki (one agent per task)", + description="For each task, read that arm's train-split runs across models and replicates " + "and append a terse [version][model] block to wiki//observations.md and hypothesis.md. " + "Cumulative and idempotent: an arm already recorded for a task is skipped.", + ) + wiki.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") + wiki.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") + wiki.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + wiki.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") + wiki.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") + wiki_arm = wiki.add_mutually_exclusive_group() + wiki_arm.add_argument("--no-skill", action="store_true", help="record the baseline (noskill) arm") + wiki_arm.add_argument("--skill", metavar="VERSION", help="record one skill version, e.g. v1 (default: latest)") + wiki.add_argument("--model", help="override config meta_model") + wiki.add_argument("--max-concurrency", type=int, help="override config max_concurrency") + wiki.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") + wiki.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") + _add_auth_arg(wiki) + _add_log_args(wiki) + wiki.set_defaults(func=_cmd_wiki) + + epoch = sub.add_parser( + "epoch", + help="run one training epoch: bench train, update wiki, improve, bench valid", + description="One training epoch end to end: bench the current arm on the training signal, " + "distil it into the wiki, create/improve the skill, then bench the new version on the " + "held-out valid signal. Fully resumable — re-run to continue a crashed epoch or start the " + "next one.", + ) + epoch.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") + epoch.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") + epoch.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + epoch.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") + epoch.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") + epoch.add_argument("--model", help="override config meta_model") + epoch.add_argument("--max-concurrency", type=int, help="override config max_concurrency") + epoch.add_argument("--replicates", type=int, help="override config n_replicates") + epoch.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") + epoch.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") + epoch.add_argument("--verbose", action="store_true", help="use the full scrolling logs instead of progress bars") + _add_auth_arg(epoch) + _add_feedback_arg(epoch, extra=" (passed to the improver; do NOT paste valid-split answers)") + _add_log_args(epoch) + epoch.set_defaults(func=_cmd_epoch) + + fit = sub.add_parser( + "fit", + help="run many training epochs with early stopping (like training a model)", + description="Run `acumen epoch` back to back until validation stops improving (patience) or " + "a hard cap is hit, writing a per-epoch training curve to training.csv and a progress line " + "each epoch. Fully resumable — re-run to continue.", + ) + fit.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") + fit.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") + fit.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + fit.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") + fit.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") + fit.add_argument("--out", type=Path, default=Path("training.csv"), help="training-curve CSV to write") + fit.add_argument( + "--patience", type=int, default=2, help="stop after this many epochs with no validation gain (default: 2)" + ) + fit.add_argument("--max-epochs", type=int, default=10, help="hard cap on epochs (default: 10)") + fit.add_argument( + "--epochs", + type=int, + default=None, + help="run exactly this many epochs (disables early stopping and ignores --max-epochs)", + ) + fit.add_argument("--model", help="override config meta_model") + fit.add_argument("--max-concurrency", type=int, help="override config max_concurrency") + fit.add_argument("--replicates", type=int, help="override config n_replicates") + fit.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") + fit.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") + fit.add_argument("--verbose", action="store_true", help="use the full scrolling logs instead of progress bars") + _add_auth_arg(fit) + _add_feedback_arg(fit, extra=" (passed to the improver; do NOT paste valid-split answers)") + _add_log_args(fit) + fit.set_defaults(func=_cmd_fit) + tasks_cmd = sub.add_parser("tasks", help="autonomously generate a tasks.yaml from the target package") tasks_cmd.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") tasks_cmd.add_argument("--out", type=Path, default=Path("tasks.yaml"), help="tasks.yaml to write") @@ -1325,8 +2040,8 @@ def main(argv: list[str] | None = None) -> int: AgentError, PriceFeedError, SkillError, - DraftError, ImproveError, + WikiError, TaskGenError, ReviewError, ShipError, diff --git a/src/acumen/draft.py b/src/acumen/draft.py deleted file mode 100644 index b77db42..0000000 --- a/src/acumen/draft.py +++ /dev/null @@ -1,235 +0,0 @@ -"""The drafting agent: write ``skills/v1/`` from the target package's source. - -Unlike a benchmark agent, the drafter reads the target's source — it is documenting -the package, so it needs to see it. It is otherwise held to the same isolation: scrubbed -env, throwaway config dir, no user settings or memories. - -What it does *not* see is agent guidance the target already ships. A skill drafted from the -maintainer's own skill is not the independent artifact the benchmark then reports on: the arms -would be comparing that skill, re-served, against no skill at all. So the drafter reads a -filtered copy of the checkout with skills and agent-instruction files stripped -(:func:`acumen.scrub.build_filtered_source`) and is denied the original tree, exactly as -``tasks`` is. The real checkout is never modified. - -The agent writes into a staging directory, not into ``skills/`` directly. Only a skill -that loads and validates is promoted to a version, so a failed or half-finished draft -never leaves a broken ``skills/vN/`` behind — versions are immutable, which means -they must be right the moment they appear. -""" - -from __future__ import annotations - -import shutil -import tempfile -from dataclasses import dataclass -from pathlib import Path - -from acumen.agents import AgentOptions, AgentResult, provider_for_model, run_agent -from acumen.config import Config -from acumen.env import AuthMode, Target, build_agent_env -from acumen.logs import LiveLog -from acumen.prices import PriceTable, price_usage, pricer, resolve_cost -from acumen.procs import label_env, reap -from acumen.prompts import draft_prompt -from acumen.scrub import build_filtered_source, make_skill_guard -from acumen.skills import ( - SKILL_FILE, - Skill, - SkillError, - load_skill, - next_version, - skill_dir, - write_meta, -) - - -class DraftError(RuntimeError): - """Raised when a skill could not be drafted.""" - - -@dataclass(frozen=True) -class DraftResult: - """A drafted skill version and what it cost to produce.""" - - skill: Skill - cost_usd: float | None - turns: int - #: Live log paths for this run, when a :class:`LiveLog` was attached. - log_jsonl: Path | None = None - log_html: Path | None = None - - -def _validate_staged(staging: Path, skill_name: str) -> None: - """Fail loudly if the agent's output isn't a usable skill, before it becomes a version.""" - if not (staging / SKILL_FILE).is_file(): - raise DraftError( - f"the drafting agent did not write {SKILL_FILE} — nothing to promote. Inspect the run log or the prompt." - ) - # load_skill enforces the frontmatter contract (name matches, description present). - try: - load_skill(staging.parent, staging.name, expect_name=skill_name) - except SkillError as err: - raise DraftError(f"the drafted skill is not valid: {err}") from err - - -async def draft_skill( - *, - cfg: Config, - target: Target, - skills_root: Path, - auth_mode: AuthMode = "session", - prices: PriceTable | None = None, - model: str | None = None, - max_turns: int | None = None, - max_usd: float | None = None, - rationale: str = "initial draft", - feedback: str | None = None, - log: LiveLog | None = None, -) -> DraftResult: - """Draft a new skill version from the target package's source. - - Parameters - ---------- - cfg - The pass config; supplies ``skill_name`` and ``meta_model``. - target - The prepared target, supplying the source checkout and interpreter. - skills_root - The ``skills/`` root. The new version is the next unused one. - auth_mode - Which credential the drafting agent authenticates with — ``"session"`` (the Claude - subscription) or ``"api"`` (see :func:`acumen.env.build_agent_env`). - model - Override for the drafting model; defaults to the config's. - max_turns, max_usd - Turn and budget caps for the drafting agent. **Unbounded by default** — the config's - ``max_turns``/``max_usd`` cap benchmark agents only; pass explicit caps to bound the - drafter. - rationale - Recorded in ``meta.json`` as why this version exists. - feedback - Optional maintainer guidance, injected into the draft prompt as a subordinated block - and recorded in ``meta.json`` as provenance. - log - A :class:`LiveLog` to stream the agent's messages to and render an HTML log from. - - Returns - ------- - The drafted skill, loaded and validated. - """ - version = next_version(skills_root) - dest = skill_dir(skills_root, version) - if dest.exists(): - raise DraftError(f"{dest} already exists — skill versions are immutable and never overwritten") - - holder = Path(tempfile.mkdtemp(prefix="acumen-draft-")) - try: - # The staging dir is named for the version so load_skill can validate it in place. - work = holder / "work" - staging = work / version - home = holder / "home" - selected_model = model or cfg.meta_model - table = prices if prices is not None else PriceTable(overrides=cfg.prices) - provider = provider_for_model(selected_model) - config_dir = home / (".claude" if provider == "claude" else ".codex") - for path in (staging, home, config_dir, home / "tmp"): - path.mkdir(parents=True, exist_ok=True) - - # Marks the agent's processes so the teardown below can find what it leaves running. - env = label_env( - build_agent_env( - config_dir=config_dir, - home=home, - extra_path=[target.bin_dir], - auth_mode=auth_mode, - extra_allow=cfg.env_passthrough, - provider=provider, - ), - holder, - ) - - # A copy of the source with the target's own skills and agent guidance stripped. The - # drafter reads this, never the real checkout. - source_copy = build_filtered_source(target.src_dir, holder / "source") - - prompt = draft_prompt( - package=target.pkg_name, - version=target.pkg_version, - src=source_copy, - python=target.python, - out=staging, - skill_name=cfg.skill_name, - feedback=feedback, - ) - options = AgentOptions( - cwd=work, - env=env, - model=selected_model, - # No default turn or budget cap: only bound the agent if the caller asked. The - # config's ``max_turns``/``max_usd`` cap benchmark agents only. - max_turns=max_turns, - max_usd=max_usd, - # Codex reports no billed figure, so a budget cap needs the run's own rate table. - price_usd=pricer(selected_model, table), - # The drafter reads the target source; benchmark agents never do. It points at the - # *filtered* copy, not the real checkout. - read_dirs=(source_copy, target.venv_dir), - write_dirs=(work,), - discover_skills=True, - # Belt-and-braces over the filtered copy: deny any call that reaches an existing - # skill/guidance artifact or the original unfiltered source, wherever pointed. Built - # only for Claude — the hook is an SDK object, and Codex gets ``deny_paths`` below. - claude_hooks={"PreToolUse": [make_skill_guard(target.src_dir)]} if provider == "claude" else None, - # Codex reads the filtered copy and is denied the original checkout. - deny_paths=(target.src_dir.resolve(),), - ) - - result: AgentResult | None = None - agent_error: Exception | None = None - try: - result = await run_agent( - prompt, - options=options, - on_event=log.append if log is not None else None, - ) - except Exception as err: # noqa: BLE001 - a failed draft is an error to report, re-raised below - agent_error = err - finally: - # Render the HTML log while the throwaway config dir still holds the native - # transcript — in a finally so an aborted run (the SDK raises on a cap breach, - # after yielding the result) is still inspectable. - if log is not None: - log.finalize(config_dir=config_dir, work_dir=work, result=result) - - if agent_error is not None: - raise DraftError(f"the drafting agent failed: {type(agent_error).__name__}: {agent_error}") from agent_error - if result is None: - raise DraftError("the drafting agent produced no result message") - if result.is_error: - raise DraftError(f"the drafting agent errored: {result.subtype} {result.errors or ''}".strip()) - - _validate_staged(staging, cfg.skill_name) - - dest.parent.mkdir(parents=True, exist_ok=True) - shutil.copytree(staging, dest) - write_meta(dest, parent=None, rationale=rationale, feedback=feedback) - skill = load_skill(skills_root, version, expect_name=cfg.skill_name) - return DraftResult( - skill=skill, - cost_usd=resolve_cost( - result.total_cost_usd, - price_usage( - result.usage, - model=selected_model, - provider=result.provider, - prices=table, - ), - ).cost_usd, - turns=result.num_turns, - log_jsonl=log.jsonl_path if log is not None else None, - log_html=log.html_path if log is not None and log.html_rendered else None, - ) - finally: - # Kill anything the agent left running before removing the directory it runs in. - reap(holder) - shutil.rmtree(holder, ignore_errors=True) diff --git a/src/acumen/env.py b/src/acumen/env.py index 536e104..bafee3b 100644 --- a/src/acumen/env.py +++ b/src/acumen/env.py @@ -99,6 +99,13 @@ class Target: pkg_name: str pkg_version: str + @property + def is_remote(self) -> bool: + """Whether ``source`` is a remote URL (fetchable) rather than a local path.""" + from acumen.config import _looks_remote + + return _looks_remote(self.source) + @property def bin_dir(self) -> Path: """The venv's ``bin`` directory — what goes on an agent's PATH.""" diff --git a/src/acumen/epoch.py b/src/acumen/epoch.py new file mode 100644 index 0000000..4a47042 --- /dev/null +++ b/src/acumen/epoch.py @@ -0,0 +1,73 @@ +"""Resolve which training epoch an ``acumen epoch`` invocation is on, from on-disk state. + +An epoch benches the current arm on ``train``, distils it into the wiki, produces the next skill +version, and benches that version on ``valid``. The resolver here decides — purely from what is on +disk — whether a fresh invocation *starts* a new epoch or *resumes* one that crashed partway, so a +crash between "improve wrote the version" and "valid bench finished" re-enters the SAME epoch and +completes it rather than starting another. +""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass + +from acumen.skills import available_versions, next_version + + +@dataclass(frozen=True) +class EpochPlan: + """What one ``acumen epoch`` invocation will do, resolved from state.""" + + #: The arm learned from this epoch: a skill version, or ``None`` for the ``noskill`` baseline. + parent_version: str | None + #: The version this epoch produces, e.g. ``"v1"`` or ``"v2"``. + new_version: str + #: Whether this is the first epoch (``parent_version is None``), which also benches the + #: ``noskill`` baseline on ``valid`` for the report. + first: bool + #: Whether ``new_version`` already exists on disk (a resumed epoch), so ``improve`` is skipped. + resumed: bool + + +def resolve_epoch(skills_root, *, valid_complete: Callable[[str], bool]) -> EpochPlan: + """Decide the epoch to run from the skills on disk and each version's valid-bench state. + + Parameters + ---------- + skills_root + The ``skills/`` root. + valid_complete + Predicate: does this version have a complete ``valid`` bench? Injected so the resolver + stays pure and unit-testable (the caller builds it from ``build_matrix``/``pending``). + + Returns + ------- + The resolved :class:`EpochPlan`. + + Notes + ----- + - No versions yet → first epoch: ``parent=None`` (noskill), ``new=v1``. + - Latest version exists but its valid bench is unfinished → resume that epoch: ``new=latest``, + ``parent`` is the version below it (or ``None`` when latest is ``v1``), ``resumed=True``. + - Otherwise → fresh epoch: ``parent=latest``, ``new=next``. + """ + versions = available_versions(skills_root) + latest = versions[-1] if versions else None + + if latest is not None and not valid_complete(latest): + parent = versions[-2] if len(versions) >= 2 else None + return EpochPlan(parent_version=parent, new_version=latest, first=parent is None, resumed=True) + + new = next_version(skills_root) + return EpochPlan(parent_version=latest, new_version=new, first=latest is None, resumed=False) + + +def completed_epochs(skills_root, *, valid_complete: Callable[[str], bool]) -> int: + """How many epochs are fully finished — versions on disk whose ``valid`` bench is complete. + + Used to resume ``fit`` toward a global epoch target: the next epoch to run is + ``completed_epochs(...) + 1``. Takes the same injected ``valid_complete`` predicate as + :func:`resolve_epoch`, so it stays pure and unit-testable. + """ + return sum(1 for version in available_versions(skills_root) if valid_complete(version)) diff --git a/src/acumen/grade.py b/src/acumen/grade.py index d7d5858..c6c5f39 100644 --- a/src/acumen/grade.py +++ b/src/acumen/grade.py @@ -26,12 +26,14 @@ "error", "provider_exhausted", "sandbox_blocked", + "connection_error", ] #: Reasons that mark a run as a harness failure rather than evidence about a model. A run #: with one of these is recorded for diagnosis but must never reach a report, an improve -#: pass, or a resume as though it measured something. -INVALID_REASONS: frozenset[Reason] = frozenset({"provider_exhausted", "sandbox_blocked"}) +#: pass, or a resume as though it measured something — and it stays pending so a later run +#: retries it (``connection_error`` is a transient network drop, not the model's fault). +INVALID_REASONS: frozenset[Reason] = frozenset({"provider_exhausted", "sandbox_blocked", "connection_error"}) _FENCE_RE = re.compile(r"^\s*```[^\n]*\n(?P.*?)\n?\s*```\s*$", re.DOTALL) _HEADER_LINE_RE = re.compile(r"^\s*#+\s") diff --git a/src/acumen/htmldiff.py b/src/acumen/htmldiff.py new file mode 100644 index 0000000..a7fb1d5 --- /dev/null +++ b/src/acumen/htmldiff.py @@ -0,0 +1,144 @@ +"""A side-by-side (split) HTML diff renderer, shared across acumen's HTML outputs. + +The report diffs each skill version against its parent; the transcript diffs a file edit's before +against its after. Both want the same thing — old on the left, new on the right, changed words +lit up — so the renderer lives here and neither page owns it. It is pure stdlib +(``difflib``/``re``/``html``) so the transcript can use it without importing the report (and its +matplotlib/pandas weight). The styling lives in :data:`acumen.theme.DIFF_CSS`; this module emits +only class names. +""" + +from __future__ import annotations + +import difflib +import html +import re +from dataclasses import dataclass + +#: Unchanged lines kept on either side of a change, as in ``diff -u``. Longer untouched runs +#: collapse to a single marker row, so a change reads as only what actually moved. +_DIFF_CONTEXT = 3 + +#: Below this line-similarity the two sides of a replaced row are treated as unrelated text, +#: so the row is tinted whole instead of being picked apart into confetti-sized highlights. +_DIFF_INLINE_RATIO = 0.5 + + +@dataclass(frozen=True) +class _DiffRow: + """One row of a split diff: the same logical line on each side, either side possibly absent. + + ``kind`` is the change type from :class:`difflib.SequenceMatcher` (``equal``, ``replace``, + ``delete``, ``insert``), plus ``gap`` for the marker standing in for an elided run of + unchanged lines. A side is ``None`` where that version has no line there at all — a pure + insertion has no left-hand text — and renders as an inert filler cell. + """ + + kind: str + left_no: int | None + left: str | None + right_no: int | None + right: str | None + + +def _equal_rows(before: list[str], after: list[str], i: int, j: int, count: int) -> list[_DiffRow]: + """``count`` rows of unchanged text, starting at line ``i`` on the left and ``j`` on the right.""" + return [_DiffRow("equal", i + k + 1, before[i + k], j + k + 1, after[j + k]) for k in range(count)] + + +def _split_diff_rows(before: list[str], after: list[str]) -> list[_DiffRow]: + """Align two versions of a file into side-by-side rows, old on the left, new on the right. + + Changed runs pair off line by line, so a rewritten paragraph sits opposite its replacement + rather than being stacked below it; where one side runs out, the other continues against + filler. Unchanged stretches beyond :data:`_DIFF_CONTEXT` lines from any change collapse to + a gap row. + """ + rows: list[_DiffRow] = [] + opcodes = difflib.SequenceMatcher(a=before, b=after, autojunk=False).get_opcodes() + for index, (tag, i1, i2, j1, j2) in enumerate(opcodes): + if tag == "equal": + head = _DIFF_CONTEXT if index > 0 else 0 + tail = _DIFF_CONTEXT if index < len(opcodes) - 1 else 0 + if i2 - i1 > head + tail: + rows += _equal_rows(before, after, i1, j1, head) + rows.append(_DiffRow("gap", None, None, None, None)) + rows += _equal_rows(before, after, i2 - tail, j2 - tail, tail) + else: + rows += _equal_rows(before, after, i1, j1, i2 - i1) + continue + left, right = before[i1:i2], after[j1:j2] + for k in range(max(len(left), len(right))): + has_left, has_right = k < len(left), k < len(right) + rows.append( + _DiffRow( + tag, + i1 + k + 1 if has_left else None, + left[k] if has_left else None, + j1 + k + 1 if has_right else None, + right[k] if has_right else None, + ) + ) + return rows + + +def _inline_pair(left: str, right: str) -> tuple[str, str]: + """Both sides of a replaced line, escaped, with the words that differ wrapped in ````. + + The comparison runs over words rather than characters, so a changed word lights up whole + instead of down to the letters it happens to share with its replacement. Lines too + dissimilar to be a rewrite of one another are left unmarked — highlighting nearly every + word says less than the row tint already does. + """ + a, b = re.findall(r"\w+|\W", left), re.findall(r"\w+|\W", right) + matcher = difflib.SequenceMatcher(a=a, b=b, autojunk=False) + if matcher.ratio() < _DIFF_INLINE_RATIO: + return html.escape(left), html.escape(right) + marked = ["", ""] + for tag, i1, i2, j1, j2 in matcher.get_opcodes(): + for side, tokens, start, end in ((0, a, i1, i2), (1, b, j1, j2)): + chunk = html.escape("".join(tokens[start:end])) + if chunk: + marked[side] += chunk if tag == "equal" else f"{chunk}" + return marked[0], marked[1] + + +def _diff_cells(number: int | None, text: str | None, marked: str | None, cls: str) -> str: + """The line-number and content cell for one side of a row, or filler where that side is empty.""" + if text is None: + return '' + body = marked if marked is not None else html.escape(text) + return f'{number}{body or " "}' + + +def split_diff_table(rel: str, before: list[str], after: list[str], labels: tuple[str, str]) -> str: + """One file's split diff as a table: line numbers and text for each version, side by side.""" + body: list[str] = [] + for row in _split_diff_rows(before, after): + if row.kind == "gap": + body.append('…') + continue + left_mark = right_mark = None + if row.kind == "replace" and row.left is not None and row.right is not None: + left_mark, right_mark = _inline_pair(row.left, row.right) + left_cls = "diff-ctx" if row.kind == "equal" else "diff-del" + right_cls = "diff-ctx" if row.kind == "equal" else "diff-add" + body.append( + "" + + _diff_cells(row.left_no, row.left, left_mark, left_cls) + + _diff_cells(row.right_no, row.right, right_mark, right_cls) + + "" + ) + head = ( + f'{html.escape(labels[0])}' + f'{html.escape(labels[1])}' + ) + return ( + f'
{html.escape(rel)}
' + f'{head}' + f"{''.join(body)}
" + ) + + +#: Backwards-compatible alias — ``report.py`` historically named this ``_split_diff_table``. +_split_diff_table = split_diff_table diff --git a/src/acumen/improve.py b/src/acumen/improve.py index 23ee1b0..3dbadbd 100644 --- a/src/acumen/improve.py +++ b/src/acumen/improve.py @@ -1,26 +1,25 @@ -"""The improving agent: read train evidence, write ``skills/v{n+1}/``. - -Where the drafter reads the package *source* to write the first skill, the improver -reads how the current skill *performed* — on the TRAIN split only — and edits it. Versions -are immutable: ``improve`` always writes a new directory, never mutates an existing one. - -The load-bearing constraint is that the improver must never see test results, or the whole -benchmark is worthless. This is enforced two ways, not one: - -1. **Structurally.** The agent's ``cwd`` is a throwaway work dir and its readable material - is a *curated copy* of train-split runs. The real ``runs/`` tree is not on its ``cwd``, - not in the read-only roots, and not in scope — so there is no test directory to reach in the - first place. -2. **Belt-and-braces.** A ``PreToolUse`` hook denies any tool call whose path resolves under - a ``runs/*/test/`` subtree, so even a call that names an absolute path back into the - project runs tree is refused. See :func:`find_test_access`. - -Prompt-level instruction alone is explicitly *not* sufficient. +"""The improving agent: read the knowledge wiki, write the next ``skills/vN/``. + +The improver no longer reads a raw dump of the parent skill's runs. It reads the **wiki** — the +distilled, per-task ``observations.md``/``hypothesis.md`` notes that :mod:`acumen.wiki` builds from +train-split runs and accumulates across versions — plus the parent skill (in improve mode), the +raw train transcripts (for drill-down), and the *filtered* package source. On the first epoch there +is no parent skill: the wiki holds only the ``noskill`` baseline and the improver **creates** v1. + +Two load-bearing isolations, both structural rather than by prompt: + +1. **No held-out (valid) results.** The improver's readable material is the wiki + staged train + transcripts — the real ``runs/`` tree is denied wholesale, and a ``PreToolUse`` hook refuses any + call that resolves under ``runs/*/valid/`` wherever it is pointed (see :func:`find_valid_access`). +2. **No skill bias from the target.** A package may already ship its own skill/agent guidance; the + improver reads the *filtered* source (:func:`acumen.scrub.build_filtered_source`) with those + artifacts stripped, and the raw checkout denied, so the optimization never re-serves a skill the + package already carries. This matters precisely because acumen's job is to ship skills into + packages. """ from __future__ import annotations -import json import shutil import tempfile from dataclasses import dataclass @@ -36,31 +35,31 @@ from acumen.logs import LiveLog from acumen.paths import ( ANSWER_FILE, - RESULT_FILE, SCRIPT_FILE, TRANSCRIPT_HTML, - TRANSCRIPT_JSONL, arm_name, - parse_run_dir, ) from acumen.prices import PriceTable, price_usage, pricer, resolve_cost from acumen.procs import label_env, reap from acumen.prompts import improve_prompt +from acumen.scrub import build_filtered_source, make_skill_guard from acumen.skills import ( SKILL_FILE, Skill, SkillError, content_files, + latest_version, load_skill, next_version, skill_dir, write_meta, ) from acumen.tasks import Task +from acumen.wiki import WIKI_DIRNAME, collect_arm_runs -#: The subtree component that marks a held-out split. A path under ``runs//test/…`` is +#: The subtree component that marks the held-out split. A path under ``runs//valid/…`` is #: what the improver must never reach. -_TEST_SPLIT = "test" +_VALID_SPLIT = "valid" #: tool_input keys that carry a filesystem path across the default Claude Code toolset. _PATH_KEYS = ("file_path", "path", "notebook_path", "filename") @@ -75,35 +74,13 @@ class ImproveError(RuntimeError): """Raised when a skill could not be improved.""" -@dataclass(frozen=True) -class TrainRun: - """One train-split benchmark run of the parent skill, as evidence for the improver. - - ``success`` and ``skill_loaded`` are independent outcomes with independent fixes: a run - that never loaded the skill says nothing about the skill's body, only about its - ``description``. Both reach the improver so it can tell the two apart. - """ - - task_id: str - model: str - rep: int - prompt: str - expected: str - answer: str | None - reason: str - success: bool - directory: Path - #: Whether the agent actually invoked the skill. ``None`` when the transcript could not - #: be read — undetermined, which is not the same as "did not load". - skill_loaded: bool | None = None - - @dataclass(frozen=True) class ImproveResult: - """A newly improved skill version and what it cost to produce.""" + """A newly created or improved skill version and what it cost to produce.""" skill: Skill - parent: str + #: The parent version, or ``None`` when this is the first (created) skill. + parent: str | None cost_usd: float | None turns: int n_train_runs: int @@ -113,198 +90,64 @@ class ImproveResult: log_html: Path | None = None -# ── Train evidence ───────────────────────────────────────────────────────────────────── - - -def collect_train_runs(runs_root: Path, arm: str, tasks: list[Task]) -> list[TrainRun]: - """Collect the parent arm's train-split runs into evidence for the improver. +# ── Staging ────────────────────────────────────────────────────────────────────────────── - Walks ``runs_root//train`` for ``result.json`` files and pairs each with its task's - train prompt and ground-truth answer. Only train runs are ever collected — the test - subtree is never even visited. - Parameters - ---------- - runs_root - The ``runs/`` root. - arm - The arm whose train runs are the improvement signal, e.g. ``"skill_v1"``. - tasks - The loaded tasks, to recover each run's prompt and expected answer. +def _seed_staging(staging: Path, parent: Skill) -> None: + """Pre-fill the staging dir with the parent skill's content, so the agent edits in place. - Returns - ------- - The train runs, failures first then by task/rep — so a reader sees what went wrong up - front. + Only content files are copied — ``meta.json`` is acumen bookkeeping and is written fresh once + the new version is promoted. """ - train_root = runs_root / arm / "train" - by_id = {task.id: task for task in tasks} - runs: list[TrainRun] = [] - if not train_root.is_dir(): - return runs - for result_path in sorted(train_root.rglob(RESULT_FILE)): - try: - data = json.loads(result_path.read_text()) - except (OSError, ValueError) as err: - raise ImproveError(f"cannot read {result_path}: {err}") from err - if data.get("valid", True) is False: - raise ImproveError( - "cannot improve from an infrastructure-invalid benchmark result: " - f"{result_path} ({data.get('reason')}). Fix the harness — replenish the provider " - "credential, or report the sandbox refusal as a bug if it refused a host — " - "and resume the benchmark first." - ) - key = parse_run_dir(runs_root, result_path.parent) - task = by_id.get(key.task_id) - if task is None: - # A run for a task no longer in tasks.yaml is stale evidence; skip it rather - # than guess a prompt for it. - continue - runs.append( - TrainRun( - task_id=key.task_id, - model=str(data.get("model", key.model)), - rep=key.rep, - prompt=task.train.prompt.strip(), - expected=task.train.answer.strip(), - answer=data.get("answer"), - reason=str(data.get("reason", "")), - success=bool(data.get("success", False)), - directory=result_path.parent, - skill_loaded=_loaded(data.get("skill_loaded")), - ) - ) - runs.sort(key=lambda r: (r.success, r.task_id, r.model, r.rep)) - return runs - - -def _loaded(value: Any) -> bool | None: - """Coerce a ``result.json`` ``skill_loaded`` field, preserving "undetermined" as ``None``.""" - return None if value is None else bool(value) - - -def load_rates(runs: list[TrainRun]) -> dict[str, tuple[int, int, int]]: - """Count how often the skill actually loaded, per model. - - Reported per model rather than pooled because the load rate varies far more across models - than across skill versions — a pooled number invites the improver to chase a model whose - behaviour no wording can reach. + staging.mkdir(parents=True, exist_ok=True) + for src in content_files(parent.directory): + dest = staging / src.relative_to(parent.directory) + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(src, dest) - Parameters - ---------- - runs - The train runs to tally. - Returns - ------- - ``{model: (loaded, total, undetermined)}``, where ``undetermined`` counts runs whose - transcript could not be read and which are therefore excluded from ``loaded``. - """ - rates: dict[str, tuple[int, int, int]] = {} - for run in runs: - loaded, total, unknown = rates.get(run.model, (0, 0, 0)) - rates[run.model] = ( - loaded + (run.skill_loaded is True), - total + 1, - unknown + (run.skill_loaded is None), - ) - return dict(sorted(rates.items())) - - -def _load_mark(loaded: bool | None) -> str: - """The phrase naming a run's load outcome, used identically in SUMMARY.md and run.md.""" - if loaded is None: - return "skill load UNDETERMINED" - return "skill LOADED" if loaded else "skill NOT LOADED" - - -def _load_section(runs: list[TrainRun]) -> list[str]: - """The SUMMARY.md section reporting per-model load rates.""" - rates = load_rates(runs) - lines = [ - "## Did the skill load at all?", - "", - "A run where the skill never loaded is not evidence about the skill's body — the agent", - "never read it. Only the `description` in the frontmatter decides whether a skill loads.", - "", - "| model | loaded | runs | rate |", - "| --- | ---: | ---: | ---: |", - ] - total_loaded = total_runs = total_unknown = 0 - for model, (loaded, total, unknown) in rates.items(): - total_loaded += loaded - total_runs += total - total_unknown += unknown - lines.append(f"| `{model}` | {loaded} | {total} | {loaded / total:.0%} |") - if total_runs: - lines.extend(["", f"**Overall: {total_loaded}/{total_runs} ({total_loaded / total_runs:.0%}).**"]) - if total_unknown: - noun = "run" if total_unknown == 1 else "runs" - lines.append( - f"({total_unknown} {noun} had no readable transcript, so loading is undetermined and is " - "counted as not loaded above.)" - ) - lines.append("") - return lines +def _stage_wiki(wiki_root: Path, dest: Path) -> None: + """Copy the whole wiki into the agent's work dir, so it never reaches back into the project.""" + dest.mkdir(parents=True, exist_ok=True) + if wiki_root.is_dir(): + shutil.copytree(wiki_root, dest, dirs_exist_ok=True) -def _write_material(train_dir: Path, runs: list[TrainRun]) -> None: - """Lay out the curated train evidence the improver reads. +def _stage_transcripts(runs_root: Path, arm: str, tasks: list[Task], dest: Path) -> int: + """Copy the parent arm's train transcripts into the work dir for drill-down. - A ``SUMMARY.md`` digest (per-model load rates, then runs with failures first) plus one - directory per run holding the run's ``script.py``, transcript, and a ``run.md`` stating - prompt/expected/actual/outcome and whether the skill loaded. Copied, not linked, so the - agent never has a path back into the real runs tree. + Returns the number of runs staged. Copies transcript/script/answer per run into + ``dest//__rep_/`` — copies, so there is no path back into the real tree. """ - train_dir.mkdir(parents=True, exist_ok=True) - n_fail = sum(1 for r in runs if not r.success) - noun = "run" if len(runs) == 1 else "runs" - lines = [ - "# Train-split evidence for the current skill", - "", - "Each run below benchmarked the current skill on one train task. Two things were", - "recorded per run, and they fail for different reasons and take different fixes:", - "whether the agent got the answer RIGHT, and whether the agent LOADED the skill at", - "all. Read the load rates first, then the failing runs. Open a run's `run.md`,", - "`script.py`, and `transcript.html` for the detail.", - "", - f"**{len(runs)} {noun}, {n_fail} failing.**", - "", - *_load_section(runs), - "## Runs", - "", - ] + runs = collect_arm_runs(runs_root, arm, tasks, split="train") for run in runs: - slug = f"{run.task_id}__{run.model}__rep_{run.rep}" - mark = "PASS" if run.success else "FAIL" - lines.append( - f"- `{slug}/` — **{mark}** ({run.reason}) — {_load_mark(run.skill_loaded)} — " - f"task `{run.task_id}` on `{run.model}` — expected `{run.expected}`, got `{run.answer!r}`" - ) - run_out = train_dir / slug + run_out = dest / run.task_id / f"{run.model}__rep_{run.rep}" run_out.mkdir(parents=True, exist_ok=True) - (run_out / "run.md").write_text( - f"# {run.task_id} — {mark} ({run.reason})\n\n" - f"- Model: `{run.model}`\n" - f"- Skill: {_load_mark(run.skill_loaded)}\n\n" - f"## Task prompt\n\n{run.prompt}\n\n" - f"## Expected answer\n\n{run.expected}\n\n" - f"## Answer the agent gave\n\n{run.answer!r}\n\n" - f"## Files\n\n- `script.py` — the script the agent wrote\n" - f"- `transcript.html` — the full run transcript\n" - ) - for name in (SCRIPT_FILE, ANSWER_FILE, TRANSCRIPT_HTML, TRANSCRIPT_JSONL): + for name in (TRANSCRIPT_HTML, SCRIPT_FILE, ANSWER_FILE): src = run.directory / name if src.is_file(): shutil.copyfile(src, run_out / name) - (train_dir / "SUMMARY.md").write_text("\n".join(lines) + "\n") + return len(runs) -# ── Test-access guard ────────────────────────────────────────────────────────────────── +def _validate_staged(staging: Path, skill_name: str) -> None: + """Fail loudly if the agent's output isn't a usable skill, before it becomes a version.""" + if not (staging / SKILL_FILE).is_file(): + raise ImproveError( + f"the improving agent left no {SKILL_FILE} in the staging directory — nothing to " + "promote. Inspect the run log or the prompt." + ) + try: + load_skill(staging.parent, staging.name, expect_name=skill_name) + except SkillError as err: + raise ImproveError(f"the improved skill is not valid: {err}") from err + + +# ── Valid-access guard ───────────────────────────────────────────────────────────────────── def _blocks(candidate: str, runs_root: Path) -> bool: - """Return whether ``candidate`` resolves under a ``runs/*/test/`` subtree.""" + """Return whether ``candidate`` resolves under a ``runs/*/valid/`` subtree.""" try: resolved = Path(candidate).expanduser().resolve() except (OSError, RuntimeError, ValueError): @@ -314,17 +157,17 @@ def _blocks(candidate: str, runs_root: Path) -> bool: except ValueError: return False parts = rel.parts - # runs_root//test/... — the split is the second component. - return len(parts) >= 2 and parts[1] == _TEST_SPLIT + # runs_root//valid/... — the split is the second component. + return len(parts) >= 2 and parts[1] == _VALID_SPLIT -def find_test_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path) -> str | None: - """Return the first path in a tool call that reaches held-out test results, else ``None``. +def find_valid_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path) -> str | None: + """Return the first path in a tool call that reaches held-out valid results, else ``None``. - Pure and side-effect free, so the enforcement can be exercised directly without - standing up an agent. Checks the path-bearing tool_input keys, and — for shell tools — - the whitespace/metacharacter-split tokens of the command, since a Bash call can name a - path no structured field would. + Pure and side-effect free, so the enforcement can be exercised directly without standing up an + agent. Checks the path-bearing tool_input keys, and — for shell tools — the whitespace/ + metacharacter-split tokens of the command, since a Bash call can name a path no structured + field would. Parameters ---------- @@ -337,7 +180,7 @@ def find_test_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path Returns ------- - The offending path string, or ``None`` if the call touches no test results. + The offending path string, or ``None`` if the call touches no valid results. """ for key in _PATH_KEYS: value = tool_input.get(key) @@ -352,8 +195,8 @@ def find_test_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path return None -def make_test_guard(runs_root: Path) -> HookMatcher: - """Build the ``PreToolUse`` hook that denies any access to held-out test results. +def make_valid_guard(runs_root: Path) -> HookMatcher: + """Build the ``PreToolUse`` hook that denies any access to held-out valid results. ``matcher=None`` fires the hook for every tool. The hook resolves paths against the real project ``runs/`` root, so it holds regardless of the agent's ``cwd``. @@ -365,7 +208,7 @@ def make_test_guard(runs_root: Path) -> HookMatcher: root = runs_root.resolve() async def guard(input_data: dict[str, Any], tool_use_id: str | None, context: Any) -> dict[str, Any]: - hit = find_test_access( + hit = find_valid_access( input_data.get("tool_name", ""), input_data.get("tool_input", {}) or {}, root, @@ -377,7 +220,7 @@ async def guard(input_data: dict[str, Any], tool_use_id: str | None, context: An "hookEventName": "PreToolUse", "permissionDecision": "deny", "permissionDecisionReason": ( - f"acumen blocks the improver from reading held-out test results ({hit}). " + f"acumen blocks the improver from reading held-out valid results ({hit}). " "Only train-split evidence is available to you." ), } @@ -386,33 +229,7 @@ async def guard(input_data: dict[str, Any], tool_use_id: str | None, context: An return HookMatcher(matcher=None, hooks=[guard]) -# ── Orchestration ────────────────────────────────────────────────────────────────────── - - -def _seed_staging(staging: Path, parent: Skill) -> None: - """Pre-fill the staging dir with the parent skill's content, so the agent edits in place. - - Only content files are copied — ``meta.json`` is acumen bookkeeping and is written - fresh once the new version is promoted. - """ - staging.mkdir(parents=True, exist_ok=True) - for src in content_files(parent.directory): - dest = staging / src.relative_to(parent.directory) - dest.parent.mkdir(parents=True, exist_ok=True) - shutil.copyfile(src, dest) - - -def _validate_staged(staging: Path, skill_name: str) -> None: - """Fail loudly if the agent's output isn't a usable skill, before it becomes a version.""" - if not (staging / SKILL_FILE).is_file(): - raise ImproveError( - f"the improving agent left no {SKILL_FILE} in the staging directory — nothing to " - "promote. Inspect the run log or the prompt." - ) - try: - load_skill(staging.parent, staging.name, expect_name=skill_name) - except SkillError as err: - raise ImproveError(f"the improved skill is not valid: {err}") from err +# ── Orchestration ────────────────────────────────────────────────────────────────────────── async def improve_skill( @@ -421,6 +238,7 @@ async def improve_skill( target: Target, skills_root: Path, runs_root: Path, + wiki_root: Path, tasks: list[Task], auth_mode: AuthMode = "session", prices: PriceTable | None = None, @@ -431,55 +249,67 @@ async def improve_skill( feedback: str | None = None, log: LiveLog | None = None, ) -> ImproveResult: - """Improve the current skill into the next version from its train-split evidence. + """Create or improve a skill from the knowledge wiki. + + When ``parent_version`` is ``None`` and no skill versions exist, the wiki holds only the + ``noskill`` baseline and the agent CREATES v1 from it + the filtered source. Otherwise it + improves ``parent_version`` (default: latest) into the next version, editing a pre-filled copy. Parameters ---------- cfg The pass config; supplies ``skill_name`` and ``meta_model``. target - The prepared target, supplying the interpreter (for verifying claims). Its source - is **not** exposed — the improver works from evidence, not code. + The prepared target. Its source is exposed only as a FILTERED copy (bundled skills + stripped); the interpreter is available for verifying claims. skills_root The ``skills/`` root; the new version is the next unused one. runs_root - The ``runs/`` root, read for train evidence and guarded against test access. + The ``runs/`` root, read for train transcripts and guarded against valid access. + wiki_root + The ``wiki/`` root, the improver's primary distilled signal. tasks - The loaded tasks, to pair runs with their prompts and expected answers. + The loaded tasks, to locate the parent arm's train transcripts. auth_mode - Which credential the improving agent authenticates with — ``"session"`` (the Claude - subscription) or ``"api"`` (see :func:`acumen.env.build_agent_env`). + Which credential the improving agent authenticates with. parent_version - The version to improve; defaults to the latest present. + The version to improve; defaults to the latest present, or ``None`` (create) when none + exist. model Override for the improving model; defaults to the config's. max_turns, max_usd - Turn and budget caps for the improving agent. **Unbounded by default** — the config's - ``max_turns``/``max_usd`` cap benchmark agents only; pass explicit caps to bound the - improver. + Turn and budget caps. Unbounded by default — the config's caps bound benchmark agents only. feedback - Optional maintainer guidance, injected into the improve prompt as a subordinated block - and recorded in ``meta.json`` as provenance. It is prompt text only — it cannot reach - the held-out test split, which stays blocked structurally and by the guard hook. + Optional maintainer guidance, subordinated below the hard rules and recorded in + ``meta.json``. It cannot reach the held-out valid split. log A :class:`LiveLog` to stream the agent's messages to and render an HTML log from. Returns ------- - The improved skill, loaded and validated, with its parent and cost. + The created/improved skill, loaded and validated, with its parent and cost. """ - parent = load_skill(skills_root, parent_version or _latest_or_error(skills_root), expect_name=cfg.skill_name) + # Resolve parent: an explicit version, else the latest present, else None (create the first). + resolved_parent = parent_version if parent_version is not None else latest_version(skills_root) + parent = None if resolved_parent is None else load_skill(skills_root, resolved_parent, expect_name=cfg.skill_name) + parent_ver = parent.version if parent is not None else None + new_version = next_version(skills_root) dest = skill_dir(skills_root, new_version) if dest.exists(): raise ImproveError(f"{dest} already exists — skill versions are immutable and never overwritten") - arm = arm_name(parent.version) - train_runs = collect_train_runs(runs_root, arm, tasks) + parent_arm = arm_name(parent_ver) + train_runs = collect_arm_runs(runs_root, parent_arm, tasks, split="train") if not train_runs: + hint = ( + "acumen bench --no-skill --split train" + if parent_ver is None + else f"acumen bench --skill {parent_ver} --split train" + ) raise ImproveError( - f"no train-split runs found for {parent.version} under {runs_root / arm / 'train'} — " - f"run `acumen bench --skill {parent.version}` first so the improver has evidence to work from" + f"no train-split runs found for {parent_arm} under {runs_root / parent_arm / 'train'} — " + f"run `{hint}` first so the wiki and improver have evidence to work from" ) n_failures = sum(1 for r in train_runs if not r.success) @@ -487,7 +317,8 @@ async def improve_skill( try: work = holder / "work" staging = work / new_version - train_dir = work / "train" + wiki_dir = work / WIKI_DIRNAME + transcripts_dir = work / "transcripts" rationale_path = work / "rationale.md" home = holder / "home" selected_model = model or cfg.meta_model @@ -497,8 +328,14 @@ async def improve_skill( for path in (work, home, config_dir, home / "tmp"): path.mkdir(parents=True, exist_ok=True) - _seed_staging(staging, parent) - _write_material(train_dir, train_runs) + if parent is not None: + _seed_staging(staging, parent) + else: + staging.mkdir(parents=True, exist_ok=True) + _stage_wiki(wiki_root, wiki_dir) + _stage_transcripts(runs_root, parent_arm, tasks, transcripts_dir) + # The filtered source: a package's own shipped skill must not bias the optimization. + source_copy = build_filtered_source(target.src_dir, holder / "source") # Marks the agent's processes so the teardown below can find what it leaves running. env = label_env( @@ -516,35 +353,40 @@ async def improve_skill( prompt = improve_prompt( package=target.pkg_name, version=target.pkg_version, + src=source_copy, python=target.python, skill_dir=staging, - train_dir=train_dir, + wiki_dir=wiki_dir, + transcripts_dir=transcripts_dir, rationale_path=rationale_path, skill_name=cfg.skill_name, - parent_version=parent.version, new_version=new_version, + parent_version=parent_ver, feedback=feedback, ) options = AgentOptions( cwd=work, env=env, model=selected_model, - # No default turn or budget cap: only bound the agent if the caller asked. The - # config's ``max_turns``/``max_usd`` cap benchmark agents only. + # No default turn or budget cap: only bound the agent if the caller asked. max_turns=max_turns, max_usd=max_usd, # Codex reports no billed figure, so a budget cap needs the run's own rate table. price_usd=pricer(selected_model, table), - read_dirs=(target.venv_dir,), + # The filtered source + venv, for verifying claims. Never the raw checkout. + read_dirs=(source_copy, target.venv_dir), write_dirs=(work,), discover_skills=True, # Belt-and-braces over the structural isolation: refuse any call that reaches a - # held-out test result, wherever the agent points it. Built only for Claude — the - # hook is an SDK object, and Codex gets the equivalent from ``deny_paths`` below. - claude_hooks={"PreToolUse": [make_test_guard(runs_root)]} if provider == "claude" else None, - # Codex gets only the staged train evidence and is denied the original - # run tree wholesale, which includes every held-out test artifact. - deny_paths=(runs_root.resolve(),), + # held-out valid result, or an unfiltered source artifact. Built only for Claude. + claude_hooks=( + {"PreToolUse": [make_valid_guard(runs_root), make_skill_guard(target.src_dir)]} + if provider == "claude" + else None + ), + # Codex gets only the staged evidence and filtered source; deny the run tree and the + # original checkout wholesale. + deny_paths=(runs_root.resolve(), target.src_dir.resolve()), ) result: AgentResult | None = None @@ -575,14 +417,14 @@ async def improve_skill( _validate_staged(staging, cfg.skill_name) - rationale = _read_rationale(rationale_path, result, parent.version, new_version) + rationale = _read_rationale(rationale_path, result, parent_ver, new_version) dest.parent.mkdir(parents=True, exist_ok=True) shutil.copytree(staging, dest) - write_meta(dest, parent=parent.version, rationale=rationale, feedback=feedback) + write_meta(dest, parent=parent_ver, rationale=rationale, feedback=feedback) skill = load_skill(skills_root, new_version, expect_name=cfg.skill_name) return ImproveResult( skill=skill, - parent=parent.version, + parent=parent_ver, cost_usd=resolve_cost( result.total_cost_usd, price_usage( @@ -604,20 +446,11 @@ async def improve_skill( shutil.rmtree(holder, ignore_errors=True) -def _latest_or_error(skills_root: Path) -> str: - from acumen.skills import latest_version - - latest = latest_version(skills_root) - if latest is None: - raise ImproveError(f"no skill versions found under {skills_root} — run `acumen draft` first, then bench it") - return latest - - -def _read_rationale(path: Path, result: AgentResult, parent: str, new_version: str) -> str: +def _read_rationale(path: Path, result: AgentResult, parent: str | None, new_version: str) -> str: """Recover the agent's rationale for ``meta.json``. - Prefers the ``rationale.md`` the prompt asks for; falls back to the agent's final - message, then to a plain provenance note, so a version always records why it exists. + Prefers the ``rationale.md`` the prompt asks for; falls back to the agent's final message, then + to a plain provenance note, so a version always records why it exists. """ if path.is_file(): text = path.read_text().strip() @@ -626,4 +459,4 @@ def _read_rationale(path: Path, result: AgentResult, parent: str, new_version: s final = (result.result or "").strip() if final: return final - return f"improved from {parent} to {new_version}" + return f"created {new_version}" if parent is None else f"improved from {parent} to {new_version}" diff --git a/src/acumen/logs.py b/src/acumen/logs.py index fb32df6..1300155 100644 --- a/src/acumen/logs.py +++ b/src/acumen/logs.py @@ -164,13 +164,17 @@ def finalize(self, *, config_dir: Path, work_dir: Path, result: ResultMessage | if result is None: return False prompt = getattr(result, "prompt", "") + # The run's authoritative usage — what it is billed on — so the footer matches result.json + # rather than re-summing the session file's per-message usage (which overcounts, since each + # turn re-counts the cached context). + usage = getattr(result, "usage", None) if getattr(result, "provider", "claude") == "codex": - traj = from_codex_events(getattr(result, "transcript", []), prompt=prompt) + traj = from_codex_events(getattr(result, "transcript", []), prompt=prompt, usage=usage) else: if not result.session_id: return False native = locate_transcript(config_dir, work_dir, result.session_id) - traj = from_claude_transcript(native, prompt=prompt) if native and native.is_file() else None + traj = from_claude_transcript(native, prompt=prompt, usage=usage) if native and native.is_file() else None if traj is None: return False write_trajectory_json(traj, self.jsonl_path.with_suffix(".trajectory.json")) diff --git a/src/acumen/markdown.py b/src/acumen/markdown.py new file mode 100644 index 0000000..2c5e371 --- /dev/null +++ b/src/acumen/markdown.py @@ -0,0 +1,158 @@ +"""A small, safe markdown-to-HTML renderer shared across acumen's HTML outputs. + +Both the transcript renderer (:mod:`acumen.trajectory`) and the report +(:mod:`acumen.report`) turn agent- or maintainer-authored markdown into HTML, and +neither wants an external renderer dependency. The renderer here is deliberately a +common-case subset of CommonMark — paragraphs, headings, bullet/ordered lists, fenced +code, pipe tables and inline emphasis — with everything HTML-escaped first, so no input +can inject markup and anything unrecognized falls through as plain text. +""" + +from __future__ import annotations + +import re +from html import escape + +#: How much of one fenced code block the page keeps. The full text lives in the source +#: beside the rendered HTML (e.g. the transcript JSONL). +OUTPUT_CAP = 20_000 + + +def clip(text: str) -> str: + """Truncate ``text`` to :data:`OUTPUT_CAP`, noting how much was dropped.""" + if len(text) <= OUTPUT_CAP: + return text + return text[:OUTPUT_CAP] + f"\n… {len(text) - OUTPUT_CAP} more characters" + + +def _md_inline(text: str) -> str: + """Render inline markdown (code, bold, italic, links) on one line, escaping HTML first.""" + codes: list[str] = [] + + def stash(match: re.Match[str]) -> str: + codes.append(match.group(1)) + return f"\x00{len(codes) - 1}\x00" + + text = re.sub(r"`([^`]+)`", stash, text) + text = escape(text) + text = re.sub(r"\[([^\]]+)\]\((https?://[^\s)]+)\)", r'\1', text) + text = re.sub(r"\*\*([^*]+)\*\*", r"\1", text) + text = re.sub(r"__([^_]+)__", r"\1", text) + text = re.sub(r"(?\1", text) + text = re.sub(r"(?\1", text) + return re.sub(r"\x00(\d+)\x00", lambda m: f"{escape(codes[int(m.group(1))])}", text) + + +def _table_cells(line: str) -> list[str]: + """Split one pipe-table row into cells, dropping the optional outer pipes.""" + stripped = line.strip() + stripped = stripped.removeprefix("|").removesuffix("|") + return stripped.split("|") + + +def _is_table_delimiter(line: str) -> bool: + """Whether ``line`` is a GFM table delimiter row (``| --- | :--: |``).""" + cells = _table_cells(line) + return "|" in line and bool(cells) and all(re.fullmatch(r"\s*:?-+:?\s*", cell) for cell in cells) + + +def _table_align(spec: str) -> str: + spec = spec.strip() + left, right = spec.startswith(":"), spec.endswith(":") + if left and right: + return "center" + if right: + return "right" + return "left" if left else "" + + +def _render_table(header: list[str], aligns: list[str], rows: list[list[str]]) -> str: + def cell(tag: str, text: str, index: int) -> str: + align = aligns[index] if index < len(aligns) else "" + style = f' style="text-align:{align}"' if align else "" + return f"<{tag}{style}>{_md_inline(text.strip())}" + + head = "" + "".join(cell("th", value, i) for i, value in enumerate(header)) + "" + body = "".join("" + "".join(cell("td", value, i) for i, value in enumerate(row)) + "" for row in rows) + return f"{head}{body}
" + + +def render_markdown(text: str) -> str: + """A small, safe markdown-to-HTML renderer — headings, lists, code, tables, inline. + + Deliberately a common-case subset (not full CommonMark): the input is paragraphs, bullet + lists, fenced code and inline emphasis. Everything is HTML-escaped, so no input can inject + markup, and anything unrecognized falls through as plain text. + """ + lines = text.replace("\r\n", "\n").replace("\r", "\n").split("\n") + out: list[str] = [] + para: list[str] = [] + items: list[str] = [] + list_tag = "" + + def flush_para() -> None: + if para: + out.append("

" + "
".join(_md_inline(line) for line in para) + "

") + para.clear() + + def flush_list() -> None: + nonlocal list_tag + if items: + out.append(f"<{list_tag}>" + "".join(f"
  • {_md_inline(it)}
    • " for it in items) + f"") + items.clear() + list_tag = "" + + i = 0 + while i < len(lines): + line = lines[i] + stripped = line.strip() + if stripped.startswith("```"): + flush_para() + flush_list() + i += 1 + code: list[str] = [] + while i < len(lines) and not lines[i].strip().startswith("```"): + code.append(lines[i]) + i += 1 + i += 1 # skip the closing fence + out.append(f"
    {escape(clip(chr(10).join(code)))}
    ") + continue + if "|" in stripped and i + 1 < len(lines) and _is_table_delimiter(lines[i + 1]): + flush_para() + flush_list() + header = _table_cells(line) + aligns = [_table_align(spec) for spec in _table_cells(lines[i + 1])] + i += 2 + rows: list[list[str]] = [] + while i < len(lines) and lines[i].strip() and "|" in lines[i]: + rows.append(_table_cells(lines[i])) + i += 1 + out.append(_render_table(header, aligns, rows)) + continue + if not stripped: + flush_para() + flush_list() + elif heading := re.match(r"(#{1,6})\s+(.*)", stripped): + flush_para() + flush_list() + level = min(len(heading.group(1)) + 2, 6) # start at h3 so content never out-shouts the page + out.append(f"{_md_inline(heading.group(2))}") + elif bullet := re.match(r"[-*+]\s+(.*)", stripped): + flush_para() + if list_tag and list_tag != "ul": + flush_list() + list_tag = "ul" + items.append(bullet.group(1)) + elif ordered := re.match(r"\d+[.)]\s+(.*)", stripped): + flush_para() + if list_tag and list_tag != "ol": + flush_list() + list_tag = "ol" + items.append(ordered.group(1)) + else: + flush_list() + para.append(stripped) + i += 1 + flush_para() + flush_list() + return "".join(out) diff --git a/src/acumen/paths.py b/src/acumen/paths.py index 18c4392..904f67b 100644 --- a/src/acumen/paths.py +++ b/src/acumen/paths.py @@ -13,9 +13,11 @@ from pathlib import Path from typing import Literal -Split = Literal["train", "test"] +# ``valid`` is the held-out split — the improver never sees it. ``test`` is reserved for a +# future train/valid/test three-way split; only train/valid exist today. +Split = Literal["train", "valid"] -SPLITS: tuple[Split, ...] = ("train", "test") +SPLITS: tuple[Split, ...] = ("train", "valid") NOSKILL_ARM = "noskill" diff --git a/src/acumen/prompts.py b/src/acumen/prompts.py index 0ba5c20..702bbe8 100644 --- a/src/acumen/prompts.py +++ b/src/acumen/prompts.py @@ -19,12 +19,12 @@ def feedback_block(feedback: str | None) -> str: ```` delimiters and placed (by the templates) *after* the hard rules, with explicit wording that it is guidance and does NOT override anything above it. That subordination is deliberate: the feedback comes from a trusted maintainer, but it must not be - able to talk an agent out of the isolation rules (test-split, skill-bias) or the anti-overfit + able to talk an agent out of the isolation rules (valid-split, skill-bias) or the anti-overfit rules — those are enforced structurally regardless, and the prompt says so. Note: feedback steers *within* each command's methodology; it cannot redefine it. It won't, for instance, stop ``tasks`` from running the package to verify answers, nor let ``improve`` - reach the test split to cheat. + reach the valid split to cheat. Parameters ---------- @@ -61,7 +61,10 @@ def feedback_block(feedback: str | None) -> str: - The target package (`{package}`) is already installed. Run Python with `{python}`, which is also `python` on your PATH. Do not create virtualenvs and do not install or upgrade packages. -- You have web access. Use it if it helps. +- Use only the already-installed `{package}` — it is the source of truth. Do NOT clone, + download, fetch, or `pip install`/`pip download` its source repository or source + distribution (for example from GitHub or PyPI). +- You have web access for general reference. Use it if it helps. # What you must leave behind @@ -102,190 +105,214 @@ def feedback_block(feedback: str | None) -> str: """ -DRAFT_PROMPT = """\ -You are writing an agent skill for the Python package `{package}` (version {version}). +#: Shared guidance for the description trigger and anti-overfit body rules, used by both the +#: create and improve prompts so the two never drift apart. +_SKILL_CRAFT = """\ +# The description is the trigger + +The `description` is the ONE sentence an agent sees before deciding whether to open the skill. +It is the entire loading decision. Its job is to make the skill load exactly when it is +relevant and not otherwise. + +- **State the goals a user would actually phrase**, in their words, not the package's. An agent + matches the description against the task in front of it. Name the outcomes and the kinds of + question the skill answers. +- **Do not overfit it.** Widening the description with the specific datasets, methods, or + phrasings from the tasks is cheating: it buys train load rate and loses on the held-out valid + split. Widen to the CATEGORY of goal, never to the instances you saw. +- **Do not oversell.** A description that claims coverage the body does not deliver loads the + skill on tasks it cannot help with, and costs every one of those runs its tokens for nothing. + +# How to write the body -A skill is documentation written for an agent, not for a human. Its only purpose is to -make an agent that has never used `{package}` succeed at real tasks with it on the first -try. It is not a tutorial, not a README, and not a sales pitch. +- **Organize around goals, not modules.** Structure the skill so a stated goal maps to the right + entry point and the right sequence of steps — do not just mirror the module layout. +- **Write what is not guessable.** The agent already knows Python and can read a traceback. Spend + words on what it would get WRONG by guessing: non-obvious defaults, required preprocessing, the + function that looks right but isn't, where results land, argument shapes, the right order of steps. +- **Generalise — never overfit.** NEVER name a specific dataset, parameter value, column, expected + answer, or task in the skill. It must help on tasks you have not seen; enumerating the cases you + saw is cheating and fails the valid split. +- **Progressive disclosure.** Keep `SKILL.md` short and route depth to `references/*.md`. An agent + pays for every token of `SKILL.md` on every task, including the ones where it is irrelevant. +- **Prefer removing text over adding it.** A correct short code example beats a paragraph. If a + passage changed no outcome, cut it. +- **Verify before you write.** Do not assert a default, a return type, or where an output lands + without confirming it in the installed package or the docs. No hedging — say what to do.""" -The agent arrives with a goal stated in plain English — what a user wants done, not which -function to call. The skill's job is to get it from that goal to a working result: route it -to the right entry point and the right sequence of steps, so it never has to reverse-engineer -that from the module layout. + +IMPROVE_PROMPT = """\ +You are improving an agent skill for the Python package `{package}` (version {version}). + +A skill is documentation written for an agent, not a human. Its only purpose is to make an +agent that has never used `{package}` succeed at real tasks with it on the first try. + +You are producing version {new_version} — an improvement of version {parent_version}. # What you can read -- The package's source is at `{src}`. Read it — the source, the docstrings, the examples, - the docs directory. This is the ground truth about how the package behaves. -- `{package}` is also installed; run `{python}` to check anything you are unsure about. - Verify claims before you write them down. +- `{skill_dir}` — the current skill ({parent_version}). This directory has been pre-filled with a + copy of it. EDIT THESE FILES IN PLACE; what they contain when you finish becomes {new_version}. +- `{wiki_dir}` — the KNOWLEDGE WIKI: one directory per task, each with `observations.md` (what + agents did across models and replicates, how often the skill loaded, and whether loading led to + success) and `hypothesis.md` (why it did or did not work). Entries are tagged `[version][model]` + and accumulate across versions — `noskill` is the baseline, then `v1`, `v2`, … READ THIS FIRST. + It is the distilled signal of what the skill gets right and wrong; the trend across versions + tells you what past changes did. +- `{transcripts_dir}` — the raw TRAIN-split transcripts behind the wiki, for the parent skill. + Drill in here only when the wiki is not specific enough to act on. +- `{src}` — the package source (filtered: any skill or agent-guidance files a package ships have + been stripped and are BLOCKED — write from the API, not from someone else's skill). `{package}` + is installed; run `{python}` to verify a claim before writing it. Do not install packages. +- You have web access if the published docs help. -# Ignore any existing skills or agent instructions — deliberately hidden +# What you are NOT allowed to see -The package may already ship skills or agent-instruction files written for it (`SKILL.md`, -`.agents/skills/`, `.claude/skills/`, `.codex/`, `CLAUDE.md`, `AGENTS.md`, `.cursor/`, -Copilot instructions). These have been stripped from the source above, and any attempt to reach -them — or the original unfiltered checkout — is BLOCKED. This is on purpose: a skill copied from -pre-written guidance is not a skill written from the package, and this one is about to be -measured against no skill at all. Write yours from the API, the source, and the user-facing docs. +You are optimising against the TRAIN split. A separate, held-out VALID split measures whether your +changes generalise rather than memorise these tasks. Any tool call that reaches valid results is +BLOCKED. Do not attempt it — reaching valid data would invalidate the whole benchmark. -# What you must write +# Two different failures, two different fixes -Your working directory is `{out}`. Write: +The wiki records whether the agent LOADED the skill and whether it SUCCEEDED. Separate them before +changing anything — they look identical in a list of failures and have opposite fixes. -1. `{out}/SKILL.md` — required. It must begin with YAML frontmatter, exactly: +1. **The skill never loaded.** The agent never read the body, so nothing in the body caused this + and nothing you write in the body can fix it. The only lever is the `description`. +2. **The skill loaded and the run still failed.** Now the body is on trial. Fix it, using the + cause the wiki (or a transcript) actually shows — not one you imagine. +3. **The skill loaded and the run passed.** Evidence the body works. Do not rewrite it for style. ---- -name: {skill_name} -description: ---- +A skill that is never loaded scores exactly like no skill at all. If the wiki shows load rates low +across the board, that is the biggest available win — and the only lever is the `description`. If +one model loads reliably and another almost never does, that gap is mostly the model's behaviour; +do not contort the sentence chasing it. - The `name` must be exactly `{skill_name}`. - - The `description` is load-bearing and must be HONEST. It is the only part of the skill - an agent sees before deciding whether to open it, and it is the only thing that gets - the skill loaded at the right moment. - State what the skill covers and the goals it applies to — name the outcomes a user would - actually phrase, so the skill loads when one of them comes up. Do not oversell it, and do - not claim coverage the body does not deliver. Use this format: - [What it does] + [When to use it: the goals/triggers it fires on] + [Key capabilities] - -2. `{out}/references/*.md` — optional. Use these for detail that only some tasks need. - -# How to write it - -- **Organize around goals, not modules.** Work out what people actually use `{package}` to - accomplish — read its examples, tutorials, and docs, not just its API — and structure the - skill so a stated goal maps to the right entry point and the right sequence of steps. Do not - just mirror the package's module layout. -- **Write what is not guessable.** An agent already knows Python and can read a - traceback. Spend your words on what it would get WRONG by guessing: non-obvious - defaults, required preprocessing, the function that looks right but isn't, where - results are written, argument shapes and orientation, footguns the API invites, right order - of steps to follow. -- **Generalize within each goal.** Organize around categories of goal, but keep the guidance - under each one general enough to cover any task in that category. Don't enumerate one-off - recipes — that's the failure mode on the other side of module-mirroring. -- **Progressive disclosure.** `SKILL.md` should be short and route to `references/` for - depth. An agent pays for every token of it on every task, including the tasks where it - is irrelevant. If `SKILL.md` is long, you are taxing every run. -- **Be concrete.** A correct short code example beats a paragraph of prose. Show the real - call, with the arguments that matter. -- **Prefer removing text over adding it.** Anything that merely restates the obvious is - worse than nothing: it costs tokens and buries the parts that matter. -- **No hedging.** Say what to do. - -# Verify before you finish - -Do not write claims you have not checked. If you assert a default value, a return type, -or where an output lands, confirm it in the source or by running `{python}`. A skill that -confidently states something false is worse than no skill at all — it will send an agent -in the wrong direction with full confidence. +{craft} + +# What you must write + +1. Edit the skill in place under `{skill_dir}`. When you finish, `{skill_dir}/SKILL.md` must still + begin with YAML frontmatter whose `name` is exactly `{skill_name}`. +2. `{rationale_path}` — one short paragraph stating WHAT you changed and WHY, grounded in the wiki. + Write it OUTSIDE the skill directory; do not put it inside `{skill_dir}`. {feedback} -When you are done, `{out}/SKILL.md` must exist and start with the frontmatter above. +When you are done, `{skill_dir}/SKILL.md` exists and starts with the frontmatter above, and +`{rationale_path}` contains your rationale. """ -IMPROVE_PROMPT = """\ -You are improving an agent skill for the Python package `{package}` (version {version}). - -A skill is documentation written for an agent, not a human. Its only purpose is to make an -agent that has never used `{package}` succeed at real tasks with it on the first try. +CREATE_PROMPT = """\ +You are writing the FIRST agent skill (version {new_version}) for the Python package `{package}` +(version {version}). -You are producing version {new_version} — an improvement of version {parent_version}. +A skill is documentation written for an agent, not a human. Its only purpose is to make an agent +that has never used `{package}` succeed at real tasks with it on the first try. It is not a +tutorial, not a README, not a sales pitch. The agent arrives with a goal in plain English; the +skill routes it to the right entry point and sequence of steps. # What you can read -- `{skill_dir}` — the current skill ({parent_version}). This directory has been pre-filled - with a copy of it. EDIT THESE FILES IN PLACE; what they contain when you finish becomes - version {new_version}. -- `{train_dir}` — evidence from benchmarking the current skill on the TRAIN split. Read - `{train_dir}/SUMMARY.md` first. For each run you will find the task prompt, the expected - answer, the answer the agent actually gave, whether it passed, WHETHER THE AGENT LOADED THE - SKILL AT ALL, which model ran it, the `script.py` it wrote, and its full transcript. This is - your only signal about what the skill gets right and what it gets wrong. -- `{package}` is installed; run `{python}` (also `python` on your PATH) to verify any - claim about the API before you write it down. Do not install or upgrade packages. +- `{wiki_dir}` — the KNOWLEDGE WIKI: one directory per task, each with `observations.md` (what + agents did WITHOUT any skill — the `noskill` baseline — across models and replicates, and how + often they got it right) and `hypothesis.md` (why they succeeded or failed). READ THIS FIRST: it + tells you exactly where an unaided agent goes wrong, which is precisely what your skill must fix. +- `{transcripts_dir}` — the raw TRAIN-split transcripts behind the wiki. Drill in when the wiki is + not specific enough. +- `{src}` — the package source (filtered: any skill or agent-guidance files a package ships have + been stripped and are BLOCKED — write from the API, the source, and the user-facing docs, not + from someone else's skill). `{package}` is installed; run `{python}` to verify claims. Do not + install packages. - You have web access if the published docs help. # What you are NOT allowed to see -You are optimising against a TRAIN split. A separate, held-out TEST split is what measures -whether your changes actually generalise rather than memorising these particular tasks. You -must not see the test split, and any tool call that reaches test results will be BLOCKED. -Do not attempt it — reaching test data would invalidate the whole benchmark. +The wiki and transcripts are from the TRAIN split only. A held-out VALID split measures whether +your skill generalises. Any tool call that reaches valid results is BLOCKED — do not attempt it. # What you must write -1. Edit the skill in place under `{skill_dir}`. When you finish, `{skill_dir}/SKILL.md` - must still begin with YAML frontmatter whose `name` is exactly `{skill_name}` and whose - `description` is an honest one-sentence statement of what the skill covers and when to - use it. The `description` is the ONLY thing that decides whether the skill loads — see - below. - -2. `{rationale_path}` — one short paragraph stating WHAT you changed and WHY, grounded in - the train evidence. Write it here, OUTSIDE the skill directory. Do not put the rationale - inside `{skill_dir}`. - -# Two different failures, two different fixes +Your staging directory is `{skill_dir}`. Write: -Every run records whether the agent LOADED the skill and whether it SUCCEEDED. Separate them -before you change anything — they look identical in a list of failures and have opposite fixes. +1. `{skill_dir}/SKILL.md` — required. It must begin with YAML frontmatter, exactly: -1. **The skill never loaded.** The agent never read a word of the body, so nothing in the body - caused this and nothing you write in the body can fix it. The only lever is the - `description`. Editing the body in response to these runs is wasted work. -2. **The skill loaded and the run still failed.** Now the body is on trial: it steered the - agent wrong, or it was silent where it should have spoken. Fix the body, using the cause - you can see in that run's transcript — not one you imagine. -3. **The skill loaded and the run passed.** Evidence the body works. Do not rewrite it for - style. +--- +name: {skill_name} +description: +--- -A skill that is never loaded scores exactly like no skill at all, however good its body is. If -the load rate in `SUMMARY.md` is low, that is the biggest available win — and the only thing -you can do about it is the `description`. + The `name` must be exactly `{skill_name}`. The `description` is load-bearing and must be HONEST — + it is the only thing an agent sees before deciding whether to open the skill. -# The description is the trigger +2. `{skill_dir}/references/*.md` — optional, for detail only some tasks need. -The `description` is the ONE sentence an agent sees before deciding whether to open the skill. -It is the entire loading decision. Its job is to make the skill load exactly when it is -relevant and not otherwise. - -- **Read the load rates per model.** They are in `SUMMARY.md`, broken down by model. If one - model loads the skill reliably and another almost never does, that gap is mostly the model's - behaviour, not your wording — do not contort the sentence chasing it. Act on a rate that is - low across the board. -- **State the goals a user would actually phrase**, in their words, not the package's. An - agent matches the description against the task in front of it. Name the outcomes and the - kinds of question the skill answers. -- **Do not overfit it to the train tasks.** Widening the description with the specific - datasets, methods, or phrasings you just read in the train runs is the same cheating as - putting them in the body: it buys train load rate and loses on the test split. Widen to - the CATEGORY of goal, never to the instances you saw. -- **Do not oversell.** A description that claims coverage the body does not deliver loads the - skill on tasks it cannot help with, and costs every one of those runs its tokens for nothing. +3. `{rationale_path}` — one short paragraph on what the skill covers and why, grounded in the + baseline wiki. Write it OUTSIDE `{skill_dir}`. -# How to improve the body - -- **Fix what the evidence shows is broken.** Work from the runs where the skill loaded and - the agent still failed. Address the cause visible in the transcript, not one you imagine. -- **Generalise — never overfit.** NEVER name a specific dataset, parameter value, column, - expected answer, or task from the train runs in the skill. The skill must help on tasks - you have not seen. Guidance that enumerates these particular cases is cheating and will - fail the test split. -- **Prefer removing text over adding it.** A shorter skill that an agent reads and follows - beats a longer one it skims. Every token is paid on every task, including the ones where - the skill is irrelevant. If a passage did not change any outcome, cut it. -- **Verify before you write.** Do not assert a default, a return type, or where an output - lands without confirming it in the installed package or the docs. -- **No hedging.** Say what to do. +{craft} {feedback} When you are done, `{skill_dir}/SKILL.md` exists and starts with the frontmatter above, and `{rationale_path}` contains your rationale. """ +WIKI_PROMPT = """\ +You are a benchmark analyst keeping a running knowledge wiki about how well an agent skill for the +Python package `{package}` helps on ONE task. Your job this round: read what happened when the +`[{version}]` arm was benchmarked on this task across several models and replicates, and record it +BRIEFLY into two files. + +`[{version}]` is the skill version under review. `noskill` means no skill was installed (the +baseline). {skill_note} + +# What you are given + +- `{evidence_dir}/INDEX.md` — every run of `[{version}]` on task `{task_id}`, one line each: pass or + fail, whether the skill LOADED, the model, the expected answer, and the answer given. +- `{evidence_dir}/__rep_/` — that run's `transcript.html` (what the agent actually did), + its `answer.md`, and its `script.py`. Open a few to understand the pattern; you do NOT need to + read every one. +- The package source is at `{src}` and it is installed — run `{python}` if a fact helps you explain + an outcome. Optional; do not go down a rabbit hole. +{skill_body_note} +# What you must write — APPEND, do not rewrite + +Two files already exist and may contain entries from earlier arms. LEAVE those untouched and ADD +your new entries at the end. `noskill` is the first arm; later arms are `v1`, `v2`, … + +1. `{observations_path}` — add ONE line per model, in EXACTLY this format: + + - [{version}][]: + + Say the general idea, not a play-by-play. For `noskill`, "loaded" does not apply — just say what + the agents did and how often they got it right. + +2. `{hypothesis_path}` — add ONE line per model, in EXACTLY this format: + + - [{version}][]: + +# BE BRIEF — this is the whole point + +This wiki is read in full by the skill improver every round and grows forever. A verbose wiki is +WORSE than a short one. Hard rules: + +- One to two lines per entry. No transcript quotes, no step-by-step, no per-run breakdown, no + restating the task. Compress across replicates into the pattern. +- The hypothesis is one or two short sentences. No essays, no hedging. + +GOOD observation: + - [{version}][claude-opus-5]: Loaded 3/3; agents used the right entry point and passed every run. +GOOD hypothesis: + - [{version}][claude-opus-5]: The skill named the correct function and output location, which is + the step agents otherwise guess wrong. +BAD (too verbose): a paragraph recounting each replicate's tool calls and reasoning. + +When you are done, `{observations_path}` and `{hypothesis_path}` each contain your new +`[{version}]` line(s) appended after whatever was already there — and nothing else changed. +""" + + TASKGEN_PROMPT = """\ You are writing a benchmark of real analysis tasks for a Python package (`{package}`). Each task states a GOAL a user has, in plain language, plus the single answer a correct analysis @@ -351,9 +378,9 @@ def feedback_block(feedback: str | None) -> str: - GOOD (a lazy goal): "Using the pbmc3k data, find which transcription factor is most active in the monocytes. Give only the factor's symbol." -# Train and test variants +# Train and valid variants -Give each task a train and a test variant of the SAME goal, differing only in the input or the +Give each task a train and a valid variant of the SAME goal, differing only in the input or the target it asks about (a different cell type, group, condition, or dataset). Two instances of one analysis with two different correct answers — so a skill cannot pass by memorising one answer. @@ -369,7 +396,7 @@ def feedback_block(feedback: str | None) -> str: The script you run to obtain an answer is NOT scratch. Save one per split to `{scripts_dir}/-.py`, using the SAME `id` you gave the task in `{out}` — so the task -`bulk` needs `{scripts_dir}/bulk-train.py` and `{scripts_dir}/bulk-test.py`. These are what +`bulk` needs `{scripts_dir}/bulk-train.py` and `{scripts_dir}/bulk-valid.py`. These are what `acumen check` reruns later to confirm the answer still holds, so each one must: - Be SELF-CONTAINED and runnable from ANY empty working directory: `