feat(v1): grade in an isolated box, with Harbor-native artifacts - #2144
Merged
Conversation
Grading in the box the agent worked in leaves a seam a policy under RL pressure
will find: an editable test file, tamperable grading state, an artifact that
leaks the answer. This carries only what a task declares into a second box and
grades there.
Two channels, non-overlapping:
- `Trace.info` stays the durable record (`patch`, `verdict`) and never travels.
- `/logs/artifacts/` is transport — Harbor's in-sandbox convention, collected
with no declaration, restored in the grading box at the original path
("no translation", as in Harbor).
`Task.finalize` is the producer hook — it already means "runtime live, agent
done, before scoring mutates anything", which is exactly Harbor's collect-hook
moment, so `[[verifier.collect]]` maps onto it rather than needing a new stage.
The env composes the boxes; no `Task.scoring_runtime`, and no change to the
`Runtime` teardown contract.
Harbor `task.toml` support: `artifacts = [...]` (string and object form,
`exclude` honored) and `[[verifier.collect]]`. Sidecar `service`, `[verifier].user`
and an explicit `[verifier.environment]` image are rejected at load; `destination`
is inert, being host-trial-directory placement that verifiers has no equivalent
for. A failing collect hook fails the rollout, unlike `harbor run` which logs and
continues — here the output is a grading input, not observability.
`agentic-judge` gains `--env.topology isolated|shared`, defaulting to isolated,
and stops overwriting the judge's runtime policy when it has its own box. The
judge's workspace note is topology-specific: told it stands in the agent's
workspace when it stands in a fresh one, it reads an unmodified tree as failure.
One archive per source rather than one combined tar: BusyBox tar (every
alpine-based image) has no `-r` to append, and per-source `exclude` patterns
cannot share a single create either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
rasdani
commented
Jul 27, 2026
rasdani
commented
Jul 27, 2026
Review pass on the module.
Removed, as dead or redundant:
- `Collected`, a wrapper over a list whose `total_bytes` nothing ever called;
`collect` now returns `list[CollectedArtifact]` directly.
- `_SYSTEM_ROOTS`, thirteen paths guarding against an author typing `/usr` as a
source. Sources come from task config, not the agent, and the size cap already
refuses anything image-sized. Only `/` needs rejecting, because `lstrip('/')`
makes it empty and tar then fails confusingly.
- the overlap check in `_normalize`. It was load-bearing when collection built one
combined tar; once `restore` began clearing every root before extracting any,
overlapping sources became duplicated bytes rather than a correctness bug.
- `_over_cap`, a function that only formatted a string, and `MAX_ARTIFACT_FILES`,
which counted what the in-box `wc -c` already caps.
Narrowed the link rule, which was both unsafe to relax and too blunt as written.
Refusing every symlink rejects real content — `/etc` and any repo carrying a
`.venv` — while dereferencing instead (`tar -h`) fails the whole collection on a
single dangling link. An outward link is only a way into the grading box when
another member sits underneath it, since that member is what gets written through
the link. So leaf links are kept, in-root and dangling links are kept, and the
write-through shape is refused.
224 lines from 356, then 260 with the link rule.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_vet` and `_escapes` defended a shape our own tooling cannot produce. Neither BusyBox nor GNU tar recurses into a symlinked directory, so `tar -c` never emits the write-through pair (`link -> /etc` plus a member at `link/passwd`) that the symlink analysis existed for; and tar strips leading `/` while no real path component is `..`, so the member-name checks were equally unreachable. What remained was a defence against one attacker: an agent that replaces the tar binary in its own box to poison the grading container. That is the only channel between the two boxes, but it is a remote attack, and fifty lines of subtle analysis to close it was a bad trade — the subtlety misfired twice while being written, first refusing every symlink (which would have broken any repo carrying a .venv) and then reading hardlink linknames in the wrong frame. Accepted risk: an agent that tampers with its own tar can place arbitrary files at arbitrary paths in the grading box. Note that extracting into a scratch directory instead would not have avoided the analysis — it closes relative traversal, but an absolute symlink target still redirects the write. 260 -> 198 lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`publish` came from Harbor's vocabulary for `/logs/artifacts/` ("the conventional
agent publish directory"), but nothing in this codebase names a parameter that
way. The dominant convention is `<what>_path` — `output_path`, `session_path`,
`script_path`, `config_path` — and `write_path` mirrors `Runtime.write`, which is
the call the argument drives. `output_path` was unavailable: it already means the
host-side eval output directory.
The failure key follows: `patch_publish_error` -> `patch_write_error`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The type only ever held `(root, archive)`, so a dict keyed by source path says the same thing with no class: it preserves declaration order, makes a duplicate root impossible by construction, and drops the last dataclass from the module. `collect() -> dict[str, bytes]`, `restore(runtime, collected)` unchanged at its call sites. 191 lines, from 356 at the start of the branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`release()` was runtime lifecycle wearing an artifacts hat. It read and wrote `runtime.stopped` — another object's lifecycle state — from a module that has nothing to do with lifecycle, and kept its own `_PENDING` set of unawaited teardowns while `runtimes/base.py` already owns exactly that kind of bookkeeping in `_LIVE`. It now sits next to the machinery it belongs to, as `Runtime.stop_nowait`: `stopped` is set on self rather than poked from outside, `_PENDING` sits beside `_LIVE` with the atexit backstop that covers both, and the `_nowait` suffix says what it is — `stop`, without waiting — instead of inventing a verb. `release` was also already taken in v1, by `RolloutSession.release`. This does touch `runtimes/base.py`, having earlier concluded it need not. That still holds for what it was about: adding a re-entry guard to `stop()` would have made a failed teardown unretryable. Adding a sibling method changes no existing semantics. artifacts.py is down to 168 lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured what the optimisation was buying: an awaited docker teardown is 66ms. For that it cost a public method on the Runtime ABC, a `_PENDING` set of unawaited tasks, and a defused context manager in env code — `pop_all()` to disarm `provision`, then taking ownership by hand — with a window between the two where a cancellation would orphan the box until the atexit backstop ran at process exit. It was also backwards on the resource question. Overlapping the solver's teardown with the judge's startup means an episode can hold two boxes at once; awaiting means it holds one. On a paid runtime that matters more than the latency does, and the latency is not on the throughput path anyway — the concurrency gate wraps the agent run, not teardown. The barrier is unchanged and was never the teardown: `collect()` returning is what gates everything downstream. `runtimes/base.py` is untouched again, byte-identical to main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_checked` rejected a relative source, a `..` component, and bare `/`. Those are author mistakes in task config, not anything an agent controls, and a capable author does not need the framework second-guessing a path they wrote. What remains is the one line that was doing work: stripping a trailing slash, so `/work` and `/work/` cannot key two entries for the same tree — the source doubles as the dict key and as restore's `rm -rf` target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rasdani
marked this pull request as ready for review
July 27, 2026 04:09
ApprovabilityVerdict: Needs human review This PR introduces a substantial new feature for isolated grading boxes with Harbor-native artifact transport. It adds new modules, new configuration options ( You can customize Macroscope's approvability policy. Learn more. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
`AgentConfig.runtime` defaults to `SubprocessConfig` and has no `None` to tell unset from chosen, so with `isolated` as the default topology the subprocess guard rejected every config that pinned only the solver — including the bundled `configs/agentic_judge.toml` and `test_env_id_agentic_judge`, which is how this surfaced. The env raised at construction before any episode ran. The judge now falls back to the solver's runtime policy when it has not pinned a container, which is the same treatment `harness`, `model` and `sampling` already get for unpinned seats in `Env._episode_agents`. It is also the right default on its own terms: the grading box is supposed to boot from the solver's image, which is what makes "only the delta travels" true. A judge that pins its own runtime keeps it under `isolated`; under `shared` the solver's still wins, since there the judge's effective runtime IS the solver's box. The explicit subprocess guard goes with it — unreachable now that the fallback runs first, and the solver-side check already covers both-are-subprocess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four conflicts, all from tooling churn rather than overlapping design — nobody else has been in artifacts.py, the Harbor artifact wiring, or the topology logic. - verifiers/v1/__init__.py, task.py, tasksets/harbor/taskset.py: import ordering. #2146 relocated `configs.task`/`configs.taskset` and #2148's isort re-sorted the package; my artifacts imports had landed on the same lines. Kept both sides. - envs/agentic_judge/env.py: #2147 changed the solver-runtime guard from `ValueError` to `TypeError` for Ruff 0.16, on the same line where I had rewritten the message (the check no longer means "the judge plays in the solver's box"). Took their exception type with my message. Also converted three pydantic mutable defaults to `Field(default_factory=...)` — `Artifact.exclude`, `TaskData.artifacts`, `HarborData.collect`. Not conflicts, but Ruff 0.16 flags them and CI would have failed on the merged result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Harbor permits a relative `source` for the main service, and the existence probe resolves one against the runtime's workdir — but the tar runs `-C /`, so it did not. `solution.txt` probed `$workdir/solution.txt`, passed, then archived `/solution.txt`. Not an error: a different file, collected and graded silently. Reproduced with a decoy at the root, which is what got collected. Resolution happens at collection rather than at parse: the workdir is a runtime property and `TaskData.workdir` can override it, so it isn't known when task.toml is read. Also applies review comments on the docs — drops a Harbor paragraph that repeated the Shortcomings list two lines below it, and settles on "sandbox" over "box" in the artifacts section. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`PurePosixPath` join already does the whole job — an absolute source discards the workdir, a relative one joins onto it, and a trailing slash normalises away — so the helper and its `rstrip` were both redundant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…failure `capture_patch` treated every failure the same — record `patch_error`, score the rollout anyway. Two different things were being conflated: - the box answered and git refused: a stale `index.lock` from a killed agent command, a deleted `.git`, a `base_commit` the agent rewrote away, a disk it filled. The agent's own environment, so it still records and still scores. A run with no patch grades as a run that changed nothing, which is the right reward. - the box never answered: sandbox gone, exec timed out, transport dropped. Nothing the policy did, so it now raises. Scoring it would feed training a zero that says only that our infrastructure failed. Telling them apart needs more than the exception boundary: `DockerRuntime.run` returns `docker exec`'s own non-zero result rather than raising, so a dead container is indistinguishable from broken git by exit code alone. On the failure path only, one `true` probe asks the box whether it is still there. The judge sandbox is no longer provisioned when the solver rollout errored — its scoring never ran either, so grading it spends a second box to reproduce a failure the trace already records. `finalize` tolerates the missing judge trace. Also corrects the `trace.info` documentation: it is not a transport channel, but an agentic judge does receive the whole serialized trace including `info` at `/tmp/trace.json`, so anything left there is visible to the grader. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comment argued only the cost — a second sandbox spent reproducing a known failure. The correctness argument was the one I got wrong twice while reviewing: `episode.ok` follows the failed trace on its own, and `episode_should_retry` classifies off that trace's errors, so the real exception type is what decides whether to retry. Raising an `EnvError` here would add a second, less specific error and bury it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI caught this: test_env_id_agentic_judge failed on one of three Python jobs and passed on the other two — not a version difference, a flaky judge. Under the isolated default it was put in a fresh box, told "the agent's environment is gone", and asked for a criterion that reads "you verified it with real execution". echo-v1 publishes nothing, so there was nothing to execute; two models talked themselves into a verdict and the third stalled without writing one. The fixture is not the real problem. Nothing in research-environments publishes to the box yet — `git grep -l "write_path\|CONVENTION_DIR\|logs/artifacts"` over its environments/ is empty, because `write_path` is new here and unused. So isolated-by-default would have changed grading for every consumer of that repo's judge setups, and the three plain-Harbor tasksets would have broken outright: their hints open with "Reconstruct the agent's change from the box: `git status`, `git diff`, `git log` in /testbed", which under isolation is a pristine checkout that grades every attempt as untouched. So the default returns to `shared` — the topology that has actually been measured — and `isolated` becomes opt-in per taskset, adopted once that taskset puts its evidence somewhere that travels. The seven capture_patch tasksets are nearly there already; the trace record they rely on does travel. The isolated path itself is unchanged and still verified: two sandboxes, solver's torn down first, artifacts restored at their original paths, trace uploaded, judge told it is in a fresh box. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`git add -A` staged everything untracked, so a captured patch carried files the task image shipped and the agent never opened. On R2E-Gym that is three per box (`datasets`, `install.sh`, `run_tests.sh`), and the resulting patch fails `git apply` in a fresh container of that same image — which is exactly what an isolated grading box is. The isolated-judge path was broken for every taskset whose image ships untracked files. Drop untracked files whose mtime predates the agent's first turn. The margin is not subtle: R2E-Gym's are dated 2025-01, the rollouts run now. Pass the cutoff as an age rather than an absolute timestamp — a remote sandbox runs on its own machine, and only a duration survives the clock skew. The listing has to happen before `add -A`, after which nothing is "other" any more. `ignore=` unstages named paths on top, for a taskset that knows its image better than the mtime rule does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mtime cutoff leans on the image's untracked files being old. R2E-Gym's are 556 days old, so it holds today — but it holds for no image we build ourselves, where everything is minutes old at rollout time. Record the untracked set in `setup`, before the agent runs, and hand it to `capture_patch` as `ignore`. Same host-memory pattern as `resolve_head`'s SHA, and no clock in the answer. Tasksets that don't call it keep the mtime rule. Also round the mtime cutoff backwards rather than truncating it: truncation and the round trip into the box both push it later, and a cutoff past the agent's first write drops real work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two mechanisms for one job, and the heuristic was the weaker half: it needs the image's untracked files to be old, which is true of R2E-Gym's and of no image we build ourselves. `snapshot_untracked` answers the same question from a record taken before the agent ran, with no clock in it. `capture_patch` is back to `git add -A` plus one unstage of `ignore`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
xeophon
previously approved these changes
Jul 29, 2026
hallerite
reviewed
Jul 29, 2026
hallerite
reviewed
Jul 29, 2026
hallerite
reviewed
Jul 29, 2026
hallerite
reviewed
Jul 29, 2026
hallerite
reviewed
Jul 29, 2026
# Conflicts: # docs/v1/env.md # tests/v1/test_e2e.py # uv.lock # verifiers/v1/envs/agentic_judge/env.py # verifiers/v1/tasksets/harbor/taskset.py
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit fef21cd. Configure here.
hallerite
approved these changes
Jul 31, 2026
mikasenghaas
added a commit
that referenced
this pull request
Jul 31, 2026
Migrates #2144's new records (Artifact, CollectHook) off the removed StrictBaseModel. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hallerite
pushed a commit
that referenced
this pull request
Jul 31, 2026
`RolloutRun` has always handed `task.score` the agent's own runtime, so a taskset's grader runs where the agent had root. That is reachable: an agent-planted `/testbed/conftest.py` with a `pytest_runtest_makereport` hook forcing `passed` scores 1.0 with no fix applied (verified on r2e-gym). Hiding the grading script only moves the target — pytest's plugin surface alone offers conftest, sitecustomize, ini files, and the venv's own entry points. `Task.scoring_runtime` returns a context manager to score inside, or None to keep today's behaviour. The task owns whatever has to survive the move, via collect/restore while its own box is still up. Three deliberate choices in the rollout: - Resolved in `open()`, before the box boots, so a task that scores elsewhere but borrowed its runtime or has shared tool servers fails before anything is provisioned rather than after a full agent run. - Harness and task scoring serialize instead of gathering. The harness's metrics describe the agent's session, so they are read off the agent's box before the task's context manager can tear it down. - Provisioning happens inside the scoring deadline. A grading box that can't be reached in time raises a scoring timeout; it must never become reward 0, which would look like a failed attempt. Harness cleanup is skipped when the runtime is already stopped, since reaching a second box means the first one is gone. #2144 argued the env should own sandbox topology and that this hook should not exist. That holds for the agentic judge, which composes seats it already owns. It does not extend to a taskset's own programmatic grader: `task.score` runs inside the rollout, where no env can reach it. No caller yet — Harbor is the first, in the next commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hallerite
pushed a commit
that referenced
this pull request
Jul 31, 2026
Harbor tasks can declare that their verifier runs in its own container (`[verifier].environment_mode`, `[verifier.environment]`). We ignored the first and rejected the second, so a task that asked for isolation was graded in the box the agent had just had root in. Harbor resolves the verifier's environment as `[verifier.environment]` if declared, else a deep copy of `[environment]`. That gives three cases, and only the third is a problem: - mode-only separate → the task's own image, so a second box is all it takes; - a declared environment with a `docker_image` → pull that instead; - a declared environment *without* one → Harbor builds the verifier image from `tests/Dockerfile`. Verifiers pulls and never builds, so this is rejected with a message saying to build and push the image and name the ref. `ignore_dockerfile` remains the one escape hatch, and now warns, because falling back to the agent's image runs the verifier somewhere the task never declared. Only declared artifacts cross over, which is Harbor's own model (`Trial._run_separate_verifier` uploads artifacts and nothing else) — so #2144's `collect`/`restore` and its 32 MB budget carry this unchanged. Artifacts are restored before `tests/` is staged, and `/tests` is wiped first: an artifact entry pointing into `/tests` would otherwise hand the agent the grader's own scripts, and a fresh container of the task image can ship a stale `/tests` of its own. `[[verifier.collect]]` keeps working here. The hooks run in `finalize`, in the agent's box, producing exactly the files that then travel — the two compose, so unlike #2067 there is no need to reject them. Two smaller changes: - The verifier's declared network mode is enforced rather than warned about, via the `allow`/`block` policy the runtimes already have. `allow_internet = false` means the grader really has no network, which is the point of declaring it. - A failed `tests/` staging now raises instead of leaving `test.sh` missing and scoring 0. A false zero reads as a failed attempt. `--taskset.ignore-separate-verifier` forces shared grading when a sandbox per task is too expensive. Design and prior art: xeophon in #2067, rewritten against #2144. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 31, 2026
hallerite
pushed a commit
that referenced
this pull request
Jul 31, 2026
Harbor tasks can declare that their verifier runs in its own container (`[verifier].environment_mode`, `[verifier.environment]`). We ignored the first and rejected the second, so a task that asked for isolation was graded in the box the agent had just had root in. Harbor resolves the verifier's environment as `[verifier.environment]` if declared, else a deep copy of `[environment]`. That gives three cases, and only the third is a problem: - mode-only separate → the task's own image, so a second box is all it takes; - a declared environment with a `docker_image` → pull that instead; - a declared environment *without* one → Harbor builds the verifier image from `tests/Dockerfile`. Verifiers pulls and never builds, so this is rejected with a message saying to build and push the image and name the ref. `ignore_dockerfile` remains the one escape hatch, and now warns, because falling back to the agent's image runs the verifier somewhere the task never declared. Only declared artifacts cross over, which is Harbor's own model (`Trial._run_separate_verifier` uploads artifacts and nothing else) — so #2144's `collect`/`restore` and its 32 MB budget carry this unchanged. Artifacts are restored before `tests/` is staged, and `/tests` is wiped first: an artifact entry pointing into `/tests` would otherwise hand the agent the grader's own scripts, and a fresh container of the task image can ship a stale `/tests` of its own. `[[verifier.collect]]` keeps working here. The hooks run in `finalize`, in the agent's box, producing exactly the files that then travel — the two compose, so unlike #2067 there is no need to reject them. Two smaller changes: - The verifier's declared network mode is enforced rather than warned about, via the `allow`/`block` policy the runtimes already have. `allow_internet = false` means the grader really has no network, which is the point of declaring it. - A failed `tests/` staging now raises instead of leaving `test.sh` missing and scoring 0. A false zero reads as a failed attempt. `--taskset.ignore-separate-verifier` forces shared grading when a sandbox per task is too expensive. Design and prior art: xeophon in #2067, rewritten against #2144. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grades in a second sandbox instead of the one the agent worked in, so a policy under RL pressure can't reach the grading machinery. Only what the task declares crosses over.
Supersedes #2067 and unblocks the judge-env port @mikasenghaas asked for there, now that multi-agent has landed (#1939).
Opt-in.
agentic-judgestill defaults toshared. Isolation is worth having only once a taskset publishes its evidence somewhere that travels — see Adoption below.Usage
A SWE task produces its delta in
finalizeand declares nothing — anything written to/logs/artifacts/is collected by convention:Anything outside the convention dir is declared on the task row:
A Harbor
task.tomlis read as-is:Run it:
An env composes the sandboxes itself with the two primitives:
Design
Trace.infois the durable record (patch,verdict). It is not a transport channel — though an agentic judge does receive the whole serialized trace,infoincluded, at/tmp/trace.json./logs/artifacts/is transport: box → host → box, discarded after restore.Task.finalizeis the producer hook — it already means "runtime live, agent done, before scoring mutates anything", which is Harbor's collect-hook moment, so[[verifier.collect]]maps onto it rather than needing a new stage.collect()returning is the barrier; the env owns sandbox topology.The grading sandbox boots from the task's own image, so only the delta travels. No
Task.scoring_runtimehook.runtimes/,rollout.py,agent.pyandenv.pyare untouched.Harbor
artifacts = [...](string and object form, relative sources resolved against the runtime workdir,excludehonored) and[[verifier.collect]]. Two deliberate divergences fromharbor run:destinationis inert; it places files in Harbor's host trial directory, which verifiers has no equivalent forRejected at load: sidecar
service,[verifier].user, explicit[verifier.environment]image.agentic-judge
--env.topology shared|isolated, defaulting toshared— what #2109 measured. Underisolatedthe judge gets its own sandbox and the workspace note changes with it: a judge told it stands in the agent's workspace when it stands in a fresh one reads an unmodified tree as failure.An unpinned judge runtime inherits the solver's, matching how
harness/model/samplingalready resolve for unpinned seats.capture_patchnow distinguishes its two failure modes: the sandbox answering and git refusing (agent's own environment — recordspatch_error, still scores) from the sandbox not answering (raises, since a zero there says only that our infrastructure failed). Telling them apart needs a liveness probe, becauseDockerRuntime.runreturnsdocker exec's non-zero result rather than raising.Adoption
Nothing in
research-environmentspublishes to the sandbox yet —write_pathis new here. Under isolation today:capture_patchtasksets work, because their judge hints point atinfo.patchin the trace record, which travelsgit status,git diff,git login/testbed", which under isolation is a pristine checkoutSo adopt per taskset, once its evidence travels.
Verification
ruff,ty check verifiers, 909 non-e2e tests. Against real docker sandboxes, via scratch scripts (not committed, perAGENTS.md):collect/restore— 11 cases: convention sweep, declared paths,exclude, symlink clobber, strict missing-source, relative-source resolution, over-cap, subprocess refusalrun()composition — 8 cases with the two agent runs stubbed and everything else real: two distinct sandboxes, solver's torn down before the judge's is provisioned, artifact restored at its original path, trace record uploaded, judge sandbox built from the solver's imagecapture_patchattribution — healthy capture, broken.gitrecords and continues, dead sandbox raisesNo eval has been run. The isolated path has never executed with a real model or on Prime. On Prime I'd expect judge-sandbox tunnel reachability to break first: it is remote, and nothing has previously provisioned a second remote sandbox needing the interception server within one episode.
Notes
tarhas no-r, so collection builds one archive per source.restore()refuses the subprocess runtime — extraction writes to absolute paths, which there is the host filesystem.tarin its own sandbox can place arbitrary files in the grading sandbox; our owntar -ccannot produce an escaping archive, since it does not recurse into symlinked directories.MAX_ARTIFACT_BYTESis 32 MB on the same-image/delta-only assumption. feat(v1): support Harbor's separate verifier environments #2067 budgeted 256 MB for full-tree transfer.artifacts.pyis 166 lines.🤖 Generated with Claude Code
Note
Add isolated grading topology to
AgenticJudgeEnvwith artifact transporttopologyfield ('shared'|'isolated') toAgenticJudgeEnvConfig; in'isolated'mode the solver runs in one container, artifacts are collected, and a fresh judge container is provisioned to grade using only those artifacts.collectandrestorefunctions that tar declared artifact sources (plus a convention dir sweep) out of the solver box and extract them into the judge box, enforcing a per-rollout size budget.capture_patchin git.py to write the patch to an in-sandbox file and unstage specified paths before diffing; addssnapshot_untrackedhelper.[[verifier.artifact]]and[[verifier.collect]]entries fromtask.tomlintoTaskData.artifactsand run collect hooks duringfinalize, failing the rollout on hook error.MAX_ARTIFACT_BYTESfails the entire rollout;restoreraisesArtifactErrorif called on a subprocess runtime.Changes since #2144 opened
topologyconfiguration withshare_runtimeboolean to control runtime isolation in agentic judge environment [9ac6daf]ArtifactErrorexception class and converted all artifact-related error handling to raiseRuntimeError[9ac6daf]JudgeTaskwith a unified note referencingvf.ARTIFACTS_DIRandTRACE_FILE, removing conditional variants based on sandbox type [960d45b]HarborTask.finalizemethod to execute collect hook commands with an empty environment mapping instead of the verifier-specific environment [132d4b3]AgenticJudgeEnv.__init__to preserve the judge agent's configured runtime whenshare_runtimeis disabled, removing the previous logic that forcibly overrode the judge runtime to the solver's runtime when the judge was configured as aSubprocessConfig[4228c44]JudgeTask.from_traceby introducing three module-level constants (_RECORD_NOTE,SHARED_WORKSPACE_NOTE,ISOLATED_WORKSPACE_NOTE) and selecting betweenSHARED_WORKSPACE_NOTEandISOLATED_WORKSPACE_NOTEbased on theshare_runtimeargument during prompt assembly [4228c44]TaskData.artifactsfield docstring and reworded comments in the restore function to emphasize that artifacts represent paths collected from one runtime and restored in another, with explicit refusal to restore into subprocess runtime [4228c44]Traceschema to version 1 with agent-scoped fields, auto-populated metadata, and renamed error/run recording APIs [fef21cd]EnvServerConfigwith separateenv,serve, andlegacyconfiguration blocks inEvalConfigandServeConfig[fef21cd]HarnessErrorinstead of stopping with timeout condition [fef21cd]EnvConfig.max_concurrenttomax_concurrent_agentswith default 1 and moved episode agent concurrency gating intoEnvimplementation [fef21cd]Taskmodel with structured environment fields and MB-unit resource extraction [fef21cd]JudgeConfigto accept only file-based prompts viapromptfield, removing inline string andprompt_fileoptions [fef21cd]TasksetConfig.system_promptfield andTask.with_system_promptmethod to enable taskset-level prompt overrides with GEPA best prompt export [fef21cd]is_installedfunction to useimportlib.metadata.versioninstead of subprocess-based package detection [fef21cd]verifiers.v1.configs.cli.envtoverifiers.v1.configs.serveand introducednarrow_taskset_confighelper for CLI resolution [fef21cd]WikiSearchJudgeclass with environment-specific prompt file and config for wiki-search taskset [fef21cd]PrimeRuntime.runwith single call toAsyncSandboxClient.run_background_joband preserved additionalserver_kwargsinserve_envconversion [fef21cd]validate_pairingcall inAgenticJudgeEnvinitialization to enforce judge, task, and runtime compatibility [fef21cd]renderersto >=0.1.9.dev9,harborto 0.20.0, and default Harbor version inTerminus2HarnessConfigto '0.20.0' [fef21cd]JudgeTaskConfigto acceptpromptandhintfields as string,Path, or explicitTextFileobjects, and modifiedJudgeTaskConfig._resolvemethod to treat bare strings as inline text rather than using file extension heuristics to determine whether to read from disk [96f2b59]TextFileto the public exports of theverifiers.v1.envs.agentic_judgemodule [96f2b59]task.promptas an object with apathfield instead of a string path, and added atask.hintstring field [96f2b59]Macroscope summarized 615efbb.
Note
High Risk
Changes grading topology, cross-runtime file transport, Harbor finalize strictness, and patch-capture failure attribution—areas that directly affect scores and rollout success.
Overview
Adds
vf.collect/vf.restoreandTaskData.artifactsso declared paths (plus optional/logs/artifacts/) can leave the agent sandbox and be replayed at the same absolute paths in another runtime, with a per-rollout size cap and strict errors for missing declared sources.agentic-judgegainsshare_runtime(default true). When false, the solver runs alone, artifacts ontrace.statefeed a fresh judge container viaJudgeTask.setup, and the judge prompt switches to an isolated-workspace note instead of “same box as the agent.” Judgeprompt/hintconfig can be inline strings, paths, orTextFile.Harbor loads
artifactsand[[verifier.collect]]fromtask.toml, runs collect hooks inHarborTask.finalize(failures fail the rollout), then stores collected bytes on trace state. Unsupported verifier sidecar settings are rejected at parse time.capture_patchcanignorepre-agent untracked paths (snapshot_untracked), optionallywrite_pathinto the artifact convention dir, and treats dead sandbox vs git failure differently (SandboxErrorvspatch_error).Reviewed by Cursor Bugbot for commit 858ac72. Bugbot is set up for automated code reviews on this repo. Configure here.