feat(v1): Harbor separate verifier environments

[rasdani]

#2144 is merged. This PR is rebased onto current main and now contains only the Harbor separate-verifier stack.

Wires up Harbor's [verifier].environment_mode / [verifier.environment], which we currently ignore and hard-reject respectively — so a task that asks to be graded away from the agent is graded inside the box the agent just had root in.

Design and prior art are @xeophon's, from #2067. This is that work rewritten against #2144 rather than merged with it; see Why not merge #2067 at the end.

Why this is a correctness gap, not a nicety

Verified on r2e-gym while testing something else: an agent-planted /testbed/conftest.py with a pytest_runtest_makereport hook forcing passed scores reward 1.0 with no fix applied. It never touches the grading script or the test files, so hiding those does nothing. Pytest's plugin surface alone offers conftest, sitecustomize, ini files and the venv's own entry points — any grader running in the agent's box is reachable, and patching individual paths is whack-a-mole.

What lands

Task.scoring_runtime (task.py, rollout.py) — returns a context manager to score inside, or None for today's behaviour. RolloutRun resolves it in open(), before the box boots, so a task that scores elsewhere but borrowed its runtime or has shared tool servers fails immediately rather than after a full agent run. Harness and task scoring stop being gathered: the harness's metrics describe the agent's session, so they're read off the agent's box before the task's context manager can tear it down.

Runtime primitives — provision_runtime (the start/stop context manager Agent.provision already had inline), stop_confirmed/teardown_confirmed, and read_bounded. Confirmed teardown is the point of the second one: teardown is best-effort because a leaked container is a billing problem, but a caller standing up a grading box needs the first box gone — an agent can leave a background process running past its final turn.

Harbor wiring (tasksets/harbor/taskset.py) — mode resolution via Harbor's own resolve_task_verifier_mode / resolve_effective_verifier_env_config, a VerifierConfig on HarborData, and verifier_runtime_config deriving the grading box from the agent's.

reward.json — Harbor's separate verifiers write it, not reward.txt. Without this every separate-mode task would score 0, which reads as a failed attempt.

Three cases, and why one is rejected

Harbor resolves the verifier environment as [verifier.environment] if declared, else a deep copy of [environment]:

task.toml	Harbor's verifier image	here
environment_mode = "separate", no [verifier.environment]	the task's own image	supported
[verifier.environment] with docker_image	that image	supported
[verifier.environment] without docker_image	built from tests/Dockerfile	rejected at load

The third is rejected because verifiers pulls images and never builds them — the same rule [environment] already follows. The error says to build and push the image and name the ref. ignore_dockerfile remains the one escape hatch and now warns, because falling back to the agent's image runs the verifier somewhere the task never declared.

Only declared artifacts and the /logs/artifacts/ convention directory cross over. That's Harbor's own model — Trial._run_separate_verifier uploads artifacts and nothing else, with no full-tree copy anywhere — so #2144's collect/restore and its 32 MB budget carry this unchanged, and #2067's parallel artifact layer and 256 MB budget are not ported.

Smaller decisions worth a look

• [[verifier.collect]] keeps working in separate mode. feat(v1): support Harbor's separate verifier environments #2067 rejects it. It composes fine: the hooks already run in finalize, in the agent's box, producing exactly the files that then travel.
• The verifier's network mode is enforced, not warned about, using the allow/block policy the runtimes already have. allow_internet = false means the grader really has no network — verified below. feat(v1): support Harbor's separate verifier environments #2067 warns instead; I think a declaration that silently doesn't hold is worse than one that occasionally bites.
• Artifacts are restored before tests/ is staged, and /tests is wiped first. An artifact entry pointing into /tests would otherwise hand the agent the grader's own scripts, and a fresh container of the task image can ship a stale /tests.
• A failed tests/ staging now raises instead of leaving test.sh missing and scoring 0.
• A reward-less reward.json object sums. {"correctness": 1.0, "style": 0.25} records two rewards and totals 1.25 — v1's existing keyed-reward convention, and feat(v1): support Harbor's separate verifier environments #2067's behaviour, but it can exceed 1.0. Flagging rather than changing.
• ignore_separate_verifier forces shared grading when a sandbox per task is too expensive. Not a Harbor knob; same family as ignore_timeouts / ignore_dockerfile.

Two semantics reviewers should check

• Provisioning now runs inside _scoring_timeout, which Harbor sets from verifier.timeout_sec (default 600s). A grading box that can't be reached in time raises a scoring timeout; it must never become reward 0. Verified below with a bad image ref.
• This reverses feat(v1): grade in an isolated box, with Harbor-native artifacts #2144's stated position that the env owns sandbox topology and that this hook shouldn't exist. That holds for the agentic judge, which composes seats it already owns. It doesn't extend to a taskset's own programmatic grader: task.score runs inside the rollout, where no env can reach it. The conftest result above is the argument.

Verification

No public dataset exercises this feature. 104,131 cached task.toml files declare zero environment_mode or [verifier.environment]; neither Harbor Hub (80 datasets) nor laude-institute/harbor-datasets (103) ships one. So: local fixtures for each case, driven through the real finalize → scoring_runtime → score path against live Docker containers.

PASS  shared grades in the agent's box
PASS  shared still sees the agent's leftovers
PASS  fresh_copy gets its own box
PASS  declared_image gets its own box
PASS  agent box gone before grading (fresh_copy)
PASS  agent box gone before grading (declared)
PASS  declared artifact travelled (fresh_copy)
PASS  declared artifact travelled (declared)
PASS  undeclared agent file did NOT travel (fresh_copy)
PASS  undeclared agent file did NOT travel (declared)
PASS  stale /tests wiped in the verifier box
PASS  declared verifier image actually used          # python 3.12 vs the task's 3.11
PASS  fresh copy keeps the task image
PASS  allow_internet=false enforced                  # NET=DOWN in the verifier box
PASS  public verifier keeps network
PASS  reward.json read (shared / fresh_copy / declared)
PASS  wrong answer scores 0

Case C rejected at load with the build-and-push message; loads under --taskset.ignore-dockerfile with a warning; --taskset.ignore-separate-verifier returns verifier=None. A bad verifier image ref raises SandboxError rather than scoring 0. _reward_json checked against 14 payload shapes (scalar, map, map-with-reward, empty, string values, bools, list, malformed, missing).

910 local tests pass (63 credential-gated tests skipped); ruff check, ruff format --check, ty check verifiers, and pre-commit are all green. Scratch scripts and fixtures live in /tmp and ~/.cache/harbor, not committed (AGENTS.md).

Not verified: Prime. Everything above is Docker. PrimeRuntime.teardown_confirmed is written but unexercised, and Prime provisioning has been timing out independently — which is exactly the path where the scoring-timeout interaction matters most.

Why not merge #2067

A real test merge of the two branches conflicts in 3 files / 9 hunks, with seven semantic collisions underneath: two Artifact classes, TaskData.artifacts vs HarborData.artifacts, two artifact transports, opposite strictness on the convention dir, and contradictory handling of [[verifier.collect]] and [verifier.environment]. #2067 is also 7 commits behind main. Resolving it would have meant deleting most of its diff, so the surviving design was rebuilt on #2144's transport instead.

Open question for @xeophon

DeepSWE v1.1 isn't on Harbor Hub, in the public dataset repo, or in the local cache, so I couldn't read one of its task.tomls. From your two review comments on #2067 — [verifier.environment] present, allow_internet = false, no docker_image, Harbor building tests/Dockerfile — it looks like case C, the one this PR rejects, meaning DeepSWE would need its verifier image built and pushed first. One real task.toml would settle it.

🤖 Generated with Claude Code

---

Note

High Risk
Changes rollout scoring order and sandbox lifecycle (confirmed teardown can fail the rollout; verifier provisioning counts against scoring timeout) on security-critical grading paths; Prime confirmed teardown is implemented but not exercised in the PR’s verification.

Overview
Adds isolated Harbor grading when task.toml sets [verifier].environment_mode = "separate": the agent box is torn down with confirmed removal, declared artifacts and tests/ are staged into a fresh verifier runtime, and scoring runs there instead of in the compromised agent environment.

Framework: Task.scoring_runtime() lets tasks supply an async context manager for the grading box; RolloutRun resolves it at open() (fails fast if the rollout borrows the runtime or uses shared MCP tools), runs harness scoring on the agent box then task scoring in the separate box serially under the scoring timeout, and skips harness cleanup when the agent runtime is already stopped. Shared provision_runtime, stop_confirmed / teardown_confirmed (Docker + Prime), and read_bounded support the handoff.

Harbor: Parses separate verifier mode via Harbor’s resolvers into VerifierConfig on HarborData; ignore_separate_verifier opts out. Rejects verifier environments that would require building tests/Dockerfile (same pull-only image rule as agent env). Reads reward.json (bounded) with fallback to reward.txt. Docs cover separate verifiers, stricter collect-hook failures, and remaining gaps.

^{Reviewed by Cursor Bugbot for commit 5fe32f4. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add separate verifier environments for Harbor task grading

• Harbor tasks can now declare a separate verifier environment; grading runs in a clean, freshly provisioned runtime after the agent box is torn down, with collected artifacts and test files staged into the new box.
• Introduces VerifierConfig and parse_verifier_environment in taskset.py to parse verifier image, resources, network, and timeout from task config; the ignore_separate_verifier flag on HarborConfig forces grading back into the agent runtime.
• Adds provision_runtime async context manager in runtimes/init.py to guarantee start/stop lifecycle, and stop_confirmed/teardown_confirmed to verify container or sandbox removal before grading begins.
• Task.scoring_runtime in task.py provides an extension point for tasks to supply a separate scoring runtime; rollout.py enforces that separate scoring requires an owned runtime with no shared tool servers.
• Risk: separate scoring serializes harness scoring and task scoring (previously concurrent), and confirmed teardown raises SandboxError if the agent container is still present, which will fail the rollout.

Changes since #2152 opened

• Refactored network policy derivation in verifier_runtime_config function to use with_task_network_policy method [0dd56a8]
• Changed network policy resolution in parse_verifier_environment function to use explicit phase policy or environment baseline [0dd56a8]
• Reworked PrimeRuntime.teardown_confirmed method to handle already-deleted sandboxes, poll for termination confirmation, and use exponential backoff [5fe32f4]
• Added _is_not_found utility function to detect HTTP 404 status codes in exception chains [5fe32f4]

^{Macroscope summarized 7377697.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a substantial new feature (separate verifier environments) with new runtime lifecycle methods, changes to the core scoring flow, and significant new configuration parsing logic. Additionally, there is an unresolved high-severity bug regarding Prime verifier config validation that needs attention.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5fe32f4. Configure here.}

[cursor]

Prime verifier omits required VM

High Severity

verifier_runtime_config can build a Prime verifier with a restricted policy (allow_internet = false / allowlist) while leaving vm=False from an unrestricted agent config. PrimeConfig then rejects that combination during model_validate, so the rollout fails in open() before the agent runs—even though Docker handles the same task fine.

Additional Locations (1)

• verifiers/v1/runtimes/prime.py#L84-L91

 

^{Reviewed by Cursor Bugbot for commit 5fe32f4. Configure here.}

clanker approved without my consent