[WIP] add SWE-bench Lite accuracy eval / 添加 SWE-bench Lite 准确率评估#1947
Open
adibarra wants to merge 29 commits into
Open
[WIP] add SWE-bench Lite accuracy eval / 添加 SWE-bench Lite 准确率评估#1947adibarra wants to merge 29 commits into
adibarra wants to merge 29 commits into
Conversation
Contributor
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. 感谢你的贡献！对于 vLLM 与 SGLang，请确保你的 recipe 与官方 vLLM recipes 和/或 SGLang cookbook 保持一致 如果不一致，请先创建一个 PR，之后我们才能将你的单节点 PR 合并到 master 分支。让我们确保文档保持一流水准，使整个 ML 社区都能从你的辛勤工作中受益！谢谢 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。如果选择重新运行失败的任务，PR 作者有责任确保其最终通过。参见 GitHub 关于重新运行失败任务的文档：https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow 一般而言，PR 作者应先向相应公司的 CODEOWNERS 请求审阅并获得 PR 批准，然后再请求核心维护者审阅。 如需更多帮助，PR 作者可通过 Slack 联系核心维护者。 |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
…arness scoring, Modal-capable) Add a SWE-bench Lite accuracy eval that generates patches via the lm-eval harness and scores them with the official swebench evaluation harness. - utils/evals/swebench_lite.yaml: lm-eval task config for SWE-bench Lite generation (prompt/doc-to-text, generation kwargs, dataset wiring). - utils/evals/swebench_score.py: post-processing + scoring. Extracts model patches from lm-eval output, feeds them to the swebench harness, and emits a "resolved" rate. Supports running the harness locally or on Modal via SWEBENCH_USE_MODAL (Modal pass-through so scoring can run off-box). - utils/collect_eval_results.py: extract_lm_metrics learns a "resolved" filter branch so the swebench resolved metric is collected alongside the existing lm-eval metrics. - utils/evals/thresholds.json: add the swebench_lite threshold entry. - utils/evals/EVALS.md: document the SWE-bench Lite eval and how scoring works. - benchmarks/benchmark_lib.sh: add run_swebench_eval, _install_swebench_deps, maybe_run_eval, and Modal pass-through. run_eval now picks a per-scenario default framework (agentic-coding -> swebench, fixed-seq-len -> lm-eval); an explicit EVAL_FRAMEWORK env var or --framework arg overrides the default. EVAL_TASKS_DIR selects the task yaml. - utils/evals/test_swebench_eval.py, utils/evals/test_run_eval_dispatch.py: tests for the scorer and the scenario/framework dispatch precedence.
…gentic configs) Wire the SWE-bench Lite eval into the sweep matrix so it runs on agentic coding configs, and route it through e2e-tests. - utils/matrix_logic/generate_sweep_configs.py: add mark_eval_entries and mark_all_eval_entries. For agentic configs these mark exactly one eval entry per (model, runner, framework, precision) group at the highest concurrency, single-node only, so each unique agentic config gets one swebench eval run rather than one per concurrency point. - utils/matrix_logic/test_generate_sweep_configs.py: add test_marks_agentic_entry_for_swebench and update TestMarkAllEvalEntries to cover the agentic marking behavior. - .github/workflows/e2e-tests.yml: add the agentic-eval-config bucket, a test-sweep-agentic-evals job, and make collect-evals depend on it. The AGENTIC_EVAL filter (agentic + no prefill + run-eval) selects the eval entries; the throughput AGENTIC filter (agentic + not run-eval) excludes them so throughput and eval runs don't collide. - benchmarks/single_node/agentic/kimik2.5_fp4_b300.sh: add the eval hook so the recipe triggers the agentic swebench eval.
…1.0) + bootstrap Modal creds from env swebench 4.1.0 exposes --max_workers in both Docker and Modal modes; --parallelism does not exist. Fix run_harness() to emit --max_workers in the Modal branch. Add _ensure_modal_credentials() to benchmark_lib.sh: swebench's credential check only looks for ~/.modal.toml, but CI supplies MODAL_TOKEN_ID/ MODAL_TOKEN_SECRET env vars (GitHub secret). The helper bootstraps the file from the env vars when the file is absent, so the harness check passes. Called in run_swebench_eval() right after _install_swebench_deps, scoring path only. Update the Modal test name and assertions, the run_swebench_eval docstring, and the EVALS.md knobs bullet to document the credential bootstrapping.
Apply the EVAL_ONLY=true if/else gating pattern (already present in kimik2.5_fp4_b300.sh) to the remaining 24 single-node agentic recipes in benchmarks/single_node/agentic/. In eval-only mode each recipe skips the multi-turn agentic replay and calls maybe_run_eval "$PORT" against the live server; run_eval auto-selects swebench for the agentic-coding scenario. The deprecated/ subdirectory was not touched.
adibarra
force-pushed
the
feat/swebench-lite-eval
branch
from
ce358e4 to
e34035d
Compare
…job env GitHub secrets MODAL_TOKEN_ID/MODAL_TOKEN_SECRET are now available; bootstrap into ~/.modal.toml happens in benchmark_lib.sh:_ensure_modal_credentials. SWEBENCH_USE_MODAL is only read by swebench-path functions, so it is inert for lm-eval/gsm8k jobs.
functionstackx
changed the title
# Conflicts: # benchmarks/single_node/agentic/dsr1_fp4_b200.sh # benchmarks/single_node/agentic/dsr1_fp4_mi355x.sh # benchmarks/single_node/agentic/glm5.1_fp4_mi355x.sh # benchmarks/single_node/agentic/glm5_fp8_b200.sh # benchmarks/single_node/agentic/gptoss_fp4_b200.sh # benchmarks/single_node/agentic/gptoss_fp4_h100.sh # benchmarks/single_node/agentic/gptoss_fp4_h200.sh # benchmarks/single_node/agentic/gptoss_fp4_mi300x.sh # benchmarks/single_node/agentic/gptoss_fp4_mi325x.sh
- Re-sync test-sweep-agentic-evals inputs with main's test-sweep-agentic: offloading -> kv-offloading + kv-offload-backend + total-cpu-dram-gb. - Add EVAL_ONLY/maybe_run_eval tail gating to the agentic recipes AgentX v1.0 added (dsv4_fp4_b200_sglang, dsv4_fp4_b300_sglang, minimaxm3_fp8_h100/ h200/mi300x/mi325x) so eval-only runs skip the replay like the others. - test_run_eval_dispatch: set KV_OFFLOADING=none so the new source-time agentic guard in benchmark_lib.sh is satisfied (dispatch logic unaffected).
…dling Add EVAL_LIMIT env var to run_lm_eval() so --limit N is appended to the lm_eval invocation when set, enabling small smoke runs (e.g. 10 instances) without touching the full ~300-instance swebench suite. Wire the knob through benchmark-tmpl.yml (new eval-limit input + EVAL_LIMIT env) and e2e-tests.yml (both workflow_dispatch and workflow_call inputs; passed through to test-sweep-evals and test-sweep-agentic-evals with: blocks). Document the variable in utils/evals/EVALS.md. Harden _ensure_modal_credentials against b300 slurm/pyxis containers where --export=ALL propagates the HOST's HOME into the container; if HOME is unset, mkdir -p fails, or the directory isn't writable, remap HOME to /tmp/inferencex-modal-home before writing ~/.modal.toml. Remap is scoped to the write path (SWEBENCH_USE_MODAL=true, file absent, tokens present). Tests: functional shim tests for --limit presence/absence; HOME-remap tests covering writable home (no remap), read-only parent (remap + 600 perms), and non-writable existing dir (remap); and a no-op test when SWEBENCH_USE_MODAL=false.
…mpty SWEBENCH_NAMESPACE arg
- Add `include_agentic: bool = False` to `mark_eval_entries`; wrap the
`ag_sn_groups` agentic-marking block in `if include_agentic:` so that
default sweeps no longer set `run-eval: true` on any agentic entry.
The e2e-tests.yml AGENTIC filter (`not x.get('run-eval', False)`) then
routes all agentic entries to the throughput job, restoring main parity.
- Pass `include_agentic=args.evals_only or args.all_evals` in `main()` so
--evals-only and --all-evals continue to mark and select agentic entries.
- Replace `${SWEBENCH_NAMESPACE+--namespace "$SWEBENCH_NAMESPACE"}` with an
`ns_args` array in `run_swebench_eval`; when `SWEBENCH_NAMESPACE=""` the
old form word-split to a bare `--namespace` (argparse error); the array
form safely expands `--namespace ""` or nothing when unset.
- Tests: `test_marks_agentic_entry_for_swebench` updated to pass
`include_agentic=True`; new `test_default_mode_does_not_mark_agentic`
asserts zero agentic entries marked in default mode; new ns_args unit
tests cover unset/empty/value cases plus a static assertion that the old
pattern is gone from benchmark_lib.sh.
… KeyErrors on unregistered task-name paths) The pinned lm-eval (0.4.9.2, ref b315ef3) crashes with KeyError: '<task_name>' in pretty_print_task (tasks/__init__.py:681) when --tasks is given a file path to an external YAML whose task: name is not in lm-eval's bundled registry. gsm8k/gpqa_diamond are immune because those names exist in the bundled registry; swebench_lite is not. Fix: in run_lm_eval(), add optional EVAL_INCLUDE_PATH support — when set, injects --include_path "$EVAL_INCLUDE_PATH" just before --tasks; inert when unset (gsm8k/gpqa production invocations are byte-identical). In run_swebench_eval(), switch the generation call from EVAL_TASKS_DIR="$yaml_path" (path form → KeyError) to EVAL_TASKS_DIR="$task_name" (name form) EVAL_INCLUDE_PATH="$(dirname "$yaml_path")" (registers the dir) with save/restore of both vars so EVAL_INCLUDE_PATH does not leak to subsequent lm-eval invocations. The dataset_path-from-YAML derivation (awk over yaml_path) is unchanged — generation and scoring remain in lockstep. Tests: two shim-based dynamic tests (EVAL_INCLUDE_PATH set/unset → flag present/absent in argv; --tasks carries name vs. yaml path) and one static assertion that run_swebench_eval source contains EVAL_INCLUDE_PATH wiring.
…iling newline fail validation)
Live probe proved it: MODAL_TOKEN_SECRET secret has a trailing whitespace char;
raw auth fails ('Token validation failed'), whitespace-stripped auth succeeds.
Strip whitespace/quotes and re-export in _ensure_modal_credentials so both the
modal client (env) and the bootstrapped ~/.modal.toml are clean.
…lure
- run_swebench_eval: wrap scoring in timeout ${SWEBENCH_SCORE_TIMEOUT:-7200}s.
The overnight 300-instance run stalled ~7h in Modal image builds and held the
b300 allocation until the slurm wall; a stalled backend now fails fast.
- maybe_run_eval: always stage eval artifacts (append_lm_eval_summary) even when
the eval fails, then propagate the rc — samples/predictions survive for
diagnosis instead of dying in the job sandbox.
…ation input) Agent harnesses (SWE-agent / mini-swe-agent) emit standard predictions.jsonl directly; this bypasses lm-eval samples parsing and feeds the existing Modal scoring + results pipeline unchanged. Groundwork for agentic swebench.
…sandboxes SWEBENCH_GEN_MODE=agentic runs a real agent loop per instance instead of the single-shot prompt: mini-swe-agent (2.4.5) drives the local OpenAI-compatible endpoint; each instance's shell executes in a Modal sandbox (swe-rex[modal], official swebench per-instance images -- no docker needed on the GPU node). preds.json feeds the existing Modal scoring via --predictions-file (which now also accepts the dict-keyed preds.json format directly). - benchmark_lib.sh: _run_swebench_agentic_generation (config overlay, slice via EVAL_LIMIT, workers/step/timeout knobs), _install_swebench_agent_deps (mini-swe-agent==2.4.5 + swe-rex[modal]==1.4.0), gen-mode branch in run_swebench_eval feeding scoring via score_input array. - swebench_score.py: --predictions-file accepts dict preds.json or JSONL. - workflows: swebench-gen-mode input threaded e2e-tests -> benchmark-tmpl env. - tests: shim-driven agentic-generation test + predictions-file format tests. Single-shot remains the default; agentic is the real SWE-bench setting.
Fresh installs print a multi-line version banner on import; take only the last stdout line and validate it is a file. Shim test now emulates the banner.
mini's default startup_timeout=60s is consumed by the cold GB-scale swebench
image pull alone ('Runtime did not start within 0s'). Default 900s via
SWEBENCH_AGENT_STARTUP_TIMEOUT; command timeout 300s (mini default 60s is too
tight for running repo test suites) via SWEBENCH_AGENT_CMD_TIMEOUT.
Trajectories are the primary forensic artifact for agent tuning; they previously died with the job's temp dir. Copy *.traj* flat into the eval output (append_lm_eval_summary flattens *.json* into the workspace root), upload via new globs, and clean up post-upload.
Findings from 10-trajectory deep-dive (first-10 Lite, DSv4): - 3/5 unresolved agents submitted without ever running the failing test - 1 agent had the CORRECT fix on disk at step 31, burned 44 steps fighting an unfixable sandbox C-extension build, and hit the step cap without submitting - CoT leaks into visible content (deepseek_v4 reasoning parser init failure, recipe-side follow-up) -- 'execute over prose' guidance mitigates Replace the static config heredoc with a runtime merger that appends targeted guidance to mini's instance_template: verify-before-submit, build-failure escape hatch, submission discipline, step-budget framing. Single merged config replaces the dual -c chain.
Every agent sandbox was billing a full hour for ~7-minute instances (observed: batches dying at 59m59s on the Modal dashboard). Three leaks: - mini-swe-agent 2.4.5 process_instance() never calls env.stop(), even on success, so every sandbox lives until runtime_timeout (3600s default). - swe-rex 1.4.0 ModalDeployment.stop() has its poll check inverted: it terminates only sandboxes that already exited and skips running ones. - ModalDeployment.start() leaks the sandbox when the runtime never comes alive (the startup-timeout failure mode). Fix: _patch_swebench_agent_cleanup() patches the installed files at dep install (idempotent, anchor-checked against the pinned versions) so sandboxes terminate the moment their instance finishes; a post-generation workspace sweep reaps anything that slips through (crashed workers, outer timeout kills; SWEBENCH_SANDBOX_SWEEP=0 disables for tests); and the merged config now sets runtime_timeout explicitly (SWEBENCH_AGENT_RUNTIME_TIMEOUT, default 3600) as a pure backstop. No agent-visible behavior change: cleanup happens after instance completion, so resolved-rate comparisons across runs stay clean.
… budget exhaustion Run-1/3 findings (50 instances, tuned template): - Metric bug: the harness report's total_instances is the full dataset size (300) even with EVAL_LIMIT=50, so a 32/50 (64%) run was published as 0.107 and nearly tripped the 0.10 threshold gate. parse_resolved now prefers submitted_instances over total_instances (identical for full-split runs). - 6/50 instances hit LimitsExceeded after 75 steps and submitted NOTHING, despite forensics showing fixes can be complete mid-run. patched process_instance now falls back to submitting `git diff` of the working tree when an instance ends abnormally with a live sandbox (requires rc 0 and a `diff --git` prefix so an error string can never become a patch). Empty submissions score zero, so the fallback is strictly >=. - Stage the swebench harness report as swebench_report_<task>.json and upload it; it names resolved/unresolved per instance and was previously left behind on the node.
Run-2/3 verified the sandbox-cleanup patches (applied on the node, sweep found 0 lingering sandboxes) but 0 fallback submissions fired while 6 instances still ended LimitsExceeded with empty patches. Root cause: mini's agent run loop absorbs InterruptAgentFlow (Submitted, LimitsExceeded, ...) and RETURNS normally with an empty submission -- LimitsExceeded never reaches process_instance's except branch, which is where the fallback hook lived (their trajectories carry no traceback/exception_str keys, confirming the normal-return path). Move the primary hook to just after agent.run(): any empty submission with a live sandbox now submits `git diff` of the tree (same rc-0 + "diff --git"-prefix guards). The except-path hook stays for real exceptions.
Two full-300 Modal scorings measured ~$80 each in eval sandboxes alone (vs $0.99-5.91 for image builds -- caching was never the cost driver). Root cause: swebench's run_evaluation_modal.py hardcodes cpu=4 per sandbox; Modal bills reserved cores and the test runs are predominantly single-threaded pytest. Patch the installed file at dep install (idempotent, anchor-checked, numeric-validated) to SWEBENCH_EVAL_SANDBOX_CPU (default 2). Per-instance tests run somewhat slower on fewer cores; scoring parallelism absorbs it.
…tion Run 29039988325 (full-300, workers=144): all 300 predictions were on disk by t+60min but mini-extra never exited -- a probabilistic hang-on-exit at high worker counts (the identical previous run exited cleanly). The process idled 3h into SWEBENCH_AGENT_TIMEOUT, and the rc!=0 path then deleted the complete preds.json. - Completion watchdog: run mini-extra in the background and poll preds.json (written incrementally per instance); once all expected instances are present, grant SWEBENCH_AGENT_EXIT_GRACE (300s) for a clean exit, then kill and count generation complete. Overall SWEBENCH_AGENT_TIMEOUT deadline retained. - Salvage: if generation fails with N>0 predictions written, warn and score the partial set instead of discarding real work (denominator is submitted instances, so partial runs report honestly over what ran). - Tests: hung-mini watchdog kill, partial-preds salvage, zero-preds still-fails.
…fault, workers=64 Decisions 2026-07-09 after three full-300 validation runs (162/162/163 resolved, 54.0-54.3%): - Threshold 0.10 -> 0.50: the old value predates the denominator fix and was effectively decorative; 0.50 sits ~4pts under the observed full-run floor and well under the 50-slice range (62-68%). - EVAL_LIMIT empty now defaults to the 50-instance CI slice (~45min GPU + ~$9 Modal); EVAL_LIMIT=full runs the whole split (~1.75h + ~$44) for release-grade checks. Applies to the agentic swebench path only. - SWEBENCH_AGENT_WORKERS default 8 -> 64: saturates the serving point without entering the 144-worker teardown-race territory; full-300 generation drops ~3h -> ~55min. - Known-bad b300 nodes (005: NCCL init death, 006: wedged nvidia driver) excluded via SALLOC_EXCLUDE with a tracking comment; remove as infra repairs them.
…E unset Label/changelog-triggered evals pass no swebench-gen-mode, which fell through to single-shot generation -- ~10% resolved on a healthy config, an instant false-negative against the new 0.50 gate. Agentic scenarios now default to the agent loop; explicit SWEBENCH_GEN_MODE still wins.
Unset SWEBENCH_GEN_MODE now means the agent loop unconditionally, not just for agentic scenarios -- SWE-bench without the agent loop is not a meaningful eval (~10% resolved) and the 0.50 gate is calibrated to agentic scores. single-shot remains solely as an explicit SWEBENCH_GEN_MODE=single-shot debugging escape hatch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[WIP]
lm-eval cannot score SWE-bench, so this reuses lm-eval for patch generation and adds a scoring step that runs the official swebench harness, emitting an lm-eval-shaped results JSON so the existing collect/validate pipeline works unchanged.
中文说明
新增 SWE-bench Lite 准确率评估支持。由于 lm-eval 无法直接评分 SWE-bench，本 PR 复用 lm-eval 进行补丁生成，并增加一个评分步骤来运行官方 swebench 评估工具，输出符合 lm-eval 格式的结果 JSON，使现有的收集/验证流水线无需改动即可使用。