feat(benchmarks): single-command reproduction (make reproduce) + banner refresh#80
Merged
Conversation
One source of truth for reproducing every benchmark and ablation number:
`make reproduce` -> benchmarks/reproduce.sh. Same clean-DB / production-recall
pipeline every time, deterministic output.
benchmarks/reproduce.sh owns one ephemeral pgvector container and drives the
EXISTING harnesses through it — no new benchmark logic, no forked retriever:
- LongMemEval-S, LoCoMo, BEAM-100K via each run_benchmark.py --results-out
- ablation sweep (baseline + the 13-mechanism v4.0 group) via
benchmarks/lib/ablation_runner.py
- JSON per phase under benchmarks/results/repro/<ts>/ + MANIFEST.json
(git sha, dataset sha, image, package versions), one consolidated table.
Determinism rests on BenchmarkDB: open() purges is_benchmark rows (bench_db.py
line 63) and close()->cleanup() deletes its own, so phases are independent.
The v4.0 ablation mechanism list is defined in ONE place (V4_MECHANISMS in
reproduce.sh); ablation_runner validates each name against the live Mechanism
enum. B1 procedural memory intentionally excluded (opt-in recall_skills tool,
no ablation guard, outside recall fusion). ACTIVE_FORGETTING included (predates
v4.0 but is the circuit that soft-deletes rows — prime multi-session suspect).
Scope flags: --only, --no-ablation/--ablation-only, --ablate-on, --quick,
--limit, --keep-db. Short benchmark names normalized to runner IDs
(beam->beam-100K, longmemeval->longmemeval-s).
repro_longmemeval.sh is now a thin `exec` wrapper into reproduce.sh; Makefile
targets (reproduce, reproduce-smoke, longmemeval, longmemeval-smoke) all
delegate to the one script; benchmarks/README.md headlines the single command.
Statically validated (Docker/PG/model are outside the dev sandbox, so the live
run is the user's first `make reproduce-smoke`): bash -n on both scripts, both
embedded Python blocks parse, all Makefile targets resolve to reproduce.sh, the
--only filter and beam/longmemeval ID normalization tested in isolation, and
the ablation_runner --benchmark/--mechanism/--quick flag contract confirmed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One source of truth for reproducing every benchmark + ablation number:
make reproduce→benchmarks/reproduce.sh. One ephemeral pgvector container drives the EXISTING harnesses (LongMemEval-S, LoCoMo, BEAM-100K via production recall + the 13-mechanism v4.0 ablation sweep via ablation_runner.py), writes per-phase JSON + MANIFEST, one consolidated table, clean teardown. Verified end-to-end viamake reproduce-smokeon a live pgvector:pg16 container (all phases + ablation + MANIFEST, clean teardown). Also includes the v4.0 banner refresh.🤖 Generated with Claude Code
https://claude.ai/code/session_01E4KjBXAdabUoy5K6Hi41GC