fix(benchmarks): correct numbers + floors to reproducible values (make reproduce passes)#81
Merged
Merged
Conversation
…lues `make reproduce` (locked stack ST 5.4.1 / transformers 5.5.4 / torch 2.11.0, clean DB, single process, production recall path) yields: LongMemEval-S R@10 98.2% MRR 0.915 (n=500) LoCoMo R@10 91.5% MRR 0.805 (n=1982) BEAM-100K retrieval-proxy MRR 0.55 The prior LoCoMo 94.2% / 0.8278 (adr015, 2026-06-22) is NOT reproducible from code + uv.lock + clean DB: the exact June-22 code on the locked stack now produces 90.3%, and v4.0 produces 91.5% — so there is no v4.0 regression, and the 94.2% figure depended on an original run environment the artifacts don't capture. Rather than ship numbers `make reproduce` contradicts (it failed its own 94.2% floor), the floors and public tables are set to what the harness actually reproduces. LongMemEval reproduces cleanly (98.2 vs 98.4, within the 0.005 gate). BEAM-100K stays un-gated (README scopes it to within-system). The arXiv LaTeX (docs/arxiv-thermodynamic/main.tex, main.pdf) is a dated snapshot and is intentionally NOT edited here — it needs a separate recompile pass with the narrative/derived figures reconciled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E4KjBXAdabUoy5K6Hi41GC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
main currently has the
make reproduceharness (PR #80) but with stale 94.2% LoCoMo floors that fail its own gate. This sets the floors + public benchmark tables (README, CLAUDE.md, banner, GitHub description) to whatmake reproduceactually reproduces on the locked stack + clean DB:Why: the 94.2% (adr015, 2026-06-22) is not reproducible from code + uv.lock + clean DB — the exact June-22 code on the locked stack produces 90.3% today, and v4.0 produces 91.5%, so there is no v4.0 regression; 94.2% depended on an original run environment the artifacts don't capture. Shipping numbers
make reproducecontradicts is the real credibility risk, so the repo is made self-consistent. LongMemEval reproduces cleanly. The arXiv LaTeX is a dated snapshot, updated separately.🤖 Generated with Claude Code
https://claude.ai/code/session_01E4KjBXAdabUoy5K6Hi41GC