Fresh-install embedding fix + one-command LongMemEval reproduction harness#79
Merged
Conversation
…hable SentenceTransformer loading set HF_HUB_OFFLINE=1 before import, then unset the env var and re-imported on cache miss — but huggingface_hub freezes that env var into module constants at first import, so the retry still ran offline and fresh installs failed with a misleading "couldn't connect to huggingface.co" error. Load with an explicit local_files_only=True first (hermetic when cached, same privacy posture), fall back to a plain online load on cache miss. Reproduced and verified in a clean-env harness run on 2026-07-03. Requires sentence-transformers >= 3.0.0 (local_files_only kwarg absent in v2.3.0, present in v3.0.0 — verified against tagged sources); floors raised accordingly. Also add psycopg-pool to the benchmarks extra: pg_store.py imports ConnectionPool at module level, so the extra could not run the benchmarks it exists for. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVjk4LSGVEjuzCgLMBzvQ2
make longmemeval runs the full official 500-question benchmark from a clean machine: downloads the dataset from the authors' HF repository (sha256-pinned, byte-identical to the file behind every published result), provisions an ephemeral PostgreSQL + pgvector container isolated from any existing install, runs the untouched production runner through the production PL/pgSQL recall path, prints the metric table, and tears down. make longmemeval-smoke gives a 10-question harness check. benchmarks/README.md documents requirements, measured wall-clock (39.6 min full run, a3_longmemeval_post_refactor.md), and the metric scope: session-level retrieval Recall@10/MRR versus the LongMemEval paper's best retrieval configuration (78.4% R@10) — explicitly not comparable to end-to-end QA-accuracy leaderboards. Verified three ways on 2026-07-03: warm run, clean-environment run (fresh uv cache, fresh HF cache, no dataset, no image), and a full 500-question run reproducing the published R@10 98.4% exactly (MRR 0.915; record in benchmarks/results/harness_repro/). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVjk4LSGVEjuzCgLMBzvQ2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
HF_HUB_OFFLINE=1was set before importing sentence-transformers, and the unset-and-retry fallback was dead code because huggingface_hub freezes the env var into module constants at first import. New installs without a cached model failed with a misleading "couldn't connect to huggingface.co". Fixed with an explicitlocal_files_only=Truefirst attempt (hermetic when cached, same privacy posture) and a plain online load on cache miss. Floors raised tosentence-transformers>=3.0.0(kwarg introduced in v3.0.0, verified against tagged sources);psycopg-pooladded to thebenchmarksextra (pg_store.pyimportsConnectionPoolat module level).make longmemeval: one-command reproduction of the full official 500-question benchmark — sha256-pinned dataset download from the authors' HF repo, ephemeral PostgreSQL+pgvector container isolated from any existing install, untouched production runner through the production PL/pgSQL recall path, table + repro manifest out, teardown.make longmemeval-smokefor a 10-question harness check.benchmarks/README.md: requirements, measured wall-clock (39.6 min full run), and explicit metric scoping — session-level retrieval R@10/MRR vs the LongMemEval paper's best retrieval configuration (78.4%), not comparable to end-to-end QA-accuracy leaderboards.Zetetic Checklist
benchmarks/results/a3_longmemeval_post_refactor.md)Review findings
No critical issues. Checked and resolved during review: container reuse under
KEEP_DB=1cannot contaminate results (BenchmarkDB.open()purges all benchmark rows before every run); the ephemeral container's hardcoded password guards a localhost-only throwaway DB with public benchmark data; the embedding change leaves the failure surface unchanged for genuine network errors.Verification
Three independent reproductions on 2026-07-03: warm run, clean-environment run (fresh uv cache, fresh HF cache, no dataset, no Docker image — this run caught both fresh-install bugs), and a full 500-question harness run reproducing the published R@10 98.4% exactly (MRR 0.915; record committed at
benchmarks/results/harness_repro/longmemeval_full_20260703.json).🤖 Generated with Claude Code
https://claude.ai/code/session_01KVjk4LSGVEjuzCgLMBzvQ2