Backtest: one replay tool for every detector by EdbertChan · Pull Request #371 · EdbertChan/catstack

EdbertChan · 2026-09-10T18:32:46Z

Summary

One command now replays any detector over your real past chats and shows what it would have caught.

Before, three detectors each had their own replay script, and each worked its own way.

Now one shared script replays any of them. This is a refactor: no hook's logic changes.

Review Claim

Move Function: the three per-detector backtest.py scripts become one shared runner, scripts/backtest_detector.py, and each hook gains a replay function the runner calls, with no change to any hook's logic.

Review Lane

refactor

Review Unit

engine-runtime

Safety Invariant

Refactor only: no hook's logic changes. wait-needs-wakeup's check was split into smaller functions so the replay tool can reuse them; its old tests pass on the new code (20 of 22; the other 2 tested the deleted replay script). repeat-error-stop only gains lines. Transcripts stay on the local machine; nothing from them is committed.

Slice Rationale

This PR used to carry the runner plus a gate that asked for its numbers. That gate watched regex patterns, which the phrase-list work is replacing, so it is dropped here and comes back later watching phrase lists instead.

The branch is now one commit on main. It previously sat on the phrase-list PR, which has merged, and carried that PR's commits as well; those are gone, so the diff is this PR's own sixteen files.

Non-goals

No behavior change: no hook's block or allow decision changes.
No gate: the check that asks for backtest numbers comes back in a later PR, for phrase lists.
The deleted backtest.py scripts get no wrappers; their READMEs and cost-audit.md show the runner command.
Transcripts stay local; nothing from them is committed.

Architecture

Before

graph TD
    T["local transcripts"] --> A["repeat-error-stop/backtest.py"]
    T --> B["wait-needs-wakeup/backtest.py"]
    T --> C["reflect/scripts/backtest.py"]
    A --> D1["repeat-error-stop detect.py"]
    B --> D2["wait-needs-wakeup detect.py"]
    C --> D3["token_audit.py frustration check"]

After

graph TD
    T["local transcripts (Claude Code, OMP)"] --> R["scripts/backtest_detector.py"]
    R -->|"--detector PATH:CALLABLE"| D["replay functions: replay_blocks, replay_stop, pretooluse_reason, replay_frustration"]
    R -->|"--compare REF"| G["same detector at a git ref"]
    R --> N["hits, newly caught, newly missed"]

Test Plan

Run on the rebuilt branch, one commit on main:

python3 -m unittest discover -s tests -p test_backtest_detector.py: Ran 20 tests ... OK
python3 -m unittest discover -s engine/hooks/repeat-error-stop/tests: Ran 32 tests ... OK
python3 -m unittest discover -s engine/hooks/wait-needs-wakeup/tests: Ran 25 tests ... OK
python3 -m unittest discover -s engine/skills/reflect/scripts/tests: Ran 180 tests ... OK
uvx ruff check . --select E9,F (CI's lint): All checks passed!
The rebuild keeps exactly the old changes: git patch-id --stable matches the old range's patch id
The old engine/hooks/wait-needs-wakeup/tests/test_hooks.py run against this branch's detect.py, with an empty stand-in for the deleted backtest module: Ran 22 tests, FAILED (errors=2), both errors in TestBacktestReproducesTheIncident, which tested the deleted script. The 20 tests of the hook's own decisions pass.
python3 scripts/backtest_detector.py --detector engine/hooks/diu-stop/claude_stop_check.py:find_unverified_claim --limit 3: exit 0
Python 3.9 syntax scan (ast.parse(..., feature_version=(3, 9))) of the changed .py files: problems: 0

Two fixes were needed to move onto current code:

--compare HEAD on macOS loaded the working tree instead of HEAD: /var is a link to /private/var, so the path relative to the git toplevel climbed out of the archived copy. test_compare_against_git_ref_loads_that_revision_and_its_siblings failed before the fix and passes now.
The near-miss test used a bare UNVERIFIED: line, which diu-stop no longer honours; it now uses inline code excused by fenced output.

Revert Plan

Safe to revert? Yes. Nothing imports the runner yet.
Revert command: git revert <merge-sha>
Post-revert steps: None. The three per-detector backtest.py scripts come back.
Data migration? No

🤖 Generated with Claude Code

https://claude.ai/code/session_01Gnua5cuAAF1ey1b7zRqJCC

Note

Low Risk
Refactor-focused: hook blocking logic is largely unchanged aside from small extractions for replay; new runner is not imported by production hooks yet.

Overview
Introduces scripts/backtest_detector.py, a shared CLI that replays any PATH:CALLABLE detector over local Claude/OMP JSONL (discovery, streaming, hit rates, --json, and --compare against another detector or a git revision).

Per-detector replay hooks are added or exposed instead of bespoke runners: replay_blocks (repeat-error-stop), pretooluse_reason + replay_stop (wait-needs-wakeup, with WakeupTracker / stop_gaps extracted for reuse), and replay_frustration (reflect token_audit, plus direct_human_claude_rows for row streaming). wait-needs-wakeup/backtest.py and reflect/scripts/backtest.py (and its unit tests) are removed; docs point at the shared runner. repeat-error-stop keeps its legacy backtest.py but documents the new entry point.

Tests: new tests/test_backtest_detector.py (20 cases); hook tests updated for replay APIs (TestReplayBlocks, TestBacktestDetectors, TestReplayFrustration).

^{Reviewed by Cursor Bugbot for commit 6b4ee38. Bugbot is set up for automated code reviews on this repo. Configure here.}

cursor · 2026-09-10T18:32:58Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.