[Unverified tag check](2) llm-judge inbox shows the judge's report line by EdbertChan · Pull Request #540 · EdbertChan/catstack

EdbertChan · 2026-09-13T06:20:30Z

Summary

When the background judge flags a reply, the note the agent gets now ends with the judge's own one-line report, so the agent sees what was found.

Reports are cut to 600 characters. Flags with no report, clean results, and results the judge could not produce read exactly as before.

The judge's guide page now describes the added line.

Review Claim

Approve adding a non-empty judge report to the end of a flag note, while every note without a report stays the same.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

A hit without a string report produces the exact existing message. Clean verdicts still produce no message, and unchecked verdicts remain unchanged.

Slice Rationale

This slice carries the optional report from the finished verdict into the inbox and documents that contract alongside its tests.

Non-goals

No changes to judge prompts or verdict generation.
No changes to clean or unchecked handling.
No changes to other hooks.
No README changes outside the inbox contract.

Architecture

Before

graph TD
    A["finished hit verdict"] --> B["fixed inbox message"]
    B --> C["agent"]

After

graph TD
    A["finished hit verdict with optional report"] --> B["inbox message with report clipped to 600 characters"]
    B --> C["agent sees the judge finding"]

Test Plan

python3 -m unittest discover -s engine/hooks/llm-judge/tests -p test_inbox.py -k report -v
python3 -m unittest discover -s engine/hooks/llm-judge/tests -v
python3 scripts/check_no_new_comments.py --base origin/main
bash scripts/scrub-handoff-artifacts.sh

Revert Plan

Safe to revert? Yes
Revert command: git revert <this PR commit>
Post-revert steps: None
Data migration? No

Note

Low Risk
Narrow inbox formatting change with backward-compatible behavior when no string report is present; no judge, auth, or data-path changes.

Overview
When a background llm-judge verdict is a hit, the agent’s inbox line is no longer only the job’s on_hit text: if the verdict’s answer includes a non-empty string report, that string is appended after a space, trimmed, and clipped to 600 characters.

Clean and unchecked inbox behavior is unchanged, as are hits with no string report (including numeric or list report values, which are ignored). The inbox README section documents the new hit format, and test_inbox.py adds direct verdict seeding plus coverage for append, clip, type filtering, and clean-with-report.

^{Reviewed by Cursor Bugbot for commit ef227f4. Bugbot is set up for automated code reviews on this repo. Configure here.}

cursor · 2026-09-13T06:20:36Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_e43d3da5-aac5-4162-abd0-a8395202e09d)

mergify · 2026-09-13T18:16:44Z

Queued — the merge queue status continues in this comment ↓.

EdbertChan · 2026-09-14T03:54:55Z

Mergify repair stopped: required check failed: validate. The retry cap was reached for current head b0d3d9c.

…tep 1 for the llm-judge investigate mode. Review claim: The investigate-mode unit tests run and pass. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The run exits 0 and reports at least one test; Python exits 5 when -k matches nothing. Slice rationale: One proof step for this workflow's single change. Architectural effect: None; verification only. Goal: Prove the investigate runner argv and time limit deterministically. Motivation: Each slice carries its own executable proof. Alternative considerations: Manual inspection was rejected as non-deterministic. Implementation details: Run the filtered unittest discovery once. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 964f7535-7395-4bb3-b657-37a77b357f44

…or the llm-judge investigate mode. Review claim: Every llm-judge test still passes after the change. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The whole llm-judge test folder exits 0. Slice rationale: One proof step guarding current judge callers. Architectural effect: None; verification only. Goal: Prove default jobs behave as before. Motivation: wrong-check-reflect and diu-stop already depend on the judge. Alternative considerations: Running only the new tests was rejected; it would miss a regression in the default runner set. Implementation details: Run the whole llm-judge test folder. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: c94db032-2d49-4594-b6e7-b3ac43e7b56b

…p 3 for the llm-judge investigate mode. Review claim: The change adds no explanatory code comments. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The comment gate exits 0 against origin/main. Slice rationale: One proof step for catstack's required comment gate. Architectural effect: None; verification only. Goal: Keep the CI comment gate green. Motivation: CI runs this gate on every pull request. Alternative considerations: Waiting for CI was rejected; it is slower to find. Implementation details: Run the comment gate against origin/main. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 492bf3e7-0cff-46b3-b0d1-89687e47b6b7

…ephemeral inter-task handoff files remain before the merge gate. Review claim: Only ephemeral handoff files are checked; no product or test file is touched. Review lane: cleanup Safety invariant: The scrub script runs read-only and never touches the home Invoker ledger. Effectiveness measurement: The scrub script exits 0 and reports no handoff path. Slice rationale: One cleanup slice required on every implementation plan that opens a pull request, kept separate from behavior and proof work. Architectural effect: None. Goal: Leave a clean worktree for the pull request. Motivation: Ephemeral inter-task files must not leak into the reviewed diff. Alternative considerations: Leaving this step out was rejected; it is a hard requirement for every plan that opens a pull request. Implementation details: Run the repository's handoff-scrub script without --apply. Non-goals: No product edits in this task. Layer: docs Feature state: active Exit code: 0 Invoker-Finalize-Id: 2aafd01f-794a-46f1-8763-0a04ed358458

… judge answer's report string to a hit message in the llm-judge inbox. Review claim: inbox.messages() returns on_hit followed by a space and answer["report"] when the verdict is a hit and report is a non-empty string, clipped to 600 characters; every other verdict produces the same text as before. Review lane: behavior Safety invariant: A verdict whose answer carries no "report" string yields byte-for-byte the same message as today, so wrong-check-reflect and diu-stop messages are unchanged. Clean verdicts still produce nothing. Effectiveness measurement: Unit tests prove a hit with a report ends with that report, a hit lacking one equals on_hit exactly, a report over 600 characters is clipped, and a non-string report is ignored. Slice rationale: One behavior slice in the inbox, dormant because no current judge prompt asks for a report key; the hook that asks for it lands next. Architectural effect: The inbox can carry per-verdict evidence text from the model to the agent. Goal: Let a judge tell the agent what it found, not just that it found something. Motivation: The planned unverified-tag check must report whether a "cannot verify" blocker and its claim were true or false, with the file or quote that shows it. A fixed on_hit sentence cannot carry that. Alternative considerations: Building the message inside the hook at enqueue time was rejected because the answer does not exist until the background run ends. A new verdict field was rejected as more surface than one optional answer key. Implementation details: In engine/hooks/llm-judge/inbox.py, in messages(), after picking the hit text, read item.get("answer"); when it is a dict whose "report" value is a str with non-blank content, append " " plus that value stripped and clipped to REPORT_LIMIT = 600 characters. Add tests to engine/hooks/llm-judge/tests/test_inbox.py. Non-goals: No change to judge.py, to unchecked or clean handling, or to any hook. No README edit in this task. Layer: domain Feature state: dormant Files: - engine/hooks/llm-judge/inbox.py - engine/hooks/llm-judge/tests/test_inbox.py Change types: - engine/hooks/llm-judge/inbox.py: modify - engine/hooks/llm-judge/tests/test_inbox.py: modify Acceptance criteria: - Tests whose names contain "report" pass. - The whole llm-judge test folder passes. - python3 scripts/check_no_new_comments.py --base origin/main exits 0. Solution: Append the judge answer's report string to a hit message in the llm-judge inbox. Review claim: inbox.messages() returns on_hit followed by a space and answer["report"] when the verdict is a hit and report is a non-empty string, clipped to 600 characters; every other verdict produces the same text as before. Review lane: behavior Safety invariant: A verdict whose answer carries no "report" string yields byte-for-byte the same message as today, so wrong-check-reflect and diu-stop messages are unchanged. Clean verdicts still produce nothing. Effectiveness measurement: Unit tests prove a hit with a report ends with that report, a hit lacking one equals on_hit exactly, a report over 600 characters is clipped, and a non-string report is ignored. Slice rationale: One behavior slice in the inbox, dormant because no current judge prompt asks for a report key; the hook that asks for it lands next. Architectural effect: The inbox can carry per-verdict evidence text from the model to the agent. Goal: Let a judge tell the agent what it found, not just that it found something. Motivation: The planned unverified-tag check must report whether a "cannot verify" blocker and its claim were true or false, with the file or quote that shows it. A fixed on_hit sentence cannot carry that. Alternative considerations: Building the message inside the hook at enqueue time was rejected because the answer does not exist until the background run ends. A new verdict field was rejected as more surface than one optional answer key. Implementation details: In engine/hooks/llm-judge/inbox.py, in messages(), after picking the hit text, read item.get("answer"); when it is a dict whose "report" value is a str with non-blank content, append " " plus that value stripped and clipped to REPORT_LIMIT = 600 characters. Add tests to engine/hooks/llm-judge/tests/test_inbox.py. Non-goals: No change to judge.py, to unchecked or clean handling, or to any hook. No README edit in this task. Layer: domain Feature state: dormant Files: - engine/hooks/llm-judge/inbox.py - engine/hooks/llm-judge/tests/test_inbox.py Change types: - engine/hooks/llm-judge/inbox.py: modify - engine/hooks/llm-judge/tests/test_inbox.py: modify Acceptance criteria: - Tests whose names contain "report" pass. - The whole llm-judge test folder passes. - python3 scripts/check_no_new_comments.py --base origin/main exits 0. Invoker-Finalize-Id: c48ed200-f1d8-4426-b7f9-e0b2f09a285c

… for the inbox report line. Review claim: The report-line unit tests run and pass. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The run exits 0 and reports at least one test; Python exits 5 when -k matches nothing. Slice rationale: One proof step for this workflow's single change. Architectural effect: None; verification only. Goal: Prove the report line deterministically. Motivation: Each slice carries its own executable proof. Alternative considerations: Manual inspection was rejected as non-deterministic. Implementation details: Run the filtered unittest discovery once. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 7ffa5643-18c9-4676-875e-a4105c1be37b

…or the inbox report line. Review claim: Every llm-judge test still passes after the change. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The whole llm-judge test folder exits 0. Slice rationale: One proof step guarding current inbox callers. Architectural effect: None; verification only. Goal: Prove messages for current callers are unchanged. Motivation: wrong-check-reflect and diu-stop already deliver through the inbox. Alternative considerations: Running only the new tests was rejected; it would miss a regression for current callers. Implementation details: Run the whole llm-judge test folder. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 4c6eee23-2412-44e4-bb8e-665509f2f540

…p 3 for the inbox report line. Review claim: The change adds no explanatory code comments. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The comment gate exits 0 against origin/main. Slice rationale: One proof step for catstack's required comment gate. Architectural effect: None; verification only. Goal: Keep the CI comment gate green. Motivation: CI runs this gate on every pull request. Alternative considerations: Waiting for CI was rejected; it is slower to find. Implementation details: Run the comment gate against origin/main. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 49853f19-8166-4bc3-89a1-dbf73ed9688c

…e report line in the llm-judge README's inbox section. Review claim: The "Delivery: the inbox" section of engine/hooks/llm-judge/README.md says a hit whose answer carries a report string ends with that report, clipped to 600 characters. Review lane: docs Safety invariant: Only engine/hooks/llm-judge/README.md changes; no code or test file is edited. Effectiveness measurement: grep finds the word report and the number 600 in the README inbox section. Slice rationale: The README line for the behavior in this same pull request, which catstack's ship-a-detector playbook requires to ship together. Architectural effect: None; README only. Goal: Keep the inbox contract in the README accurate. Motivation: catstack keeps every hook's contract in its README. Alternative considerations: A later README-only pull request was rejected; the playbook counts that as a known repair pattern. Implementation details: Edit the hit bullet under "## Delivery: the inbox" in engine/hooks/llm-judge/README.md. Non-goals: No code or test edits. Layer: docs Feature state: active Files: - engine/hooks/llm-judge/README.md Change types: - engine/hooks/llm-judge/README.md: modify Acceptance criteria: - `grep -n "report" engine/hooks/llm-judge/README.md` exits 0. Context: wf-1789279268766-11/implement-inbox-report-line (0d98bfa): Append the judge answer's report string to a hit message in the llm-judge inbox. Review claim: inbox.messages() returns on_hit followed by a space and answer["report"] when the verdict is a hit and report is a non-empty string, clipped to 600 characters; every other verdict produces the same text as before. Review lane: behavior Safety invariant: A verdict whose answer carries no "report" string yields byte-for-byte the same message as today, so wrong-check-reflect and diu-stop messages are unchanged. Clean verdicts still produce nothing. Effectiveness measurement: Unit tests prove a hit with a report ends with that report, a hit lacking one equals on_hit exactly, a report over 600 characters is clipped, and a non-string report is ignored. Slice rationale: One behavior slice in the inbox, dormant because no current judge prompt asks for a report key; the hook that asks for it lands next. Architectural effect: The inbox can carry per-verdict evidence text from the model to the agent. Goal: Let a judge tell the agent what it found, not just that it found something. Motivation: The planned unverified-tag check must report whether a "cannot verify" blocker and its claim were true or false, with the file or quote that shows it. A fixed on_hit sentence cannot carry that. Alternative considerations: Building the message inside the hook at enqueue time was rejected because the answer does not exist until the background run ends. A new verdict field was rejected as more surface than one optional answer key. Implementation details: In engine/hooks/llm-judge/inbox.py, in messages(), after picking the hit text, read item.get("answer"); when it is a dict whose "report" value is a str with non-blank content, append " " plus that value stripped and clipped to REPORT_LIMIT = 600 characters. Add tests to engine/hooks/llm-judge/tests/test_inbox.py. Non-goals: No change to judge.py, to unchecked or clean handling, or to any hook. No README edit in this task. Layer: domain Feature state: dormant Files: - engine/hooks/llm-judge/inbox.py - engine/hooks/llm-judge/tests/test_inbox.py Change types: - engine/hooks/llm-judge/inbox.py: modify - engine/hooks/llm-judge/tests/test_inbox.py: modify Acceptance criteria: - Tests whose names contain "report" pass. - The whole llm-judge test folder passes. - python3 scripts/check_no_new_comments.py --base origin/main exits 0. Solution: Describe the report line in the llm-judge README's inbox section. Review claim: The "Delivery: the inbox" section of engine/hooks/llm-judge/README.md says a hit whose answer carries a report string ends with that report, clipped to 600 characters. Review lane: docs Safety invariant: Only engine/hooks/llm-judge/README.md changes; no code or test file is edited. Effectiveness measurement: grep finds the word report and the number 600 in the README inbox section. Slice rationale: The README line for the behavior in this same pull request, which catstack's ship-a-detector playbook requires to ship together. Architectural effect: None; README only. Goal: Keep the inbox contract in the README accurate. Motivation: catstack keeps every hook's contract in its README. Alternative considerations: A later README-only pull request was rejected; the playbook counts that as a known repair pattern. Implementation details: Edit the hit bullet under "## Delivery: the inbox" in engine/hooks/llm-judge/README.md. Non-goals: No code or test edits. Layer: docs Feature state: active Files: - engine/hooks/llm-judge/README.md Change types: - engine/hooks/llm-judge/README.md: modify Acceptance criteria: - `grep -n "report" engine/hooks/llm-judge/README.md` exits 0. Invoker-Finalize-Id: c105fd49-67e1-44b6-827f-95db211a0555

…ephemeral inter-task handoff files remain before the merge gate. Review claim: Only ephemeral handoff files are checked; no product or test file is touched. Review lane: cleanup Safety invariant: The scrub script runs read-only and never touches the home Invoker ledger. Effectiveness measurement: The scrub script exits 0 and reports no handoff path. Slice rationale: One cleanup slice required on every implementation plan that opens a pull request, kept separate from behavior and proof work. Architectural effect: None. Goal: Leave a clean worktree for the pull request. Motivation: Ephemeral inter-task files must not leak into the reviewed diff. Alternative considerations: Leaving this step out was rejected; it is a hard requirement for every plan that opens a pull request. Implementation details: Run the repository's handoff-scrub script without --apply. Non-goals: No product edits in this task. Layer: docs Feature state: active Exit code: 0 Invoker-Finalize-Id: 007250b8-44a0-4a48-a256-7fd291266e8f

cursor · 2026-09-14T04:12:07Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_95fb7b31-e5d9-4f5e-a1f8-41ef6b817248)

EdbertChan · 2026-09-14T04:17:42Z

@Mergifyio queue

mergify · 2026-09-14T04:18:14Z

Merge Queue Status

✅ Entered queue — 2026-09-14 04:18 UTC · Rule: admin-bypass · triggered by @EdbertChan with the @mergifyio queue command
✅ Checks passed · on draft merge queue: checking #540 on main (344657b), stacked on #519 and #525 #587
✅ Merged — 2026-09-14 04:25 UTC · at f9a3be4f9f4d867424e9060575fad91e43f0eeb1 · squash

This pull request spent 6 minutes 46 seconds in the queue, including 4 minutes 56 seconds running CI.

Required conditions to merge

check-success = lint
check-success = test
check-success = validate

EdbertChan added the admin-bypass label Sep 13, 2026

EdbertChan mentioned this pull request Sep 13, 2026

[Unverified tag check](3) a finished reply's unverified tags get a background read-only check #550

Closed

7 tasks

Base automatically changed from plan/unverified-tag-check-1-judge-investigate-mode to main September 13, 2026 18:15

Invoker Bot added 10 commits September 13, 2026 20:58

EdbertChan force-pushed the plan/unverified-tag-check-2-inbox-report branch from b0d3d9c to ef227f4 Compare September 14, 2026 04:12

mergify Bot added the queued label Sep 14, 2026

mergify Bot mentioned this pull request Sep 14, 2026

merge queue: checking #540 on main (344657b), stacked on #519 and #525 #587

Closed

6 tasks

mergify Bot merged commit f9a3be4 into main Sep 14, 2026
6 checks passed

mergify Bot deleted the plan/unverified-tag-check-2-inbox-report branch September 14, 2026 04:24

mergify Bot removed the queued label Sep 14, 2026

Navigation Menu

[Unverified tag check](2) llm-judge inbox shows the judge's report line - #540

[Unverified tag check](2) llm-judge inbox shows the judge's report line#540
mergify[bot] merged 10 commits into
mainfrom
plan/unverified-tag-check-2-inbox-report

EdbertChan commented Sep 13, 2026 •

edited by cursor Bot

Loading

Uh oh!

cursor Bot commented Sep 13, 2026

Uh oh!

mergify Bot commented Sep 13, 2026 •

edited

Loading

Uh oh!

EdbertChan commented Sep 14, 2026

Uh oh!

cursor Bot commented Sep 14, 2026

Uh oh!

EdbertChan commented Sep 14, 2026

Uh oh!

mergify Bot commented Sep 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

EdbertChan commented Sep 13, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review Claim

Review Lane

Review Unit

Safety Invariant

Slice Rationale

Non-goals

Architecture

Before

After

Test Plan

Revert Plan

Uh oh!

cursor Bot commented Sep 13, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

mergify Bot commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EdbertChan commented Sep 14, 2026

Uh oh!

cursor Bot commented Sep 14, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

EdbertChan commented Sep 14, 2026

Uh oh!

mergify Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

EdbertChan commented Sep 13, 2026 •

edited by cursor Bot

Loading

mergify Bot commented Sep 13, 2026 •

edited

Loading

mergify Bot commented Sep 14, 2026 •

edited

Loading