Skip to content

feat(progress-review): close the drift loop with typed receipts and a recorded differential - #4854

Draft
songoow wants to merge 9 commits into
loopx-project:mainfrom
songoow:codex/jev-drift-shadow-upstream
Draft

songoow wants to merge 9 commits into
loopx-project:mainfrom
songoow:codex/jev-drift-shadow-upstream

Conversation

@songoow

@songoow songoow commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Closes the progress-review loop that #4854 started, on top of the existing LoopX contracts, and ships a reproducible differential.

  • Core capability progress-review-sentinel (default off). A per-goal policy control_plane.progress_review with modes off / shadow / assist, a strict progress_review_receipt_v0 contract, and a pure trigger that counts consecutive completed drift receipts joined to run rows by turn identity. In assist, the trigger raises the existing autonomous_replan_obligation (kind: external_progress_review_drift) after the typed repeat fuse; unknown, abstained, failed or missing receipts break the streak; an acknowledged replan re-arms it; a changed goal contract invalidates earlier receipts. The refresh-state writeback judges an acknowledgement against the same obligation that loopx status shows.
  • Policy surfaces. configure-goal --progress-review-mode|--progress-review-signal|--progress-review-drift-threshold|--clear-progress-review-configuration, the configuration catalog, the Dashboard capability editor, the chat configuration API and dashboard localization. Malformed blocks fail closed to off.
  • Observer (packages/loopx-jev). Three Noul questions next to the two Choice questions; typed drift signals derived with the label threshold; drift init --runtime-root writes one receipt per evaluated event under the goal runtime; drift label records private human truth and drift status reports agreement. The core imports nothing from the package.
  • Differential harness. loopx-jev sentinel compare replays a frozen 16-sequence matrix (6 constructed cosmetic-drift sequences, 3 mixed sequences that drift after genuine work, 7 real upstream commits labelled on-goal) against committed provider recordings; --live records fresh answers. examples/progress-review-sentinel-smoke.py replays it without a credential.
  • Closed-loop test. packages/loopx-jev/tests/test_closed_loop.py drives one real refresh-state sequence three ways: off sees nothing (the typed fuse cannot fire on self-declared advanced), shadow shows receipts and no obligation, assist raises the obligation, loopx status publishes it, a real acknowledged replan re-arms it, and one more drift round is not enough.

Why

typed_progress_repeat_trigger needs identical fingerprints plus a self-declared unchanged/blocked. An Agent that keeps declaring advanced with a fresh hypothesis_id while only renaming identifiers is invisible to it, and the periodic review looks at that Goal only after 20 durable runs.

Recorded differential (committed live run, jev-1.13.0, 35 rounds, all completed)

Typed repeat fuse noul receipts choice receipts
Drift sequences flagged at or after their gold round 0/9 (invisible by construction) 6/9 4/9
Reaching the assist obligation at threshold 2 0/9 6/9 1/9
Real on-goal upstream commits falsely flagged 0/7 0/7 0/7
Premature flags inside mixed sequences 0 1 0

All six purely cosmetic sequences, including an 18 KB rename sweep, were flagged at round 1 and would raise the obligation at round 2. Client-measured assessment latency: 807 ms median, 1.5 s p95; 1879 median input tokens.

What it misses. Once a real implementation exists in the scoped file, later cosmetic churn on that file kept serves_acceptance at 0.62–0.90, so none of the three mixed sequences was flagged at its drift round. One executed negative probe was flagged as drift in two of three live runs. Across three live runs noul flagged 6, 7 and 6 of 9; one of 35 answers in one run was rejected by the strict decoder (selected label ≠ argmax) and failed closed.

Verification

  • Package: 73 tests (real Git, real refresh-state, injected answers, receipts, labels, matrix strictness, replay determinism, closed loop).
  • Core: new tests/control_plane/test_external_progress_review.py and tests/capabilities/test_progress_review.py; regression on obligation, refresh gate, configuration UI, chat API and capability registry suites (115 + 76 passing locally).
  • ruff check on the CI scope, strict mypy on the new modules, docs-governance-smoke, repository-hygiene-smoke, progress-review-sentinel-smoke.
  • Three live recordings (each ~35 provider calls); the committed one is the last.

Boundaries and what is not claimed

  • No pause, user gate, Turn settlement or Goal acceptance authority. Escalation legs are future work.
  • The core never calls a model, never reads a raw delta, never overwrites progress_observation.
  • Gold labels for constructed cases come from their author; real commits are labelled on-goal by having merged upstream. Injected answers prove plumbing, not model quality.
  • No claim of reduced wasted work or task-completion improvement; that needs a separately authorized intervention study.

Still a Draft: maintainers decide whether this bounded closed loop and its recorded differential are worth adopting.

@songoow songoow changed the title feat(jev): add optional D1 progress shadow observations feat(jev): add optional progress review from scoped file changes Sep 21, 2026
@songoow
songoow marked this pull request as draft September 21, 2026 09:40
Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: song <liusongstep@gmail.com>
…n trigger

Add the default-off progress_review policy, the strict progress_review_receipt_v0 contract and a pure trigger that counts consecutive completed drift receipts joined to run rows by turn identity. In assist the trigger raises the existing autonomous replan obligation after the typed repeat fuse; unknown, abstained, failed or missing receipts break the streak, an acknowledged replan re-arms it, and a changed goal contract invalidates earlier receipts. The same context is read by status projections and by the refresh-state writeback, so an acknowledgement is judged against the obligation that status shows. The core imports nothing from the observer package and adds no pause, gate or acceptance authority.

Signed-off-by: song <liusongstep@gmail.com>
…aces

Register the builtin progress-review-sentinel capability and expose its per-goal policy (off, shadow, assist; drift signal; drift threshold) through configure-goal flags, the configuration catalog, the Dashboard capability editor, the chat configuration API and dashboard localization, with a bilingual capability README. The policy is default off and a malformed block fails closed to off.

Signed-off-by: song <liusongstep@gmail.com>
The observer asks three Noul questions (behavior_change, serves_acceptance, evidence_increment) next to the two Choice questions, derives the noul and choice drift signals with the configured label threshold, and, when bound to a runtime root, writes one progress_review_receipt_v0 per evaluated event under the goal runtime. Probabilities inside the undecided band are not decisions, so an evaluation with no decided answer still abstains. drift label records a private human truth and drift status reports agreement per signal; labels never enter a receipt.

Signed-off-by: song <liusongstep@gmail.com>
…d-loop test

loopx-jev sentinel compare replays a frozen 16-sequence matrix (6 constructed cosmetic-drift sequences, 3 mixed sequences that drift after genuine work, 7 real upstream commits labelled on-goal) against recorded provider answers and reports, per sequence, when the typed repeat fuse fires, when each receipt signal first flags drift, when assist would raise the obligation, and every false flag; --live records fresh answers. The closed-loop test drives one real refresh-state sequence through off, shadow and assist, checks loopx status, accepts a real acknowledged replan and verifies the re-arm. The smoke replays the committed recording without a credential.

Signed-off-by: song <liusongstep@gmail.com>
Update the operation guide and the decision record in both languages, link the capability from the README capability tables and the research RFC, and state what the committed live recording shows and does not show: the typed fuse fires on none of the sixteen sequences, the noul signal flags six of nine drift sequences at round one with no false flags on real commits, cosmetic churn after a landed implementation is missed, and one executed negative probe was flagged in two of three live runs.

Signed-off-by: song <liusongstep@gmail.com>
@songoow
songoow force-pushed the codex/jev-drift-shadow-upstream branch from 1f2f218 to 6be1537 Compare September 21, 2026 10:59
@songoow songoow changed the title feat(jev): add optional progress review from scoped file changes feat(progress-review): close the drift loop with typed receipts and a recorded differential Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant