Reality probe: what codex-plugin-cc#731's regression tests prove — and what they do not

[hippoley]

This is a CounterProof Reality Probe, not a request to override the review on the source PR.

Source PR: openai/codex-plugin-cc#731

@sylvesterkaczmarek @fscfede-beep — I used #731 as an external test case because the current review has a useful distinction between the original fail-open regression and the new durability / permissions concerns raised on 2026-09-17.

Independent before/after replay

Exact commits:

• HEAD: a81f52f9c2c366d7d5e095179d2473a9f7e12798
• BASE: db52e28f4d9ded852ab3942cea316258ae4ef346

Changed tests replayed:

• tests/runtime.test.mjs
• tests/state.test.mjs

CounterProof result:

• HEAD + changed tests: behavioral PASS
• BASE + the same changed tests: behavioral FAIL
• evidence mode: precise
• result protocol: json-v1
• status: WITNESSED
• digest: sha256:01c447fbfca700a6dcbfcec2753ca3c55a0760b5b6200a751d44a948bdbf5c97

Runner:
https://github.com/hippoley/CounterProof/actions/runs/35952687043

On BASE, the two PR tests fail on the behavior they were added to pin:

1. the cross-plugin-data Stop integration no longer gets the expected blocking result under the old implementation;
2. the state-level A→B plugin-data test reads false where the PR expects the durable true.

So the submitted tests do genuinely distinguish the PR from its base for the original authority-boundary / fail-open bug.

What this does NOT prove

The latest human review asks for two additional properties around the durable config file:

• restrictive file permissions (e.g. 0600);
• preservation of the previous valid config if a replacement write fails / is interrupted, via atomic same-directory replacement.

I inspected the current changed test files at a81f52f9. They test cross-CLAUDE_PLUGIN_DATA authority, but they do not exercise either of those two failure modes.

So this receipt is not evidence that the current CHANGES_REQUESTED concerns are resolved.

The useful statement is narrower:

#731 has a real regression witness for the original fail-open bug; the current reviewer-requested durability and permission properties remain unproven by the PR's tests.

One extra reality finding: before a small CounterProof fix, these .test.mjs files were not detected at all. The exact external PR exposed that gap; it is now fixed in CounterProof main via #12.

Question for the reviewer

Would this separation — what the PR's tests genuinely prove vs. what the current review still leaves unproven — materially help you review an AI-assisted change like this?

If not, what evidence format or additional replay would be more useful?

[sylvesterkaczmarek]

Yes — this separation is materially useful to me as a reviewer, especially for AI-assisted changes.

The strongest part is that it does not turn “BASE fails / HEAD passes” into an overall correctness claim. It identifies the exact property witnessed by the changed tests, then separately names the current review concerns those tests do not exercise.

The format I would find most useful is a compact claim/evidence matrix:

• review claim or failure mode;
• exact test(s) that exercise it;
• BASE result;
• HEAD result;
• whether the test oracle itself was independently checked;
• status: witnessed / contradicted / unproven.

For #731 that would make the distinction very clear: the original authority-boundary regression has a genuine red→green witness, while atomic replacement and restrictive permissions remain unproven. I would keep those “unproven” labels explicit rather than rolling them into a single PASS/FAIL badge.

The digest, exact SHAs, test filenames and runner link are enough provenance for me to reproduce or audit the replay. If CounterProof can also show the relevant assertion/failure excerpt for each witnessed property, that would reduce review time further without pretending the replay proves more than it does.

[hippoley]

@sylvesterkaczmarek — this is exactly the kind of feedback I was hoping for. I did not turn it into a generic feature yet; I first wrote the smallest manual artifact for the exact PR you reviewed:

https://github.com/hippoley/CounterProof/blob/reality/reviewer-requested-matrices/docs/reality/codex-plugin-cc-731-claim-matrix.md

It follows your requested shape:

• review claim / failure mode
• exact test evidence
• BASE
• HEAD
• oracle independently checked?
• status = WITNESSED / UNPROVEN

For #731 it keeps the original authority-boundary regression as WITNESSED, while permissions and atomic replacement stay explicitly UNPROVEN.

I also added an assertion/failure section, but kept it intentionally short rather than dumping logs.

Before I encode this into CounterProof: does this manual matrix look like the thing you meant, or is there a column / evidence excerpt you would change first?

[sylvesterkaczmarek]

Yes, this is the matrix shape I meant.

I would make two small changes before automating it:

1. Rename Oracle independently checked? to something like Oracle alignment, with a fixed vocabulary: UNVERIFIED / ALIGNED / CONTRADICTED. That makes this matrix and the #89404 matrix use the same semantics.
2. Scope the first-row status explicitly as WITNESSED (submitted judge) rather than bare WITNESSED, so nobody can read it as product-level correctness when the oracle column is still UNVERIFIED.

The short assertion/failure section is enough. I would not dump logs. A direct link to the review comment that originated each claim would be useful provenance, but I would treat that as optional metadata rather than another core column.

[hippoley]

@sylvesterkaczmarek — I encoded the vocabulary you confirmed, kept it deliberately manual on claim authority, and merged the smallest version after full CI:

#38

It is now:

counterproof claim-matrix examples/claim_matrix/codex-plugin-cc-731.yml

The command accepts an explicit human/upstream manifest; it does not infer claims or choose an oracle.

Mechanical axes:

submitted-test evidence:
  WITNESSED / NOT_WITNESSED / UNPROVEN

oracle alignment:
  ALIGNED / CONTRADICTED / UNVERIFIED

Derived overall:

WITNESSED + ALIGNED      -> PROVEN
anything + CONTRADICTED  -> CONTRADICTED
WITNESSED + UNVERIFIED   -> WITNESSED (submitted judge)
otherwise                -> UNPROVEN

It also carries exact tests, BASE/HEAD, runner URL, evidence digest, optional review-source URL, and the short assertion/failure excerpt you asked for.

The #731 example is now an acceptance fixture in the repo:
https://github.com/hippoley/CounterProof/blob/main/examples/claim_matrix/codex-plugin-cc-731.yml

The question I care about now is not whether the code looks neat: did the automated artifact preserve the review semantics you meant, or did I make the abstraction worse somewhere?

[sylvesterkaczmarek]

Yes. The automated artifact preserves the review semantics I meant.

The important boundaries survived the automation: WITNESSED is still explicitly scoped to the submitted judge when the oracle is UNVERIFIED, the permissions/atomic-write claims stay UNPROVEN, and runner/digest/test provenance remain attached without being promoted into broader correctness claims.

I would use this shape in review. I do not see a semantic regression in the #731 acceptance fixture.