This is a CounterProof Reality Probe for a real human CHANGES_REQUESTED review.
Source PR:
PrefectHQ/prefect#22698
Human review:
PrefectHQ/prefect#22698 (review)
@desertaxle @Sanjays2402 — the August 3 review asked for two concrete things:
- stop reading
variables.properties.volumes.default directly and resolve the normal job configuration contract first;
- add a regression test proving a customized
job_configuration.volumes preserves both its configured volume and the internal /tmp bundle mount.
The current PR head now contains tests for both. I independently replayed those two focused tests against the exact pre-change base.
Independent before / after replay
Exact commits:
- HEAD:
7604947bcaaec12a83505f842fbc6adb1b88275e
- BASE:
ef4286181588e579f72854d9344f4a2f6d66a090
CounterProof run:
https://github.com/hippoley/CounterProof/actions/runs/35979249605
Result:
- HEAD: 2/2 PASS
- BASE + the exact same changed test file: 2/2 FAIL
- protocol:
json-v1
- evidence mode:
precise
- digest:
sha256:4d16ed79a10dacbfe15f025c39dbf9c4f0d7eb724eaa7c4647182ed305b9434f
The two failures match the human review concerns
Default work-pool volume
BASE:
assert 'result-storage:/result-storage' in ['/tmp/...:/tmp/']
The old code kept only the temporary bundle mount and dropped the work-pool default.
HEAD passes.
Customized template volume + internal bundle mount
BASE:
assert any(volume.endswith(':/tmp/') for volume in call_volumes)
The custom custom:/custom volume is present, but the internal temporary /tmp mount requested in the review is missing.
HEAD passes with both.
Claim / evidence matrix
I wrote the compact matrix here:
https://github.com/hippoley/CounterProof/blob/reality/reviewer-requested-matrices/docs/reality/prefect-22698-claim-matrix.md
Current shape:
default work-pool volume propagation WITNESSED
custom template volume + internal /tmp WITNESSED
live Docker runtime UNPROVEN
This is deliberately not a “PR approved” badge. It only says the current head now has red-before / green-after behavioral coverage for the two concrete failure modes requested in the human review.
For a re-review like this, is this claim/evidence matrix materially useful — enough to reduce the work of checking whether your previous concerns were actually covered — or would you rather see a different artifact?
I am asking because another external reviewer independently asked CounterProof for exactly this matrix shape. I want to know whether that generalizes across reviewer workflows before making it a product feature.
This is a CounterProof Reality Probe for a real human CHANGES_REQUESTED review.
Source PR:
PrefectHQ/prefect#22698
Human review:
PrefectHQ/prefect#22698 (review)
@desertaxle @Sanjays2402 — the August 3 review asked for two concrete things:
variables.properties.volumes.defaultdirectly and resolve the normal job configuration contract first;job_configuration.volumespreserves both its configured volume and the internal/tmpbundle mount.The current PR head now contains tests for both. I independently replayed those two focused tests against the exact pre-change base.
Independent before / after replay
Exact commits:
7604947bcaaec12a83505f842fbc6adb1b88275eef4286181588e579f72854d9344f4a2f6d66a090CounterProof run:
https://github.com/hippoley/CounterProof/actions/runs/35979249605
Result:
json-v1precisesha256:4d16ed79a10dacbfe15f025c39dbf9c4f0d7eb724eaa7c4647182ed305b9434fThe two failures match the human review concerns
Default work-pool volume
BASE:
The old code kept only the temporary bundle mount and dropped the work-pool default.
HEAD passes.
Customized template volume + internal bundle mount
BASE:
The custom
custom:/customvolume is present, but the internal temporary/tmpmount requested in the review is missing.HEAD passes with both.
Claim / evidence matrix
I wrote the compact matrix here:
https://github.com/hippoley/CounterProof/blob/reality/reviewer-requested-matrices/docs/reality/prefect-22698-claim-matrix.md
Current shape:
This is deliberately not a “PR approved” badge. It only says the current head now has red-before / green-after behavioral coverage for the two concrete failure modes requested in the human review.
Question for @desertaxle
For a re-review like this, is this claim/evidence matrix materially useful — enough to reduce the work of checking whether your previous concerns were actually covered — or would you rather see a different artifact?
I am asking because another external reviewer independently asked CounterProof for exactly this matrix shape. I want to know whether that generalizes across reviewer workflows before making it a product feature.