Skip to content

Reality probe: did Prefect #22698 now prove the two concerns from human review? #30

Description

@hippoley

This is a CounterProof Reality Probe for a real human CHANGES_REQUESTED review.

Source PR:
PrefectHQ/prefect#22698

Human review:
PrefectHQ/prefect#22698 (review)

@desertaxle @Sanjays2402 — the August 3 review asked for two concrete things:

  1. stop reading variables.properties.volumes.default directly and resolve the normal job configuration contract first;
  2. add a regression test proving a customized job_configuration.volumes preserves both its configured volume and the internal /tmp bundle mount.

The current PR head now contains tests for both. I independently replayed those two focused tests against the exact pre-change base.

Independent before / after replay

Exact commits:

  • HEAD: 7604947bcaaec12a83505f842fbc6adb1b88275e
  • BASE: ef4286181588e579f72854d9344f4a2f6d66a090

CounterProof run:
https://github.com/hippoley/CounterProof/actions/runs/35979249605

Result:

  • HEAD: 2/2 PASS
  • BASE + the exact same changed test file: 2/2 FAIL
  • protocol: json-v1
  • evidence mode: precise
  • digest: sha256:4d16ed79a10dacbfe15f025c39dbf9c4f0d7eb724eaa7c4647182ed305b9434f

The two failures match the human review concerns

Default work-pool volume

BASE:

assert 'result-storage:/result-storage' in ['/tmp/...:/tmp/']

The old code kept only the temporary bundle mount and dropped the work-pool default.

HEAD passes.

Customized template volume + internal bundle mount

BASE:

assert any(volume.endswith(':/tmp/') for volume in call_volumes)

The custom custom:/custom volume is present, but the internal temporary /tmp mount requested in the review is missing.

HEAD passes with both.

Claim / evidence matrix

I wrote the compact matrix here:

https://github.com/hippoley/CounterProof/blob/reality/reviewer-requested-matrices/docs/reality/prefect-22698-claim-matrix.md

Current shape:

default work-pool volume propagation       WITNESSED
custom template volume + internal /tmp     WITNESSED
live Docker runtime                        UNPROVEN

This is deliberately not a “PR approved” badge. It only says the current head now has red-before / green-after behavioral coverage for the two concrete failure modes requested in the human review.

Question for @desertaxle

For a re-review like this, is this claim/evidence matrix materially useful — enough to reduce the work of checking whether your previous concerns were actually covered — or would you rather see a different artifact?

I am asking because another external reviewer independently asked CounterProof for exactly this matrix shape. I want to know whether that generalizes across reviewer workflows before making it a product feature.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions