Reality probe: existing tests + environment can witness a fix even when the PR changes no tests

[hippoley]

Source PR:
PrefectHQ/prefect#23146

This is a CounterProof Reality Probe for a proof shape the current product does not represent well.

The PR

PrefectHQ/prefect#23146 changes only:

ui-v2/vite.config.ts

It does not add or modify test files.

The reported bug is environment-dependent: Vitest worker threads inherit the process timezone resolved at startup. CI is already UTC, so the bug is invisible there; on a non-UTC machine existing date-formatting tests fail.

What current CounterProof says

Against the exact PR:

Regression witness: no-changed-tests
Changed tests: 0

That is internally honest under today's contract: CounterProof discovers evidence from tests changed by the PR.

Direct environment-bound replay

I then ran the same existing 8 test files on exact BASE and HEAD under:

TZ=America/Los_Angeles
Node 24.20.0

Reality run:
https://github.com/hippoley/CounterProof/actions/runs/35980349898

Exact commits:

• BASE: 4647d9324933bc83544c3cbd0ebb1e97b4a3f6b3
• HEAD: ca08634e3817e67fc525b137965d4c1a75c6ccc5

Result:

BASE
126 tests
112 pass
14 fail

HEAD
126 tests
126 pass
0 fail

One BASE failure makes the environment dependency visible directly:

expected: Sep 1st, 2021 at 12:00 PM
actual:   Sep 1st, 2021 at 05:00 AM

What this teaches CounterProof

There are at least two assumptions in the current witness contract that are too narrow for this real PR:

1. Evidence set ≠ changed tests only

A PR can repair an environment/test-harness condition that makes pre-existing tests discriminative.

No new test file is required for the evidence to be real.

2. Environment is part of the evidence

The same code/test pair can be:

UTC environment          BASE green
America/Los_Angeles      BASE red

A receipt that records only SHA + command but omits the relevant environment contract can be reproducible only by accident.

Candidate proof shape — not a feature proposal yet

claim
  date-formatting tests are timezone-stable

evidence set
  8 existing test files

environment
  TZ=America/Los_Angeles
  Node=24.20.0

BASE
  112 pass / 14 fail

HEAD
  126 pass / 0 fail

status
  WITNESSED under declared environment

Open question

Should CounterProof eventually allow a reviewer/author to declare an explicit evidence set + environment contract, instead of requiring evidence to originate from changed test files?

Or is that scope better left to CI/workflow tooling, with CounterProof remaining intentionally limited to changed-test witnesses?

I am leaving this as a Reality Probe until there is reviewer/maintainer feedback.