Reality probe: Vercel AI #17096 has one real witness, one non-witness test, and an unverified production oracle

[hippoley]

Source PR: vercel/ai#17096

@doneumark — I used #17096 as a CounterProof Reality Probe because your PR has three different evidence layers that are easy to collapse into one “CI is green” statement.

I am not recommending a merge decision or re-reviewing the design choice.

Independent replay

Exact commits:

• HEAD: da4b91f82c8ccb3eb15ed5abe5217607539cc504
• BASE: 66b71512135a0246a9306e97c8847a5d0bcc57ae

After preserving the pnpm workspace dependency closure so BASE could reach actual test behavior:

• HEAD changed-test run: 38/38 PASS
• BASE + the same changed tests: 37/38 PASS, 1 FAIL
• status: WITNESSED
• digest: sha256:fe2dac513057c98dd17985e6e6be759559a49a8013a0412ec80e07d9efa89bb1

Runner:
https://github.com/hippoley/CounterProof/actions/runs/35986126197

But the two new tests do not prove the same thing

The test that actually distinguishes BASE from HEAD is:

createClaudeCode adapter
forwards the sandbox settings to the bridge start message

On BASE it fails because the start message contains no sandbox; on HEAD it passes.

So this claim is:

host -> bridge sandbox forwarding
WITNESSED (submitted judge)

The other new test:

inboundMessageSchema accepts a start message with sandbox settings

passes on both BASE and HEAD. It only asserts that parsing does not throw, so it is not a regression witness for the schema change:

schema-specific discrimination
NOT WITNESSED

Production claim stays separate

Your PR body also reports real production verification that the Claude Agent SDK applies the forwarded sandbox settings — fail-closed when unavailable, blocked non-allowlisted egress, denied filesystem paths.

CounterProof did not independently run that real SDK/provider oracle, so I kept it:

actual SDK sandbox enforcement
UNPROVEN by submitted tests
oracle alignment: UNVERIFIED

Readable matrix:
https://github.com/hippoley/CounterProof/blob/main/reality/vercel-ai-17096/CLAIM_MATRIX.md

Machine-readable fixture:
https://github.com/hippoley/CounterProof/blob/main/examples/claim_matrix/vercel-ai-17096.yml

One question

For an aging PR where full CI is green and you also have production evidence, is this split useful to a maintainer:

• what the submitted tests independently witness;
• what a submitted test does not distinguish;
• what depends on an external production oracle?

Or is the real blocker here purely the API/design decision (dedicated sandbox key vs generic query-options escape hatch), making this evidence mostly noise?

A short answer is enough.