Reality probe: CrewAI #7721 has one witnessed claim and two still-unproven review claims

[hippoley]

This is a CounterProof Reality Probe for an active external PR:

crewAIInc/crewAI#7721

@joaomdmoura — I used #7721 because it is a good example of why “CI is green” and even “the new regression is genuinely red→green” are still not the same as “every claim in the PR is proven.”

Independent before/after replay

Exact commits:

• HEAD: cf5ca5432f0d473c239194899a29d7c992ca7bae
• BASE: bdd1bc62007fcee732c912f0daa093e1f760f3fc

Changed test:

• lib/crewai/tests/tracing/test_tracing.py

CounterProof result:

• HEAD: behavioral PASS
• BASE + same changed test: behavioral FAIL
• status: WITNESSED
• protocol: json-v1
• digest: sha256:15974b0cf5d8d0bdcf12943609344192b3e2ea5fd6a3f981b2c765829dfb8e3a

Runner:
https://github.com/hippoley/CounterProof/actions/runs/35976674576

On BASE, pytest reached normal test execution and the only regression failure was:

test_a_process_with_no_terminal_answers_with_the_tracing_switch
[tracing was turned on: share it]

assert False is True
where False = prompt_user_for_trace_viewing(sharing=True)

The rest of the file remained green apart from one existing skip.

So the CREWAI_TRACING_ENABLED=true non-interactive sharing behavior has a real before/after witness.

But the current review surface is larger than that witness

The PR prose also says explicit opt-in may come from:

• programmatic tracing=True;
• persisted prior consent.

The changed test does not exercise either path.

Two automated reviewers independently raised concerns around those uncovered surfaces:

• Cursor Bugbot: the non-interactive branch calls should_enable_tracing() without the programmatic override, so tracing=True may still be invisible there.
• CodeRabbit: persisted trace_consent semantics may accept values beyond literal Boolean True.

I am not treating those bot findings as proven bugs. I am treating them as review claims with no corresponding changed regression evidence yet.

Claim / evidence matrix

I wrote the compact matrix an external human reviewer asked CounterProof to produce:

https://github.com/hippoley/CounterProof/blob/main/examples/claim_matrix/crewai-7721.yml

Current shape:

CREWAI_TRACING_ENABLED=true non-TTY sharing  WITNESSED
tracing disabled control                     PRESERVED
sharing=False control                        PRESERVED
programmatic tracing=True path               UNPROVEN
persisted consent semantics                  UNPROVEN

Question

As the author of #7721, is this split useful or annoying?

In particular: would you rather see an independent receipt say “one exact claim witnessed; two review claims still unproven” than get another generic “tests passed” badge?

I am using the answer to decide whether claim/evidence matrices belong in CounterProof at all.

Follow-up: the two uncovered concerns are now independently probed

A second clean run kept the original regression witness and directly exercised the adjacent paths:

• programmatic tracing context True, no env override, persisted consent False → expected sharing True, actual False;
• persisted trace_consent=None → True;
• persisted trace_consent="false" → True;
• Boolean True → True; Boolean False → False.

Runner:
https://github.com/hippoley/CounterProof/actions/runs/35981324820

Updated matrix:
https://github.com/hippoley/CounterProof/blob/reality/reviewer-requested-matrices/docs/reality/crewai-7721-claim-matrix.md

So the useful boundary is now stronger than “two claims unproven”: one adjacent path is directly contradicted, while malformed persisted-consent handling is independently reproducible and needs review.