Skip to content

fix(harness): stop masking lead lane failures as population binding mismatches - #187

Merged
phodal merged 1 commit into
QoderAI:mainfrom
1339190177:fix/164-lead-failure-passthrough
Sep 23, 2026
Merged

phodal merged 1 commit into
QoderAI:mainfrom
1339190177:fix/164-lead-failure-passthrough

Conversation

@1339190177

Copy link
Copy Markdown
Contributor

Summary

An evidence bundle whose lead analyzer fails for its own reason now keeps the
lead lane's original error code. Binding reconciliation no longer manufactures
lead-side errors from absent data, genuine conflicts expose their concrete
reconciliation errors, and the fail-closed contract is unchanged.

Why

Traceability and Scope

  • Spec/ADR: docs/specs/2026-09-23-164-lead-failure-passthrough.md
  • Acceptance criteria addressed:
    • AC-1: an unobserved lead lane keeps its own error code; diagnostics report
      bound with leadObserved: false; bundle status stays failed.
    • AC-2: a lead that returns data contradicting the frozen population keeps
      the fail-closed SESSION_POPULATION_BINDING_MISMATCH downgrade, and the
      diagnostics add the concrete reconciliation errors plus
      leadObserved: true.
    • AC-3: validateSessionPopulationBundle gains an optional leadObserved
      flag (default true); existing callers keep their behavior.
  • Canonical owners changed: scripts/session-analysis/session-population.mjs,
    scripts/harness-analysis/evidence-bundle/index.mjs, focused tests in
    test/reporting/better-harness-evidence-bundle.test.mjs, new spec under
    docs/specs/.
  • Explicit non-goals: no change to the fail-closed reconciliation contract,
    lane envelope shapes, discovery or qualification behavior, report schemas,
    or the underlying Windows-specific lead failure from [Bug]: lead lane always fails with SESSION_POPULATION_BINDING_MISMATCH on Claude while sessionEvidence succeeds #164.

Change Type

  • Feature
  • Bug fix
  • Tests only
  • Documentation/community
  • Refactor with no intended behavior change
  • Dependency, packaging, or infrastructure

Test and Review Evidence

Check Result
New regression test on the pre-fix implementation (source change stashed) red — 2 assertions fail
npx vitest run test/reporting/better-harness-evidence-bundle.test.mjs test/sessions/session-population.test.mjs 42 passed
npm test (root suite) 1771 passed, 6 skipped, 0 failed
npx vitest run test/skills-docs/doc-link-graph.test.mjs (new spec on disk) 8 passed
End-to-end driver on a synthetic Claude home; lead fails with SIMULATED_PROVIDER_CRASH bundle failed, diagnostics bound + leadObserved: false, lead.error.code = SIMULATED_PROVIDER_CRASH
End-to-end driver, healthy lead unchanged: bound, leadObserved: true, lead available

Manual or visual evidence: none (no rendered surface changed).

Risk and Recovery

  • Compatibility and cross-platform impact: additive diagnostics fields
    (leadObserved, errors on conflict); lane envelope shapes unchanged. The
    only behavioral delta is the previously-masked path, which now surfaces the
    original error code instead of the generic one. No OS-specific code touched;
    verified on Linux, exercised through platform-neutral unit contracts.
  • Package, plugin, schema, or generated-file impact: none (no packaged file
    list, schema version, or generated artifact changes).
  • Rollback or recovery path: revert the single commit; unobserved lead lanes
    return to the masked code.
  • Residual risk or unverified boundary: a consumer that string-matches the
    masked code for unobserved lead lanes would now see the original code; no
    such consumer exists in this repository (the code is only produced, never
    parsed). The Windows-specific lead failure in [Bug]: lead lane always fails with SESSION_POPULATION_BINDING_MISMATCH on Claude while sessionEvidence succeeds #164 remains open.

AI Involvement

  • Level: Generated
  • Human review and validation: the root-cause investigation, code, spec, and
    tests were produced by ZCode (GLM 5.3) under the fork owner's direction; the
    fork owner approved the change and this PR text. Red-green verification and
    the suite runs above are the recorded evidence.

Checklist

  • I followed AGENTS.md, CONTRIBUTING.md, and the relevant canonical-owner guidance.
  • The change is focused and does not include unrelated local or generated state.
  • Tests and documentation match the behavior actually delivered.
  • Markdown links were checked when documentation moved or changed (doc-link-graph suite re-run after adding the spec).
  • Cross-platform behavior was considered for Windows, macOS, and Linux.
  • Package/runtime verification was run when shipped files or dependencies changed (no packaged surface touched).
  • User-facing or compatibility changes are recorded in CHANGELOG.md (not edited, following the repository's change-scope rule; happy to add an entry if maintainers prefer).
  • I have the right to contribute this work under the repository's MIT License.

…ismatches

When the lead analyzer failed for its own reason, populationDiagnostics
reconciled lead-side counts from absent data, reported a conflict, and
collectEvidenceBundle replaced the lane with a generic
SESSION_POPULATION_BINDING_MISMATCH envelope. The bundle still failed
closed, but the real failure code never reached the reader, which left
reports like QoderAI#164 undiagnosable.

An unobserved lead lane now keeps its own failure on the lane. Binding
reconciliation skips lead-attributed checks through a new optional
leadObserved flag on validateSessionPopulationBundle (default true, so
existing callers keep their behavior), the diagnostics record
leadObserved and expose the concrete reconciliation errors array on
genuine conflicts, and the fail-closed SESSION_POPULATION_BINDING_MISMATCH
downgrade still applies whenever the conflict was observed against lead
data.

Validated with a new regression test that was verified red on the
pre-fix implementation, the extended conflict assertions, and the full
root suite (1771 tests). Spec:
docs/specs/2026-09-23-164-lead-failure-passthrough.md, story QoderAI#164.

Co-authored-by: ZCode (GLM 5.3) <zcode@z.ai>
@phodal
phodal merged commit 1987bdb into QoderAI:main Sep 23, 2026
3 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants