Skip to content

test(reporting): require full installed lifecycle interop matrix - #1236

Merged
bokelley merged 3 commits into
mainfrom
feat/reporting-full-installed-interop
Sep 28, 2026
Merged

bokelley merged 3 commits into
mainfrom
feat/reporting-full-installed-interop

Conversation

@bokelley

@bokelley bokelley commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

The existing installed-artifact 2×2 check proves Core reconciliation, but it does not exercise Managed Delivery, Reconciled Billing receipts, exact revision reads, or signed notification replay. This adds a second required stage to the same aggregate check. Each of the four published b16/b18 × rc.47/rc.48 cells starts a fresh PostgreSQL-backed installed Python seller and pinned installed TypeScript client, reads the exact delivered revision, submits an accepted receipt, and asserts definitive reconcileReporting() output over MCP HTTP. It then tests signed webhook retry after a simulated post-acceptance crash and checks account activity. A reused database, seller process, output directory, failed stage, or missing evidence fails the aggregate.

The runner records the published wheel and npm identities, signed rc.7 protocol source pin, per-cell results, and retained logs. The webhook receiver replaces only the network socket while the installed SDK's signing, pinning, HTTP encoding, outbox, and verification paths run. The fixture advances its recorded evaluation clock at execution time; its obligation, coverage, digest and expected semantic outcome remain fixed.

Local validation: make lint, make typecheck-all, 54 focused tests (one skipped), Node syntax check, hooks, and the exact isolated-runtime four-cell run pass on current head 39ae07c5. All four receipts were accepted, all JS reconciliations were definitive, and each cell verified two signed webhook attempts after replay. The full-lifecycle seller has a bounded 90-second shutdown grace for its PostgreSQL notification replay; other foundation callers retain the 10-second default. The full make test run on the initial branch content passed (12,109 passed, 2,397 skipped, 9 deselected, one expected failure, 81.54% coverage); the subsequent changes were confined to this runner, its focused test, and CI wiring. Hosted CI passed both installed stages on the preceding head, and fresh current-head CI is running. The combined job feeds the existing required PostgreSQL aggregate.

After this PR merges, the four cells will be rerun on integrated main for #1199 acceptance and #1172’s remaining installed interoperability proof. The separate release-PR rerun required by #1199 remains a later release gate.

@bokelley
bokelley marked this pull request as ready for review September 28, 2026 09:15
Comment thread scripts/ci/reporting_interop/python_full_server.py
aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Sep 28, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — CI test scaffolding for full-lifecycle installed-artifact matrix.

This PR adds a required full-lifecycle installed-artifact 2x2 stage (Managed Delivery, Reconciled Billing receipts, exact-revision reads, signed webhook replay) plus supporting CI reporting scripts and a new test. No public-surface, codegen, discriminated-union, ctx_metadata, or import-layering changes — the mandatory gates do not apply. Design is fail-closed throughout, and largest-file coverage was cited for both >200-line files. CI gates remain armed (no disabled tests, no dropped ruff rules, no blanket type-ignore).

Medium findings

  • scripts/ci/reporting_interop/run_full_installed_artifact_matrix.py — webhook-replay evidence is produced inside the on_shutdown handler bounded by _stop's 10s SIGTERM→SIGKILL window; a slow runner could spuriously red the newly-required gate. This is fail-closed (no false pass), and a single medium finding falls through to approve.

Decision path: no critical/high findings; gated_paths false; high_risk false; no data-loss/schema/infra medium; no prior decision; no team gate; only 1 medium finding (fewer than 3) → row 9 approve. Note: review_decision is REVIEW_REQUIRED but gated_paths is false, so row 2 does not fire.

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean pass, no blocking findings.

This PR touches CI reporting-interop scaffolding (workflow, matrix runners, test harness) — no changes to the public adcp.* surface, generated types, forward-compat layer, or credential handling. The reviewer found no Critical/High/Medium findings.

Delta since prior approve: _stop gains a backwards-compatible grace_seconds parameter (default 10, existing callers unchanged); the new full-lifecycle matrix caller passes grace_seconds=90 to allow the seller's SIGTERM-driven shutdown to complete bounded PostgreSQL notification-replay verification and write evidence before SIGKILL. Shutdown sequence stays bounded and fail-closed (SIGTERM → wait(90) → SIGKILL → wait(5); overrun records a cell failure).

Gate checks:

  • No no-auto-approve team match.
  • gated_paths: false — row 2 does not fire despite review_decision=REVIEW_REQUIRED.
  • high_risk: false — no deletions/modifications on sensitive paths.
  • No medium findings — rows 4, 5, 8 do not apply.
  • Prior decision was approve, so sticky escalation (row 6) does not apply.

None of rows 1–8 fire → row 9: approve.

@bokelley
bokelley merged commit 7549e42 into main Sep 28, 2026
55 checks passed
@bokelley
bokelley deleted the feat/reporting-full-installed-interop branch September 28, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant