Skip to content

feat(reporting): project durable status notifications - #1185

Merged
bokelley merged 14 commits into
mainfrom
conductor/reporting-status-notifications-1168c
Sep 24, 2026
Merged

bokelley merged 14 commits into
mainfrom
conductor/reporting-status-notifications-1168c

Conversation

@bokelley

@bokelley bokelley commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Reporting changes and clock deadlines produce durable reporting.status_changed notifications. Handler and projector share captured source boundaries, so committed readability and mismatch cycles survive delayed projection, crashes and restarts. Refs #1168 and adcontextprotocol/adcp#7657.

This update adopts the authenticated AdCP 3.2.0-rc.6 contract. A bilateral waiver applies only to its exact caller/account issue, immutable consumer statement and diagnosed conflict. It removes that public issue and restores underlying seller health while preserving the statement and private audit binding. A later statement or different conflict opens a new occurrence, even when the clock has not advanced. The adopter must obtain and retain bilateral consent before invoking the privileged waiver API; external_ref is inert correlation, not evidence of consent.

The latest append is 81a1c5b3aa8711229f0dedcad06e43a0cfc39871 (tree d1cbc0e0cec26bc4f3c1bade1307cb9ebd4b64a1), with sole parent 00f8b559. Earlier sole-parent history 00f8b559 -> e06a6bc0 -> ee9dd83a -> 6c6899be is preserved. Main remains 0f34c666. The latest correction is two CI/test files, +21/-4; it changes no SDK runtime, DDL, cached schema, generated model, manifest, deadline or required-gate behavior.

Implementation and migration

  • Capture one final projection input per committed source transaction; replay lifecycle intents, semantic fingerprints, checkpoint generations and logical events atomically. Database-clock sweeps use fenced leases. Baselines emit nothing. C events/delivery/activity remain separate from A/B; this slice advertises Core status only, with Managed/Reconciled expiry left to D.
  • Retain terminal waived rows and use a distinct private condition key for each later disagreement. The additive reporting_issue_waiver_bindings table binds the immutable statement and conflict under the same transaction. Rollback leaves no binding. Historical waivers lacking this evidence remain retained but cannot silently suppress a new disagreement. A recovery event follows changed readable health and has no issue IDs when the last impairment clears; unrelated consumer/seller impairments remain.
  • Preserve B's 464-object manifest byte-for-byte. C's required manifest becomes 249 objects: ten new waiver-table objects and one intentional fingerprint change for the snapshot function that now captures waiver bindings; no other fingerprint changes. Readiness comparisons remain strict. Install the additive ledger/status migration before enabling this runtime.
  • Restatement checkpoint access uses the existing transaction-aware memory and PostgreSQL helpers. Previously reviewed 99 assertion hoists and their evaluation-order limits remain unchanged.
  • The required Postgres conformance tests (Postgres 16) aggregate now requires both the core/process matrix and the separate status job, accepting only exact success for each. The three execution jobs retain 15-minute limits; exact A/B fetches retain one minute. Core/process/status dispatch remains file-based and disjoint.
  • Custom-store compatibility delegates to real ledger snapshot/page boundaries rather than inventing positional change ordinals. Continuation cursors retain their incremental lower bound. Intervening-write snapshot rejection remains the inherited fail-closed behavior; no durable snapshot-retention promise is added.
  • Explicit-field serialization covers the generated bundled reporting-delivery variant. This closes the observed default-pruning gap without claiming every notification-config path was leaking defaults.
  • Installed-package smoke now compares the exact canonical error-code vocabulary from the wheel plus the explicit NOT_SUPPORTED SDK extension, replacing a stale pre-rc.6 count. The rc.2 upgrade fixture removes the newly introduced waiver-binding child table before its lifecycle parent, then verifies that the upgrade recreates both tables and the validated foreign key.

Authenticated schema adoption

The rc.6 release commit is f7932355a52f81c64f4c8ff8cc0a36f57677d711; the signed 25,913,096-byte archive has SHA256 2837bcd4ee2d74b326ffff573ee0cdeb87b3920967fb99bc02671c51718834dd. GitHub asset digest, sidecar, raw commit identity and cosign verification agreed. The exact GitHub asset was used while CDN publication was pending; there is no latest-version fallback.

The cache, generated models/stubs, selected signing vectors and package-data pin move from rc.3 to rc.6. Canonical code-generation references resolve from the local authenticated bundle. Runtime schema paths resolve locally even when the referenced document has no $id; unknown HTTP references fail closed. Notification/activity validators use the packaged version. Absolute self-references remain references to the root schema instead of duplicating it. New sales specialisms and sandbox controller scenarios are wired into the existing opt-in framework surfaces.

Validation

Local CPython 3.12.14 / PostgreSQL 16.14 evidence, separate from CI job lifecycles:

  • Latest append: unchanged installed-wheel smoke reproduced the original line-21 failure; the corrected script and representative public API imports both passed against the same retained installed wheel. The unchanged rc.2 fixture reproduced its FK failure; after correcting its historical setup, the complete core-lifecycle module plus exact-waiver regressions passed 32 tests / 0 skipped (20.33s). The production FK remains enforced and is explicitly checked after upgrade. All parsed workflow values outside the smoke script are unchanged.

  • Installed wheel and source distribution, PG migration/restart, exact-waiver recurrence/rollback/legacy controls: 20 passed / 0 skipped (115.44s).

  • Current rc.6 status runtime, all status modules except the historical A/B rolling migration module: 211 passed / 2 existing memory-only skips (291.10s). The two historical source worktrees need 4.62 GB before overhead versus roughly 3 GB locally free; full rolling coverage remains required in fresh CI. Earlier local rolling/catalog results are retained separately, not transferred as current full-suite acceptance.

  • Broad base selection, excluding the separately exercised packaging module: 9,614 passed / 854 skipped / 9 deselected / 1 xfailed, with two sales-specialism map failures (764.79s). Those maps were then corrected; the three affected modules passed 124 tests, including both original failures. The original red run is retained, not relabelled green.

  • Local-reference/code-generation safety checks: 19 passed. Full source typing (1,394 files), strict adopter checks and lint pass. Fresh final cache-to-model reproducibility passed (69.29s); normal commit hooks, including security checks, passed. Generated enum scanner annotation is limited to the canonical pass/fail verdict and preserves its full AST/constants.

  • Strict C/ICU catalog controls agree on all 713 objects (464 B + 249 C). Owned PG clients were checked absent, databases dropped and clusters stopped. No deadline was raised.

Original negative controls established later-statement waiver suppression on unchanged source, the custom-store cursor defects and the bundled serialization gap. Failed intermediate migration/package/loader checks remain retained as diagnostic history. The superseded hidden-degraded candidate is held and not part of this implementation.

Original 00f8b559 CI 35987660734 finished with 16 success / 3 failures: downstream smoke, PG core and the strict PG aggregate. All four Python jobs passed. These local corrections do not relabel that attempt. GitGuardian check 107593985605 / App 46505 was neutral because the PR was too large to scan: it supplied no clean scan, and its zero annotations are not security qualification. Resolving that missing scan remains separate from the source correction.

Fresh exact-head CI, complete review/thread readback, genuine scanner output and bounded source review are required. All results at the original published 6c6899be remain historical: its CI passed 19/19, but Ladon failed orchestration at 61 turns against its 60-turn limit and supplied no complete approval. Its ten open findings and sealed NOT_READY disposition are preserved. No original CI or private local pass transfers to this update.

Compatibility and acceptance limits

  • Rolling controls pin A 17ee407ae3978c8a2bb54437287afbf9dafb8130 and B 0f34c666ac1961e9832fce43ef0ef6937b3c1dde. Pre-integrated A/B rolling compatibility is untested and unclaimed; the migration/release note records this. The older 21bf443e A fingerprint defect is not hidden by selecting a database locale.
  • C's additive boundary trigger still closes A's whole-trigger startup readiness. Drain A workers before enabling C; B subset readiness is the restart path. Default-off compatibility does not permit advertising an unjournaled surface.
  • Extra adopter DDL/RLS/reloptions, verified middleware identity, catalog/per-account read cost, pending retention and uncertain peer receipt remain the disclosed feature limits. PR fix(reporting): cache schema proofs and retain safe receipt diagnostics #1191 memoization is absent.
  • The separate feat(reporting): integrate signed rc6 fixtures and lease progress #1193 complete-summary future-period forecast and later train features still require integration/reconciliation with rc.6. This bounded adoption/waiver correction is not a claim that every rc.6 runtime requirement is qualified.
  • rc.6 all-tool discovery is approximately 6.35 MB for 78 tools; the historical 5 MB bound is not claimed for this expanded inventory. The existing compact list-creatives regression, portable references and deep validation are preserved.
  • Source/repository packaging checks do not constitute published-artifact, main-release, test(interop): qualify Python and TypeScript reporting artifacts across stable and skew #1199, TypeScript or later-train acceptance. No timeout, validation tolerance or scanner finding is waived.

bokelley and others added 5 commits September 16, 2026 18:49
Refs #1168. Add pure status projection, committed-boundary replay, C-only queues, clock sweeps and truthful capability readiness on reviewed #1168B.
`InMemoryReportingLedgerStore` issue-lifecycle writes mutated retained state
before validating a supplied `status_scope`. `_mutation()` only keeps a
rollback copy when notifications are enabled, so under the default
`notifications=False` an invalid scope raised
`ReportingNotificationError("invalid_status_scope")` *after* the move had
already landed: `set_issue_state` left the issue waived, `retire_issue` left it
resolved, and `ensure_issue_opened` left a new occurrence plus a consumed
generation counter. PostgreSQL rolls the statement back, so memory and
PostgreSQL disagreed on a committed outcome.

This is the seam opened by dropping `_dirty_issue`'s
`if self._notification_state is None: return` early return: the scope checks
now run unconditionally, including on default-off stores that have no rollback
copy to fall back on.

Extract the checks into a pure `_resolve_issue_scope()` and clear them before
any caller moves a record, keeping one copy of the rules and preserving
default-off lock cost (no new deepcopy). Regression is parameterized over
memory/PostgreSQL x `notifications=[False, True]`; pre-fix it fails only on
`[memory-False]`, which is why the always-notifications status harness missed
it.

Refs #1168

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`test_c_on_actual_unmigrated_artifact_suppresses_status_only[a]` and
`test_live_reviewed_a_b_workers_never_claim_or_touch_pending_c_queues` failed
in CI while passing locally: both expected reviewed A's pre-C classification
`notifications_ready` and got `a_notifications_closed`. The fetched-source and
module-origin assertions passed, so the artifacts were genuinely A's code.

Root cause is the *database default collation*, not the upgrade. Reviewed A
(`21bf443e`) digests each table's constraints as one aggregate ordered by
`pg_get_constraintdef()` -- a `text` expression sorted under the database
default collation. Against A's own freshly created, pre-C schema, 4 of A's 98
contract entries mismatch on a non-C-collated database
(`reporting_receipt_heads`, `reporting_reconciliation_changes`,
`reporting_reconciliation_records`, `reporting_revisions` constraints), so A
notification readiness is already closed there before C migrates anything. The
`postgres:16` image initdb's `en_US.utf8`; the local run used a C-collated
database, which is the whole delta.

That is A's own frozen behaviour -- C cannot patch a compatibility artifact --
and the rolling assertion is correct, so fix the environment it runs in rather
than the assertion:

- the status job initdb's with `--encoding=UTF8 --lc-collate=C --lc-ctype=C`
  (`--encoding` is required; a bare C locale defaults to SQL_ASCII);
- the rolling fixture asserts that precondition with an actionable message, so
  the gate can never silently measure the locale instead of the upgrade;
- docs state the caveat: "old A stays ready until C" holds only on a
  C-collated database.

Verified by elimination: the full status suite on a non-C (ICU `en-US`)
database fails exactly these two tests and passes the other 200, so C itself is
collation-robust. Both current manifests (453 B/A objects, 239 C objects)
validate on C and non-C databases; a new regression pins them to strictly
per-object keys so readiness can never regress to A's order-dependent
aggregates.

Also pins B/C webhook-attempt positional column parity: the activity union
joins both histories with `SELECT *` UNION ALL, which maps C's values onto B's
column names by position, so a reordered column would silently swap same-typed
fields with no error.

Refs #1168

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit pinned only `pg-reporting-status`, but the general
`pg-conformance` job executes frozen A readiness too — the B activity rolling
tests live in `test_reporting_activity_migration.py`, which is not matched by
`test_reporting_status_*.py`. On the un-pinned `en_US.utf8` cluster that job
failed the same way, with old actual-A subprocesses exiting during
`actual-a-roundtrip` / `actual-a-bootstrap`:

- `test_actual_a_binary_on_b_database_keeps_readiness_and_delivery_identity`
- `test_b_binary_on_actual_a_schema_refuses_activity_without_corrupting_work`

Reproduced locally by collation alone: both fail on an ICU `en-US` database and
the whole job passes on a C-collated one (1,486 passed locally versus CI's
1,484 passed / 2 failed — exactly this pair).

- Pin `pg-conformance`'s cluster to `--encoding=UTF8 --lc-collate=C --lc-ctype=C`,
  matching `pg-reporting-status`.
- Move the precondition into one shared
  `_generation_support.assert_c_collated_rolling_database()` and call it from
  both frozen-artifact fixtures (`actual_a_source`, `actual_sources`), so a
  future workflow drift fails loudly with an actionable message instead of
  silently measuring the locale. The actual-A assertions are unchanged.
- Cross-reference the requirement from both rolling-upgrade doc sections.

Refs #1168

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`pg-conformance` was cancelled at 15m15s on `7795d2bd` — a CI-budget defect,
not a code failure. The test step ran 14m25s before `timeout-minutes: 15`
cancelled it, while exact-head local execution passes all 1,486 in 9m49s.

The 15-minute envelope no longer covers the job. Pinning the cluster to C
collation made the frozen-A rolling tests do their real subprocess work instead
of failing fast on a locale mismatch, and that same budget also has to absorb
checkout, the reviewed-A baseline fetch, Python setup and a `[dev,pg]` install
before the suite starts. The previous run only fit because two of those tests
aborted in seconds.

Raise the job to 30 minutes — already this workflow's matrix-job budget — and
bound the suite step at 25 minutes so a genuine hang is attributed to the step
rather than silently consuming the job. This buys headroom for runner variance
instead of trading a growing serial matrix for flaky timing.

Workflow-only change: the `src/` and `tests/` trees are byte-identical to
`7795d2bd`, so that head's completed local gates remain valid.

Refs #1168

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Base automatically changed from conductor/reporting-webhook-activity-1168b to main September 23, 2026 08:51
Comment thread tests/conformance/reporting/test_reporting_status_boundaries.py Fixed
Comment thread tests/conformance/reporting/test_reporting_status_boundaries.py Fixed
Comment thread tests/conformance/reporting/test_reporting_status_boundaries.py Fixed
Comment thread tests/conformance/reporting/test_reporting_status_boundaries.py Fixed
Comment thread src/adcp/reporting/outbox/activity.py Fixed
Comment thread src/adcp/reporting/outbox/activity.py Fixed
Comment thread examples/reporting_status_notifications.py Fixed
Comment thread src/adcp/reporting/outbox/status.py Fixed
Comment thread src/adcp/reporting/outbox/status.py Fixed
Comment thread src/adcp/reporting/outbox/status.py Fixed
Comment thread .github/workflows/ci.yml
Comment thread tests/test_reporting_ledger.py
Comment thread src/adcp/reporting/ledger/status.py Outdated
Comment thread src/adcp/reporting/ledger/status.py
Comment thread src/adcp/reporting/ledger/status.py Outdated
Comment thread src/adcp/types/base.py Outdated
aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Sep 24, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — rc.3→rc.6 schema-cache adoption plus generated_poc regeneration.

The reviewer verified: waiver-chain cycle protection and fail-closed binding immutability, an additive semver signal, legitimate codegen churn (Action22→23 renumbering with no hand-edits), fail-closed schema_loader ref handlers, and dedicated test coverage for the new waiver branch. No blocking or medium findings were surfaced.

The high_risk flag is true, but every matching entry under schemas/** is (added) — new schema-cache files scaffolded by regeneration, which is normal work and not escalation-worthy on the flag alone (no (deleted)/(modified) sensitive-file findings). gated_paths is false, no author team gate applies, and there are zero medium+ findings. Rows 1–8 do not fire; falls through to row 9.

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean pass, no blocking findings.

The high-risk flag is driven entirely by schemas/** cache files that are (added) — new schema-cache scaffolding, which is inherently low risk (nothing existed to break). Per the high-risk-flag reading rules, added files matching high-risk globs are not escalation-worthy on the flag alone.

Delta since the prior approve is two test/CI files (+21/-4): ci.yml swaps a brittle STANDARD_ERROR_CODES==120 count for a schema-derived set assertion (stronger drift detection, CI gate stays armed), and the rc.2-upgrade conformance test correctly drops the FK-dependent waiver-bindings table before the lifecycle table and validates the restored FK. No SDK runtime, wire, DDL, generated-model, or public-surface change; no semver signal required.

Decision table: no critical/high/medium findings (rows 1, 4, 5, 6, 8 don't fire). gated_paths is false (row 2 n/a). No (deleted) high-risk entries (row 3 n/a). No no-auto-approve team match (row 7 n/a). Falls through to row 9 → approve.

@bokelley
bokelley merged commit 967b6e2 into main Sep 24, 2026
31 checks passed
@bokelley
bokelley deleted the conductor/reporting-status-notifications-1168c branch September 24, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant