Skip to content

feat(reporting): persist buyer receipt submission intents - #1219

Merged
bokelley merged 3 commits into
mainfrom
conductor/reporting-buyer-reconciliation-completion
Sep 26, 2026
Merged

bokelley merged 3 commits into
mainfrom
conductor/reporting-buyer-reconciliation-completion

Conversation

@bokelley

@bokelley bokelley commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Adds adcp.reporting.submissions so a buyer can reserve and replay the same receipt request after a timeout or process crash. A trusted authorization adapter supplies the seller, resolved account and canonical consumer identity. PostgreSQL reserves the complete mixed receipt plan before sending, limits each request to 100 items, and retains ordered successes and per-item failures. Existing checkpoint protocols remain compatible.

Confirmation validates exact response IDs and bodies, preserves the confirmed prefix, and compares the complete persisted state before committing. Bounded caches hold immutable validation evidence; expensive validation runs outside the PostgreSQL scope lock. Schema readiness checks every shipped CHECK constraint as well as keys and indexes.

For rollout, supply an application-owned PostgreSQL pool and call PgReportingSubmissionIntentStore.create_schema() before enabling submissions. After uncertainty, resume with submit_reporting_receipts(..., receipts=None) using the same store and authorizer. Keep the intent tables when rolling back so outstanding requests retain their identities. The memory store is a volatile reference implementation.

The Postgres core CI job now has a 30-minute budget for its full conformance and installed-artifact coverage; the process lane remains at 15 minutes. Test selection and per-case deadlines are preserved. Review cleanup documents closed error handling and strengthens immutability and corruption assertions.

Validation:

  • make lint, make typecheck-all, and make validate-generated passed.
  • Workflow actionlint and ordinary commit hooks passed.
  • Full local Postgres core selection: 2,207 passed, 6 skipped.
  • Actual Python 3.10 installed VCS/sdist wheels, with and without PG: 4 passed.
  • make test with coverage: 11,715 passed, 2,182 skipped, 9 deselected, 1 existing xfail; 81.57% coverage. The standard target ran through the repository reporting harness.

Refs #1167
Refs #1172

@gitguardian

gitguardian Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@bokelley
bokelley force-pushed the conductor/reporting-buyer-reconciliation-completion branch from 003e7d3 to c90274c Compare September 25, 2026 11:19
Comment thread tests/conformance/reporting/test_reporting_buyer_submissions.py Fixed
Comment thread src/adcp/reporting/submissions/store.py
Comment thread src/adcp/reporting/submissions/store.py
Comment thread src/adcp/reporting/submissions/store.py
Comment thread src/adcp/reporting/submissions/submit.py
Comment thread src/adcp/reporting/submissions/submit.py
Comment thread src/adcp/reporting/submissions/models.py Fixed
Comment thread src/adcp/reporting/submissions/pg.py Fixed
Keep the full core selection with a 30-minute job budget and preserve the process lane at 15 minutes. Address buyer submission review comments on redacted errors and test assertions.

@bokelley bokelley left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review verdict: REQUEST_CHANGES (posted as a comment — GitHub blocks requesting changes on your own PR; reviewer session 6bcadd0c)

Reviewed c90274c2 (current PR head). Ran make lint (pass) and the buyer slice locally against PostgreSQL 16.

P2-1 — validate_submission re-validates the entire confirmed history on every read and write

models.py:347-351 calls _restore_confirmation for every already-confirmed chunk each time a submission is validated, and that path re-does full response schema validation. It compounds: _restore_confirmation (models.py:431) rebuilds a kinds map from all plan items per chunk, and _confirmation (models.py:370) re-parses the whole plan to reach one request.

Measured on this head (memory store, full submit):

receipts seconds
100 0.54
500 9.28
1000 36.5
2000 149.3

~4x the time for 2x the items. PgReportingSubmissionIntentStore.confirm runs this inside the FOR UPDATE scope transaction, so scope lock hold grows with history length — that is the part that affects other writers, not just this caller.

P2-2 — schema readiness does not verify the seven shipped CHECK constraints

pg.py:96-116: required covers only the two primary keys and the foreign key. The DDL ships seven CHECKs (reporting_buyer_scope_digest, scope_bound, submission_id, plan_digest, confirmed_digest, plan_bound, confirmed_bound). Because the tables are created with CREATE TABLE IF NOT EXISTS, a pre-existing table that is missing or has weakened any of them passes create_schema — which is exactly the case this check exists to catch. The query also selects conname without the owning table, so an identically named CHECK on the wrong table would satisfy it.

CI — core Postgres lane exceeds its budget

ci.yml:138 sets timeout-minutes: 15 for both lanes. In run 36128723991 the core job was killed at 15m18s having reached 58% (1249 passed, 0 failed — the 38 ERROR lines in that log are PostgreSQL server-log lines from negative-path tests, not pytest errors). The buyer cells added here account for 5m45s of that 15-minute lane, 38.3% of it. There is no green full-CI run on this head; the newer green runs are PR Title Check only.


All three already appear fixed on the local branch (5e1789f1, 0a0fb51f) — that work simply is not pushed, so the PR head is still c90274c2. Push and re-run CI. I verified the slice source in 5e1789f1 is byte-identical to the correction I reviewed in detail, and the bounded byte-keyed validation caches, the snapshot/CAS split that moves validation out of the lock, and the full seven-CHECK readiness all address the two findings above correctly.

Non-blocking: encode_submission returns plan text and plan digest, but confirm discards both (pg.py:323 on the local head) — that decodes and SHA-256s up to 16 MiB per chunk for nothing.

Copy link
Copy Markdown
Contributor Author

Verification of the unpushed fix, for the record: ran the buyer slice (cas, migration, validation, uncertainty, seller, process, submissions) against real PostgreSQL 16 on the local branch head 0a0fb51f — 158 passed in 266.50s, exit 0. Slowest cell 21.65s (test_process_death_preserves_exact_requests_and_confirmed_outcomes[returned]). make lint also passes.

So the only thing standing between this PR and a clean re-review is the push.

@bokelley
bokelley force-pushed the conductor/reporting-buyer-reconciliation-completion branch from c90274c to 0a0fb51 Compare September 26, 2026 15:03
Comment thread tests/conformance/reporting/test_reporting_buyer_submission_cas.py
Comment thread tests/conformance/reporting/test_reporting_buyer_submission_cas.py
Comment thread tests/conformance/reporting/test_reporting_buyer_submission_cas.py
Comment thread tests/conformance/reporting/test_reporting_buyer_submission_cas.py
@bokelley
bokelley marked this pull request as ready for review September 26, 2026 16:19

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve.

Fully additive PR introducing the new adcp.reporting.submissions package (durable buyer receipt-submission intents) plus a CI timeout bump. No blocking or Medium findings.

Checks performed:

  • No no-auto-approve team match; author gate not triggered.
  • gated_paths: false, so the hard approval gate does not apply (review_decision REVIEW_REQUIRED is irrelevant when gated_paths is false).
  • high_risk: false; no deleted/modified sensitive-path escalation triggers.
  • Reviewer verified the two largest source files (models.py, pg.py): confirmation ID/body matching, byte-and-digest immutability round-trips, idempotent-replay normalization, and complete-snapshot CAS all fail closed; closed error codes leak no driver/SQL/wire detail.
  • Semver signal compliant (feat, additive only), type-system import layering intact, no credentials in ctx_metadata, transport timeout/cancellation handling present, ci.yml gate armed.

No findings of any severity. Rows 1–8 do not fire; falls through to row 9 → approve.

@bokelley
bokelley merged commit e77a401 into main Sep 26, 2026
53 checks passed
@bokelley
bokelley deleted the conductor/reporting-buyer-reconciliation-completion branch September 26, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant