Skip to content

bench: complex multi-system workflow benchmarks (email + spreadsheet + API + multi-app)#266

Merged
abrichr merged 4 commits into
mainfrom
bench/complex-multisystem-benchmarks
Jul 26, 2026
Merged

bench: complex multi-system workflow benchmarks (email + spreadsheet + API + multi-app)#266
abrichr merged 4 commits into
mainfrom
bench/complex-multisystem-benchmarks

Conversation

@abrichr

@abrichr abrichr commented Jul 26, 2026

Copy link
Copy Markdown
Member

Why

The existing benchmark suite is strongest on bounded, single-surface workflows. This PR adds two deterministic, synthetic back-office programs that exercise loops, branches, exception terminals, multiple applications, document/spreadsheet intake, governed API-tier actuation, and persisted-state verification.

It also closes four review gaps in the initial benchmark implementation:

  1. the governed arm now passes the real Standard-profile run gate and derives its runtime authorization from that gate report;
  2. the adjudicators enforce allowed record-level transitions across every non-echo table, catching unrelated inserts, deletes, and in-place edits;
  3. expected state comes from immutable fixture seeds/source bytes rather than the prepared worklist produced by the system under test; and
  4. CI validates the committed n=3 evidence shape and exact headline, including the correct retry denominator.

What changed

AP invoice processing

  • Deterministic vendor email and PDF intake into a five-row worklist.
  • Two fixture applications: ERP plus mail gateway.
  • A real ProgramGraph with a loop, match/exception branch, and discount branch.
  • 32 executed consequential actions on the healthy path.
  • Designed faults: missing PO, ambiguous duplicate invoice, collateral adjacent-row overwrite, and uncertain payment confirmation with idempotent retry suppression.
  • Separate persisted-state reads: read-only SQLite, payments REST read, and direct OUTBOX maildir read.

O2C reconciliation

  • Billing CSV export in, ledger comparison, results CSV write-back.
  • Two fixture applications: billing plus ledger.
  • A real ProgramGraph with a loop, three-way disposition branch, and explicit missing-record halt terminal.
  • 26 executed consequential actions on the healthy path.
  • Designed faults: missing ledger record, ambiguous duplicate, stale snapshot, and phantom workbook write.
  • Separate persisted-state reads: read-only SQLite plus direct CSV re-read.

Governed admission and adjudication

  • benchmark/multiapp-standard.yaml provides the deterministic API-tier certification policy.
  • evaluate_run_gate(..., profile=standard) performs certification, identity/effect coverage, verifier-tier, durability, settling, and manifest checks.
  • build_runtime_authorization(...) binds the successful admission to the exact sealed bundle and runtime inputs.
  • API identity contracts are revalidated by the replayer before request dispatch.
  • Direct-file adjudicators never consume HTTP acknowledgements, UI banners, verifier verdicts, or the prepared worklist. They derive expectations from immutable source fixtures and compare exact allowed business-record transitions. They are separate read paths, not separate services or failure domains.

Evidence

Each benchmark runs n=3 for every arm/scenario cell.

benchmark base runs extra duplicate-attempt checks governed healthy VERIFIED governed silent incorrect success healthy over-halts naive silent incorrect success
ap_invoice 30 (15 per arm) 6 3/3 0/15 0/3 3/3 collateral cases
o2c_recon 30 (15 per arm) 0 3/3 0/15 0/3 3/3 phantom-write cases

All runs are localhost-only, use synthetic data, and make zero model calls. This is deterministic fault-coverage evidence, not a customer reliability or production-incidence claim. UI perception is outside these two benchmarks because every action uses the replayer's API tier.

Validation

  • pytest -q tests/test_ap_invoice_benchmark.py tests/test_o2c_recon_benchmark.py tests/test_release_contract.py: 57 passed.
  • pytest -q tests/test_run_gate.py tests/test_execution_profiles.py tests/test_policy.py tests/test_policy_program.py: 120 passed.
  • ruff check and ruff format --check on all changed benchmark/test code: passed.
  • scripts/check_release_consistency.py: passed; regenerated public-artifacts.json includes both result files and the benchmark policy.
  • scripts/check_consistency.py: passed.
  • scripts/validate_claims.py --check: passed.
  • Committed n=3 result matrices were regenerated from clean commit c1231aa and committed as 9d132e1.

Scope

  • Synthetic fixture code and results only; no customer, PHI, real-EMR, deployment-derived tuning, or private failure corpus data.
  • Benchmark files remain repository evidence and are not included in the PyPI wheel.
  • PR feat(effects): verifier adapter platform (Section 4) #264 remains open, so this branch has intentionally not been rebased yet.

DO NOT MERGE without review.

@abrichr
abrichr force-pushed the bench/complex-multisystem-benchmarks branch from b70754f to 676c79a Compare July 26, 2026 20:26
abrichr and others added 4 commits July 26, 2026 18:16
…+ API + multi-app)

Two new deterministic, synthetic, multi-application benchmarks that close
the complexity gap between the existing single-app linear form-fill
benchmarks (5-15 steps, no branching, no exception paths) and real
back-office work:

- benchmark/ap_invoice: email intake (maildir) + PDF invoice documents +
  ERP 3-way match across two fixture apps (ERP + mail gateway), 32 executed
  actions over a 5-invoice worklist, two branch points (match route,
  discount eligibility), and four designed exception paths (missing PO,
  ambiguous duplicate, collateral adjacent-row overwrite, uncertain payment
  delivery routed to RECONCILIATION_REQUIRED with a suppressed idempotent
  retry via the Section-3 taxonomy from openadapt_flow/transaction.py).
- benchmark/o2c_recon: order-to-cash reconciliation across two fixture apps
  (billing + ledger) with a CSV worklist in and a written-back results CSV
  re-read from disk, 26 executed actions over a 10-order worklist, a 3-way
  branch (match/adjust/missing with an explicit halt terminal), and four
  exception paths (missing record, ambiguous duplicate, stale snapshot
  optimistic-concurrency refusal, phantom results-sheet write).

Both run every consequential write through the real Replayer api tier under
a sealed-bundle standard-profile governed authorization with exact API
identity contracts, per-surface out-of-band effect verification (read-only
SQL, REST oracles, an outbox maildir file oracle, a CSV re-read oracle),
and are judged by an independent direct-file-access ground truth with a
full table-delta audit. Zero model calls; localhost only; all data
synthetic.

Measured (n=3 per scenario/arm, committed in results.json and pinned by
tests): governed silent-incorrect-success 0/30 per benchmark, over-halts
0/30, healthy-path transaction outcome VERIFIED; the naive banner-oracle
arm silently accepts the collateral overwrite and the phantom write-back
(3/3 each). benchmark/BENCHMARK.md gains an honest old-vs-new complexity
table; each new benchmark README states exactly what it does and does not
prove (synthetic, deterministic, api-tier actuation, not a customer
environment). New results.json artifacts are registered in
public-artifacts.json.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM
@abrichr
abrichr force-pushed the bench/complex-multisystem-benchmarks branch from 9d132e1 to 86d19a7 Compare July 26, 2026 22:21
@abrichr
abrichr merged commit ee8dd07 into main Jul 26, 2026
17 checks passed
@abrichr
abrichr deleted the bench/complex-multisystem-benchmarks branch July 26, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant