bench: complex multi-system workflow benchmarks (email + spreadsheet + API + multi-app)

benchmark/BENCHMARK.md

@@ -90,3 +90,45 @@ openadapt-flow benchmark --n-compiled 100 --n-agent 20 --out benchmark/
 Requires `ANTHROPIC_API_KEY` (or `~/.anthropic/api_key`). The agent arm
 costs real money (about $5.43 at list price for
 20 runs when this was generated).
+
+## Workflow complexity across the benchmark suite (2026-07-26)
+
+An honest assessment of what the suite exercised BEFORE the multi-system
+benchmarks landed: mostly single-application, linear, worklist-driven form
+fills of roughly 5-15 recorded steps, with the fault-model studies driving a
+single consequential write. Real back-office work is longer, cross-system,
+document- and email-driven, and exception-heavy. The `ap_invoice` and
+`o2c_recon` benchmarks were added to close that gap; the table below states
+the shape of each benchmark so the difference is not overstated either (the
+new benchmarks actuate through the api tier and do not measure GUI
+perception; see each README).
+
+| benchmark | steps (executed actions) | apps | input/output modalities | branching | exception paths |
+|---|---|---|---|---|---|
+| MockMed encounter (this file) | 6 compiled steps | 1 | browser GUI | none | none |
+| MockLoan disbursement | ~6 compiled steps | 1 | browser GUI | none | none |
+| `openemr_local` registration | ~15-25 UI actions | 1 | browser GUI (real EMR) | none | duplicate-search confirm |
+| `openimis_claims` claim entry | ~10-20 UI actions | 1 | browser GUI (real AGPL app, repo-only) | none | none |
+| `frappe_lending` loan application | ~10 UI actions | 1 | browser GUI (real app) | none | none |
+| `canvas_ladder` / `rdp_ladder` / `citrix_ica_hdx` | 1-3 probe actions | 1 | pixel-only surface | none | qualification refusals |
+| `effect_e2e` / `silent_wrong_action` / `lending_fault_model` | 1 consequential write | 1 | REST + SQLite | none | 10-class fault taxonomy |
+| `effectbench` task pack | 1-3 writes per task | 1 | app API/DB oracles | none | per-task faults |
+| **`ap_invoice`** (new) | **32** | **2** (ERP + mail gateway) | email in (maildir), PDF document, REST API, UI gateway, email out | 2 branch points (match route; discount eligibility) | 4: missing PO, ambiguous duplicate, collateral adjacent-row overwrite, uncertain payment delivery (`RECONCILIATION_REQUIRED` + suppressed retry) |
+| **`o2c_recon`** (new) | **26** | **2** (billing + ledger) | CSV worklist in, CSV results write-back (re-read), REST API, UI gateway, 2 SQLite systems of record | 3-way branch (match / adjust / missing) | 4: missing record (explicit halt terminal), ambiguous duplicate, stale snapshot (optimistic concurrency), phantom file write |
+
+Both new benchmarks run every consequential write through the real
+`Replayer`'s api actuation tier after the real Standard-profile run gate admits
+the sealed bundle and binds a single-use authorization to its exact inputs.
+Every write is verified through a separate persisted-state read (read-only SQL,
+REST oracles, a maildir read, or a CSV re-read) and judged by a direct-file
+adjudicator whose expectations come from immutable source fixtures rather than
+the prepared worklist. This is not an independent service/failure domain. Zero
+model calls; healthy-path
+governed runs classify `VERIFIED` under the Section-3 transaction taxonomy.
+Measured headline over both benchmarks (n=3 per cell; 60 base runs): governed
+silent-incorrect-success 0/30 governed runs and healthy-path over-halts 0/6;
+naive banner-oracle
+silent-incorrect-success 6/30 (the collateral overwrite and the phantom
+write-back classes). Deterministic coverage
+matrix, not a sampled incidence rate. See `benchmark/ap_invoice/README.md`
+and `benchmark/o2c_recon/README.md` for what each does and does not prove.

benchmark/ap_invoice/README.md

@@ -0,0 +1,100 @@
+# AP invoice benchmark (email + PDF + two apps + 3-way match)
+
+Synthetic, deterministic, multi-system workflow benchmark. Not a customer
+environment, not publication evidence of production reliability.
+
+## The workflow (one governed run, healthy path)
+
+A vendor emails an invoice as a PDF attachment; the AP workflow processes the
+whole intake queue end-to-end across TWO separate local applications:
+
+1. **Intake** (deterministic pre-pass, disclosed below): read the request
+   emails from the INBOX maildir, extract each PDF attachment, parse its
+   fields, hash the document, and derive one worklist row per invoice.
+2. **Per invoice** (workflow-program LOOP body, executed by the real
+   `Replayer` through the api actuation tier):
+   - enter the invoice draft in the ERP through the **UI gateway** (this
+     system has no invoice-entry API; `POST /api/invoice/new` answers 405),
+   - attach the PDF's SHA-256 to the record (API),
+   - run the **3-way match** (invoice vs purchase order vs receipts) (API),
+   - **BRANCH** on the match route: a clean match continues; a price
+     mismatch is routed to the **AP exception queue** and the vendor gets a
+     hold notice by email,
+   - **BRANCH** on discount terms: eligible invoices get the early-payment
+     discount applied first; expired terms pay net,
+   - approve for payment (UI gateway), schedule the payment (API), and email
+     the vendor a confirmation through the mail gateway (a second
+     application, maildir-backed).
+3. **Batch completion** record, then a success terminal.
+
+Healthy-path shape: 5 worklist invoices, **32 executed consequential
+actions**, 2 applications, 5 input/output modalities (email in, PDF document,
+REST API, UI gateway, email out), 2 branch points, plus 4 designed exception
+scenarios.
+
+## Arms
+
+- `naive`: demo profile; every write is "verified" only against the
+  application's own painted acknowledgement banner (what a screen-echo
+  automation trusts).
+- `governed`: sealed bundle admitted by the real Standard-profile run gate,
+  with the resulting single-use authorization bound to the exact inputs; an
+  exact API identity contract on every consequential write; effect
+  verification routed per record surface (read-only SQL over the ERP
+  SQLite file, a REST payments oracle, and the OUTBOX **maildir read from
+  disk** for every sent email), a collateral guard on the adjacent grid row,
+  and an at-most-once idempotency ledger.
+
+## Scenarios and measured outcomes (n=3 per cell, deterministic)
+
+| scenario | naive (banner oracle) | governed |
+|---|---|---|
+| `healthy` | completes; `COMPLETED_UNVERIFIED` (never billable) | **`VERIFIED`** (all effects confirmed through separate persisted-state reads; billable) |
+| `missing_po` | safe halt at entry | safe halt; `HALTED_BEFORE_EFFECT`; nothing persisted |
+| `duplicate_invoice` | safe halt at entry | safe halt; `HALTED_BEFORE_EFFECT`; still exactly one invoice |
+| `collateral_approve` | **SILENT WRONG** (adjacent invoice corrupted, banner says success) | **caught**: collateral guard refutes; `RECONCILIATION_REQUIRED` |
+| `payment_confirm_outage` | completes (cannot know the write landed) | `RECONCILIATION_REQUIRED`; retry under the same idempotency key SUPPRESSED (`REJECTED_POLICY`); ground truth: exactly one payment |
+
+Headline (30 base runs: 15 per arm, plus 6 duplicate-attempt checks): governed
+silent-incorrect-success **0/15**, healthy-path over-halts **0/3**, model calls
+**0**; naive
+silent-incorrect-success **3/3** on the collateral scenario. Every run is
+judged through a direct persisted-state read path that opens the SQLite file
+and maildir (no HTTP, banner, or verifier verdict reaches it), derives
+expectations from immutable source-fixture bytes, and enforces the allowed
+record transitions across every non-echo table. This is not a separate service
+or failure domain.
+
+Reproduce: `python -m benchmark.ap_invoice.run --n 3` (localhost only, $0).
+Pinned in CI by `tests/test_ap_invoice_benchmark.py`.
+
+## What this proves, and what it does not
+
+Proves (within a synthetic closed world):
+
+- the engine's Phase-2 workflow-program machinery (loop + guarded branches +
+  explicit exception routing) executes a 30+ step, two-application,
+  email/PDF/spreadsheet-era back-office flow deterministically at $0;
+- the governed contract stack (identity bindings, per-surface persisted-state
+  effect verification including a maildir file oracle, collateral guards,
+  idempotency, the Section-3 transaction taxonomy) yields zero
+  silent-incorrect-successes across the designed fault classes while a
+  banner-echo oracle silently accepts the collateral overwrite;
+- uncertain delivery is routed to `RECONCILIATION_REQUIRED` and is never
+  blind-retried.
+
+Does NOT prove:
+
+- anything about GUI perception. Every consequential action here is actuated
+  through the replayer's **api tier** (the fixture's "UI gateway" models a
+  screen that has no API, but it is still driven over HTTP). Pixel-level
+  resolution, identity-band OCR, and visual drift are measured by the
+  existing MockMed/OpenEMR/canvas/RDP benchmarks, not this one.
+- anything about model-based document understanding. The PDF and email
+  parsing in intake is a deterministic fixture parser over fixture documents;
+  real invoice extraction (OCR/ML) is out of scope and unmeasured.
+- production incidence rates. The fault taxonomy is hand-authored and
+  deterministic (a coverage matrix, not a sampled population); the fixture
+  applications are cooperative and local.
+- customer-environment behavior. All vendors, amounts, and emails are
+  synthetic; no real ERP or mail server is involved.