bench: complex multi-system workflow benchmarks (email + spreadsheet + API + multi-app)#266
Merged
Merged
Conversation
abrichr
force-pushed
the
bench/complex-multisystem-benchmarks
branch
from
July 26, 2026 20:26
b70754f to
676c79a
Compare
…+ API + multi-app) Two new deterministic, synthetic, multi-application benchmarks that close the complexity gap between the existing single-app linear form-fill benchmarks (5-15 steps, no branching, no exception paths) and real back-office work: - benchmark/ap_invoice: email intake (maildir) + PDF invoice documents + ERP 3-way match across two fixture apps (ERP + mail gateway), 32 executed actions over a 5-invoice worklist, two branch points (match route, discount eligibility), and four designed exception paths (missing PO, ambiguous duplicate, collateral adjacent-row overwrite, uncertain payment delivery routed to RECONCILIATION_REQUIRED with a suppressed idempotent retry via the Section-3 taxonomy from openadapt_flow/transaction.py). - benchmark/o2c_recon: order-to-cash reconciliation across two fixture apps (billing + ledger) with a CSV worklist in and a written-back results CSV re-read from disk, 26 executed actions over a 10-order worklist, a 3-way branch (match/adjust/missing with an explicit halt terminal), and four exception paths (missing record, ambiguous duplicate, stale snapshot optimistic-concurrency refusal, phantom results-sheet write). Both run every consequential write through the real Replayer api tier under a sealed-bundle standard-profile governed authorization with exact API identity contracts, per-surface out-of-band effect verification (read-only SQL, REST oracles, an outbox maildir file oracle, a CSV re-read oracle), and are judged by an independent direct-file-access ground truth with a full table-delta audit. Zero model calls; localhost only; all data synthetic. Measured (n=3 per scenario/arm, committed in results.json and pinned by tests): governed silent-incorrect-success 0/30 per benchmark, over-halts 0/30, healthy-path transaction outcome VERIFIED; the naive banner-oracle arm silently accepts the collateral overwrite and the phantom write-back (3/3 each). benchmark/BENCHMARK.md gains an honest old-vs-new complexity table; each new benchmark README states exactly what it does and does not prove (synthetic, deterministic, api-tier actuation, not a customer environment). New results.json artifacts are registered in public-artifacts.json. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM
abrichr
force-pushed
the
bench/complex-multisystem-benchmarks
branch
from
July 26, 2026 22:21
9d132e1 to
86d19a7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The existing benchmark suite is strongest on bounded, single-surface workflows. This PR adds two deterministic, synthetic back-office programs that exercise loops, branches, exception terminals, multiple applications, document/spreadsheet intake, governed API-tier actuation, and persisted-state verification.
It also closes four review gaps in the initial benchmark implementation:
What changed
AP invoice processing
ProgramGraphwith a loop, match/exception branch, and discount branch.O2C reconciliation
ProgramGraphwith a loop, three-way disposition branch, and explicit missing-record halt terminal.Governed admission and adjudication
benchmark/multiapp-standard.yamlprovides the deterministic API-tier certification policy.evaluate_run_gate(..., profile=standard)performs certification, identity/effect coverage, verifier-tier, durability, settling, and manifest checks.build_runtime_authorization(...)binds the successful admission to the exact sealed bundle and runtime inputs.Evidence
Each benchmark runs
n=3for every arm/scenario cell.ap_invoiceo2c_reconAll runs are localhost-only, use synthetic data, and make zero model calls. This is deterministic fault-coverage evidence, not a customer reliability or production-incidence claim. UI perception is outside these two benchmarks because every action uses the replayer's API tier.
Validation
pytest -q tests/test_ap_invoice_benchmark.py tests/test_o2c_recon_benchmark.py tests/test_release_contract.py: 57 passed.pytest -q tests/test_run_gate.py tests/test_execution_profiles.py tests/test_policy.py tests/test_policy_program.py: 120 passed.ruff checkandruff format --checkon all changed benchmark/test code: passed.scripts/check_release_consistency.py: passed; regeneratedpublic-artifacts.jsonincludes both result files and the benchmark policy.scripts/check_consistency.py: passed.scripts/validate_claims.py --check: passed.c1231aaand committed as9d132e1.Scope
DO NOT MERGE without review.