fix(harness): gate nightly load runs on transaction inclusion#454
Conversation
TestBenchmark passed on Job-completion + post-load liveness alone. seiload exits 0 on its duration deadline regardless of outcome, so a chain that accepts every submission into mempools but includes none of it in blocks produced a green run — completion and liveness are both satisfiable by an effectively write-only chain, and a six-week inclusion regression rode through nightly unseen. runSeiload now records the committed height before the load starts and, after completion + caught-up checks, sums num_txs over the window via /blockchain: zero included transactions fails the run. The walk fails closed — per-page bounded retries, then loud failure on unreachable pages or short pages (pruned range / error envelope), since a partial sum that reads as zero would defeat the gate — and returns early once the sum is positive so a healthy chain pays for a single page. Fine-grained throughput gating stays in the metrics layer; this asserts the floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR SummaryLow Risk Overview
New helpers Reviewed by Cursor Bugbot for commit 5c90c48. Bugbot is set up for automated code reviews on this repo. Configure here. |
Problem
TestBenchmarkasserts Job-completion (waitJob) + post-load follower liveness (WaitCaughtUp) — and nothing else. seiload exits 0 on its duration deadline unconditionally (sei-load/main.godeadline path), so a chain that accepts every submission into mempools but includes zero of them in blocks produces a Complete Job, live followers, and a green run. This concealed the mock_balances × TxMempool ready-gate regression (sei-chain#3720) for the entire life of the suite: nightly was green while every block was empty.Fix
runSeiloadrecords the committed height before creating the Job and, after completion + the existing caught-up checks, sumsnum_txsover the load window via CometBFT/blockchain. Zero included transactions fails the run.Fail-closed by construction (peer-review findings applied):
getJSONIntoidiom (status-gated GET+decode) with bounded per-page retries — one transient blip no longer discards a 14–90-minute run, but a persistently unreachable endpoint still fails.block_metas— a pruned range or an error envelope decoding to empty fails loud instead of silently contributing zero./blockchainenvelope handling mirrors the SDK's dual-shapetendermintStatusmodel.Fine-grained throughput gating deliberately stays in the metrics layer (podMonitor + alerts — an
inclusion_latency_count/gas_usedalert is being proposed separately); this asserts the floor the harness itself must own.Verification
go vet -tags integration ./test/integration/clean;gofmtclean.harbor-plt-787): the window sum was 0 across ~1,150 blocks — this gate turns that run red. Against the fixed image (harbor-plt-787-2): 7,968 included — passes on the first pages.🤖 Generated with Claude Code