Skip to content

fix(harness): gate nightly load runs on transaction inclusion#454

Merged
bdchatham merged 1 commit into
mainfrom
fix/nightly-inclusion-assertion
Jul 7, 2026
Merged

fix(harness): gate nightly load runs on transaction inclusion#454
bdchatham merged 1 commit into
mainfrom
fix/nightly-inclusion-assertion

Conversation

@bdchatham

Copy link
Copy Markdown
Collaborator

Problem

TestBenchmark asserts Job-completion (waitJob) + post-load follower liveness (WaitCaughtUp) — and nothing else. seiload exits 0 on its duration deadline unconditionally (sei-load/main.go deadline path), so a chain that accepts every submission into mempools but includes zero of them in blocks produces a Complete Job, live followers, and a green run. This concealed the mock_balances × TxMempool ready-gate regression (sei-chain#3720) for the entire life of the suite: nightly was green while every block was empty.

Fix

runSeiload records the committed height before creating the Job and, after completion + the existing caught-up checks, sums num_txs over the load window via CometBFT /blockchain. Zero included transactions fails the run.

Fail-closed by construction (peer-review findings applied):

  • Pages reuse the package's getJSONInto idiom (status-gated GET+decode) with bounded per-page retries — one transient blip no longer discards a 14–90-minute run, but a persistently unreachable endpoint still fails.
  • Every page must return exactly the requested number of block_metas — a pruned range or an error envelope decoding to empty fails loud instead of silently contributing zero.
  • Early return once the sum is positive: the healthy case pays for one page; only a real regression walks the full window.
  • /blockchain envelope handling mirrors the SDK's dual-shape tendermintStatus model.

Fine-grained throughput gating deliberately stays in the metrics layer (podMonitor + alerts — an inclusion_latency_count/gas_used alert is being proposed separately); this asserts the floor the harness itself must own.

Verification

  • go vet -tags integration ./test/integration/ clean; gofmt clean.
  • Against the empty-block signature reproduced on harbor (eng-fromtherain, harbor-plt-787): the window sum was 0 across ~1,150 blocks — this gate turns that run red. Against the fixed image (harbor-plt-787-2): 7,968 included — passes on the first pages.
  • Peer-reviewed (systems + idiomatic lenses); both blocking findings (silent error-as-zero pages; no transient retry) addressed as above.

🤖 Generated with Claude Code

TestBenchmark passed on Job-completion + post-load liveness alone. seiload
exits 0 on its duration deadline regardless of outcome, so a chain that
accepts every submission into mempools but includes none of it in blocks
produced a green run — completion and liveness are both satisfiable by an
effectively write-only chain, and a six-week inclusion regression rode
through nightly unseen.

runSeiload now records the committed height before the load starts and,
after completion + caught-up checks, sums num_txs over the window via
/blockchain: zero included transactions fails the run. The walk fails
closed — per-page bounded retries, then loud failure on unreachable pages
or short pages (pruned range / error envelope), since a partial sum that
reads as zero would defeat the gate — and returns early once the sum is
positive so a healthy chain pays for a single page. Fine-grained
throughput gating stays in the metrics layer; this asserts the floor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Jul 7, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Changes are confined to integration test code in seiload_test.go; no production paths, but nightly benchmark signal becomes stricter and may surface previously green empty-block regressions.

Overview
Nightly seiload runs now fail if the chain never includes transactions during the load window, not only when the Job completes and followers stay caught up.

runSeiload records committed height before the Job, then after the existing post-load liveness checks sums num_txs over blocks (start, end] via CometBFT /blockchain. Zero included txs fails the test with a clear error, closing the gap where seiload exits 0 on its duration deadline even when mempools accept traffic but blocks stay empty.

New helpers mustLatestHeight (bounded retries on /status) and includedTxCount (paginated /blockchain, dual JSON envelope like existing status parsing, per-page retries via getJSONInto, strict block_metas count vs requested range, early return once sum > 0) are fail-closed so RPC blips or short/error pages cannot masquerade as “zero inclusion.” Throughput thresholds remain in metrics/alerts; this is the harness floor: included > 0.

Reviewed by Cursor Bugbot for commit 5c90c48. Bugbot is set up for automated code reviews on this repo. Configure here.

@bdchatham bdchatham merged commit 413ab45 into main Jul 7, 2026
5 checks passed
@bdchatham bdchatham deleted the fix/nightly-inclusion-assertion branch July 7, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant