feat(inprocess): Tier-2 subprocess harness — operational suites without docker

[bdchatham]

Stacked on #3706. Adds a Tier-2 subprocess backend to the in-process harness: N real seid start processes on loopback, provisioned by the same in-process machinery (genesis, keys, gentx-derived peer mesh) but booted as OS processes so they can be killed, restarted, and upgraded. This brings the docker-bound operational suites onto the harness — with no docker.

What's here

• ControllableNetwork — Stop/Kill/Restart(+WithEnv)/IsRunning, with a background reaper (one mechanism for liveness, zombie reaping, and no-orphan teardown; processes bound to the caller ctx).
• Snapshots — opt-in Options.SnapshotInterval → cosmos state-sync snapshots on disk.
• State-sync — AddStatesyncNode: a late-joining sei-rpc-node that restores from the validators' snapshots (live-valset genesis injection + /status trust anchor).
• Upgrade — Options.GovVotingPeriod + a gov proposal API (SubmitUpgradeProposal/VoteYes/WaitProposalPassed) driving the seid CLI, paired with Restart+WithEnv.
• Runner seam generalized to a backend-agnostic nodeSource; the literal snapshot_operation.yaml / statesync_operation.yaml docker suites run unchanged via the subprocess backend (docker path untouched).

Green, no docker

consensus+EVM, crash-recovery (-race), snapshot, state-sync, upgrade — full subprocess package ~252s (incl. one seid build); gofmt/vet clean.

Impact

The operational suites were a ~28-min docker-bound floor (+ a shared ~9-min cluster build). Measured subprocess vs docker CI baselines: crash-recovery ~24s vs 6.5min (~16×), snapshot ~30s vs 4.5min (~9×), state-sync ~47s vs 5.5min (~7×), upgrade ~124s vs ~12min (~6×). With Tier-1, essentially the entire integration suite can leave docker.

Review

Whole-artifact xreview across ledger rounds 8–10 (systems-engineer dissenter + sei-network-specialist + idiomatic-reviewer), all findings resolved — including the EVM-serve gate root-cause, the signalAndWait/reaper state-corruption landmine, and a self-invalid expedited-gov-period genesis.

🤖 Generated with Claude Code

[cursor]

PR Summary

Medium Risk
Large new test-only infrastructure with process lifecycle and genesis/gov tuning; production node code paths are exercised via seid start but changes are confined to inprocess and integration test packages.

Overview
Adds a Tier-2 subprocess backend (SubprocessNetwork) that reuses shared provision() (genesis, keys, gentx peer mesh) but boots validators as real seid start OS processes instead of in-goroutine nodes—enabling Stop/Kill/Restart with WithEnv (e.g. UPGRADE_VERSION_LIST), on-disk WAL recovery, and upgrade-by-env without docker.

New harness options SnapshotInterval (cosmos snapshots under <home>/snapshots) and GovVotingPeriod (shortened gov deposit/voting in genesis for test proposals). AddStatesyncNode provisions late-joining sei-rpc-node via state-sync (trust block from /status, live validator set injected into genesis). Subprocess app.toml enables per-node gRPC (EVM serve gate), EVM/SeiDB sections, and optional snapshot cadence.

Upgrade orchestration (SubmitUpgradeProposal, VoteYes, WaitProposalPassed) drives gov software-upgrade via the seid CLI. The YAML runner generalizes to a nodeSource + hostExecer, adds WithSubprocessNetwork, SEI_SNAPSHOT_DIR, and moniker lookup for sei-rpc-node. integration_test/runner_subprocess runs bank/snapshot/statesync YAML suites against one shared subprocess cluster in TestMain. In-process Start now calls provision() before claiming the one-network-per-process slot.

^{Reviewed by Cursor Bugbot for commit 5a421b8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.36%. Comparing base (eab7a8f) to head (5a421b8).

Additional details and impacted files

@@                    Coverage Diff                     @@
##           feat/inprocess-migrations    #3709   +/-   ##
==========================================================
  Coverage                      58.36%   58.36%           
==========================================================
  Files                           2186     2186           
  Lines                         178547   178547           
==========================================================
  Hits                          104203   104203           
  Misses                         65086    65086           
  Partials                        9258     9258           

Flag	Coverage Δ
sei-chain-pr	0.00% <ø> (ø)
sei-db	70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Jul 6, 2026, 11:14 PM

[seidroid]

A well-documented Tier-2 subprocess test harness that boots real seid processes for the operational suites without docker. All changes are gated behind the //go:build inprocess tag (test-only, no production impact); no blocking correctness issues, but a few robustness/portability notes remain.

Findings: 0 blocking | 4 non-blocking | 0 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• Orphaned-process risk on hard test termination (Codex, confirmed): spawned seid children run in their own process group (Setpgid: true) with no parent-death signal. The ctx backstop is in-process cancellation, so go test -timeout's SIGQUIT (or any hard kill of the test binary) skips both defers and ctx propagation, leaving each node's group to reparent to PID 1 while still holding fixed RPC/P2P/EVM/gRPC ports — which can flake later runs on shared CI. The Close docstring acknowledges this. Consider setting Pdeathsig: syscall.SIGKILL in SysProcAttr on Linux (subprocess.go startProc) as a defense-in-depth teardown so orphans are killed even when defers don't run.
• Timing coupling in TestSubprocessUpgrade may be flaky under load: upgradeHeight = h + 70 at ~1 block/s must stay ahead of a 30s GovVotingPeriod plus tally. The relationship is documented, but on a slow/contended CI runner block production can lag, narrowing or inverting the margin. Consider computing the height margin from the actual observed block cadence, or gating the proposal-passed check before deriving the target height, to reduce flakiness.
• checkTxCode returns nil when the broadcast response fails to unmarshal, so a malformed/unexpected CLI output silently passes the CheckTx gate; the downstream polling would still eventually time out, but the error would be less specific. Minor — consider surfacing an unmarshal failure.
• REVIEW_GUIDELINES.md is empty, so no repo-specific standards were applied. The Cursor second-opinion review (cursor-review.md) was also empty — that pass produced no output. Codex produced exactly one finding, incorporated above.

[claude]

🟡 Docstrings for Node(i), Nodes(), and Len() (subprocess.go:545-559) say "validator", but all three iterate sn.net.nodes, which AddStatesyncNode appends the non-validator sei-rpc-node to — so post-AddStatesyncNode, Len() returns validators+1 and the accessors can hand back the statesync node. The behavior is intentional (the runner's nodeFor resolves sei-rpc-node by iterating Len(), and VoteYes guards with if n.moniker == statesyncNodeMoniker { continue }) — just the doc lines say "validator" where they should say "node" (or note that the statesync joiner is included).

Extended reasoning...

What the mismatch is

Three doc lines in inprocess/subprocess.go describe validator-scoped semantics:

• L545: // Node returns a handle to the i-th validator.
• L549: // Nodes returns handles to every validator in index order.
• L558: // Len is the validator count.

All three iterate sn.net.nodes. AddStatesyncNode appends the late-joining non-validator sei-rpc-node (Mode = ModeFull) to that same slice near the end of the function (sn.net.nodes = append(sn.net.nodes, n)). After AddStatesyncNode, Len() returns validators + 1 and Node(N)/Nodes() can return the statesync handle.

Why the behavior is (correctly) not what the docs say

The all-nodes semantics is load-bearing:

• integration_test/runner/runner_inprocess.go's nodeFor resolves sei-rpc-node by iterating for i := 0; i < e.net.Len(); i++ { if h := e.net.Node(i); h.Name() == node { return h, nil } }. If Len() were truly validator-only, the statesync suite would break.
• upgrade.go's VoteYes iterates sn.net.nodes and explicitly filters if n.moniker == statesyncNodeMoniker { continue } — direct evidence that the author knows the slice contains the statesync joiner and handles it at the callsite that cares.

So the code is fine; the docstrings are stale.

Step-by-step proof of the mismatch

1. Call StartSubprocess(ctx, Options{Validators: 3, ...}, seidBin). Now len(sn.net.nodes) == 3, all validators. sn.Len() == 3. Docs match reality.
2. Chain reaches the snapshot interval; caller invokes sn.AddStatesyncNode(ctx). Inside, after the join succeeds: sn.net.nodes = append(sn.net.nodes, n) where n.moniker == "sei-rpc-node" and n.tmCfg.Mode == ModeFull (not ModeValidator).
3. sn.Len() now returns 4, but the docstring says "validator count" — off by one.
4. sn.Node(3) returns the statesync handle, but the docstring says "i-th validator".
5. sn.Nodes()[3] likewise — the docstring says "every validator in index order".

The TestMain in runner_subprocess_test.go runs exactly this sequence, so any post-AddStatesyncNode caller reading Len() and treating it as a validator count is silently wrong.

Impact and refutation address

Today no callsite is broken: the upgrade suite doesn't call AddStatesyncNode, the runner's nodeFor intentionally uses Len() for all-nodes lookup, and VoteYes has the moniker guard. The refutation is correct that this is not a behavioral bug — the semantics of "all nodes" are what the runner needs. But that is exactly the argument for fixing the docstring: the docs promise a stronger contract than the code delivers, and a future contributor writing a new validator-only iteration against Len()/Nodes() (say, a delegator-side rewards check) without the statesyncNodeMoniker guard would silently target a nonexistent operator key. The three verifiers all confirmed this as a docstring nit; the refutation objects to filing it as a comment at all, but a doc line that directly contradicts deliberate design is worth a one-line fix.

How to fix

Change "validator" to "node" in the three docstrings, and add one sentence noting that after AddStatesyncNode the collection includes the non-validator statesync joiner. Approximate wording:

// Node returns a handle to the i-th node in the network (validators first,
// then the sei-rpc-node if AddStatesyncNode has been called).
// Nodes returns handles to every node in the network in the order above.
// Len is the node count — validators + 1 after AddStatesyncNode.

No code change needed; this is purely a documentation freshness fix.