From bc061ac0d71d2629380d40c033604818970233c4 Mon Sep 17 00:00:00 2001 From: bdchatham Date: Tue, 7 Jul 2026 14:10:23 -0700 Subject: [PATCH 1/2] docs(runbooks): add EVM gas-vs-execution-time benchmarking runbook Gives an engineer's agent a self-contained procedure for testing whether Sei's EVM gas schedule tracks real compute: personal single-validator + RPC-follower chain, differential-loop bytecode isolation, the privileged eBPF-attach gate on harbor's eng- namespaces, and concrete probe design against real seid/go-ethereum symbols. Co-Authored-By: Claude Sonnet 5 --- .agent/runbooks/README.md | 1 + .../benchmarking-evm-gas-vs-execution-time.md | 204 ++++++++++++++++++ 2 files changed, 205 insertions(+) create mode 100644 .agent/runbooks/benchmarking-evm-gas-vs-execution-time.md diff --git a/.agent/runbooks/README.md b/.agent/runbooks/README.md index eab850e..652ea04 100644 --- a/.agent/runbooks/README.md +++ b/.agent/runbooks/README.md @@ -32,6 +32,7 @@ Agents fetch this `README.md` to discover what's available, then `WebFetch` the | [`operating-archive-node-byov.md`](operating-archive-node-byov.md) | Required volume contents, PV/PVC spec, SeiNode/SeiNetwork spec, controller validation surface, and EBS-swap cutover sequence for archive nodes using the bring-your-own-volume (`dataVolume.import`) path. | Bringing up an archive node from a pre-populated EBS; swapping the underlying volume of an existing archive PV; debugging `ImportPVCReady=False`; receipt-store pruning concerns. | | [`migrating-validator-to-byo-secrets.md`](migrating-validator-to-byo-secrets.md) | Cutting a live validator from a legacy host onto the platform carrying its consensus identity via Secrets (`signingKey`/`nodeKey`): what migrates, SeiNetwork spec, controller validation surface, the stop-before-start double-sign discipline + layered equivocation defenses, cutover/rollback sequence, and dry-run gotchas. | Migrating an existing validator (e.g. arctic-1 node-19) off EC2 onto K8s; any cutover where a consensus key changes hosts; understanding the `replicas:1` CEL guard or the double-sign alerts. | | [`validating-flatkv-memiavl-parity-via-sharded-replay.md`](validating-flatkv-memiavl-parity-via-sharded-replay.md) | flatKV↔memIAVL storage-engine parity validation by differential historical replay: the flatKV+memIAVL replay-pair topology (same binary, same snapshot, blocks from a shared archive), the correctness gates (compare pair-not-archive; verify migration complete so flatKV reads are genuine; `historical_replay` build for pre-v6.5 txs), the seictl shadow comparator, result aggregation + Notion report, and the fan-out to 50+ shards. | Standing up a flatKV-vs-memIAVL correctness validation on harbor; driving a sharded replay campaign; debugging why a replay node is stuck or a comparison reads all-indeterminate/vacuous; understanding `migrate_evm` vs `memiavl_only`/`evm_migrated`/`flatkv_only` read routing. | +| [`benchmarking-evm-gas-vs-execution-time.md`](benchmarking-evm-gas-vs-execution-time.md) | Testing whether Sei's EVM gas schedule tracks real compute: standing up a personal single-validator + RPC-follower chain, the confounds that invalidate a naive comparison (governance-mutable `SSTORE` gas, precompiles, OCC re-execution), the differential-loop bytecode-crafting technique for per-opcode isolation, the privileged-attach gate for eBPF on harbor's `eng-` namespaces (PSS posture, scoped `kubectl debug --target=`, dedicated-node pinning), and concrete probe design against real `seid`/go-ethereum symbols (why per-opcode uprobing doesn't work, why `offcputime`/`profile` beat raw uprobe timing). | Designing a gas-vs-compute benchmark; deciding whether an opcode's gas cost is a fair compute proxy before trusting a result; attaching any eBPF probe to a harbor pod; resolving `seid`'s real symbol names before writing a probe. | ## Adding a new runbook diff --git a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md new file mode 100644 index 0000000..6362a8f --- /dev/null +++ b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md @@ -0,0 +1,204 @@ +# Benchmarking EVM Gas vs. Real Execution Time on a Personal Harbor Chain + +**Audience:** engineers (and their agents) investigating whether Sei's EVM gas schedule actually tracks real compute cost — "does `gasUsed` correspond to wall-clock execution time, or has it drifted from what the interpreter actually does." +**Scope:** standing up a personal single-validator + RPC-follower chain in your `eng-` harbor namespace, crafting opcode-isolated bytecode, the privileged-attach gate for eBPF instrumentation, concrete probe design against real `seid`/go-ethereum symbols, the submission/correlation protocol, and the confounds that will produce a false result if ignored. **No sei-chain source changes anywhere in this procedure** — everything here is external client traffic + kernel/process-level observation of a binary you don't touch. +**Not in scope:** *why* Sei's gas schedule has the specific values it does (that's a governance/design question — if this benchmarking effort produces a case for changing a gas param, capture *that* as a `/design` doc, not here). Modifying gas costs, submitting the result on-chain, or running any of this against a shared/long-lived/production chain — this is a throwaway personal chain, full stop. Full onboarding to harbor (see the `/harbor-dev` skill and its `references/preflight.md`). + +> **Read this whole section before running anything.** Three independent traps make the naive version of this experiment **silently wrong**: (1) at least one opcode's gas cost (`SSTORE`) is a **governance-mutable Sei chain param that already diverges from upstream go-ethereum** — using it as your "does gas match compute" baseline tests a value that was hand-tuned away from compute cost *by design*, not by accident; (2) per-opcode timing via a uprobe **is not achievable** on this binary — the interpreter's opcode dispatch is an indirect call with no per-opcode symbol, and Go's runtime makes `uretprobe`-based entry/return timing actively hazardous; (3) attaching any eBPF probe to a `seid` pod on harbor requires a **privileged kernel capability grant on a shared EKS cluster** — nothing in harbor's current `eng-` namespace configuration technically blocks this, but that is an unhardened gap, not a green light. §1, §2, and §4 exist to keep you out of these three holes. + +--- + +## 0. Substitutions used throughout + +| Placeholder | Meaning | +|---|---| +| `` | your harbor engineer alias; namespace is `eng-`. | +| `` | a fresh chain-id you choose for this experiment (see the `/harbor-dev` naming rule — tie it to a Linear ticket or your alias, never reuse). Also the `SeiNetwork` name. | +| `` | `-rpc-0` — the single RPC follower (§3). | +| `` / `` | the sei-chain image you're benchmarking; resolved per `references/image-resolution.md` in the `/harbor-dev` skill. | +| `` | the single EVM opcode under test for a given run (§4). | +| `` | the PID of the `seid` process inside ``'s pod — resolve fresh each pod restart. | +| `` | your local `sei-chain` clone, used to resolve real symbol names before writing any probe (§6). | + +--- + +## 1. Mental model — what this can and cannot prove (read first) + +The question is simple: for a given opcode, does the gas charged scale with the real wall-clock cost of executing it? The honest answer requires knowing, going in, which parts of the gas schedule were **derived from measured compute** and which were **set by other means** (governance, upstream inheritance, a flat precompile fee) — comparing against the latter and reporting "gas doesn't match compute" would be reporting a known, intentional divergence as if it were a finding. + +**What's a fair target vs. a confound:** + +| Category | Examples | Why | +|---|---|---| +| **Primary target — interpreter-local, no state backend** | `KECCAK256`, pure arithmetic (`ADD`/`MUL`/`PUSH`/`DUP`/…) | Executes entirely in the go-ethereum interpreter loop with no I/O; the cleanest "compute-bound opcode" available. | +| **Secondary target — real, but control for it** | `SLOAD`/`SSTORE` | **`SSTORE`'s set-gas is a governance-mutable Sei chain param (`SeiSstoreSetGasEIP2200`, `x/evm/types/params.go:43`) that already diverges from the upstream go-ethereum constant** — you are not testing "does gas match compute," you're testing "does gas match a number governance picked." If you test it anyway (worth doing as its own category), you must also control for IAVL-tree-depth / cache-warmth / cold-vs-warm-slot variance (Sei's state backend is memIAVL/flatKV, not an MPT) — separate cold-slot and warm-slot runs, and don't compare across them. | +| **Separate, expected-to-diverge category — do not average into "opcode gas"** | Sei precompiles (staking, distribution, oracle, etc. — see `/evm` skill's precompile kit for the current address table) | Precompile "execution" is a Go-native Cosmos-module call outside the interpreter entirely; its gas is a governance-tunable flat-ish fee structurally decoupled from whatever compute it triggers. Report it, but don't blend it with interpreted-opcode results. | + +**Two more confounds that apply to every run, regardless of opcode:** + +- **Per-tx fixed overhead** (signature verification, the Cosmos ante handler, the EVM↔Cosmos StateDB bridge, usei↔wei decimal conversion) is real wall-clock time attributable to *no* opcode. It doesn't disappear — it dilutes your measurement unless your loop count is high enough to make it a small fraction of total time. +- **Optimistic concurrent execution (OCC).** Sei's parallel tx execution can serialize and **re-execute** a transaction on hot-slot contention. A re-executed tx's wall-clock time is not a clean single-pass measurement. §5's one-tx-per-block sequential-submission protocol sidesteps this by construction — don't skip it to save time. + +**Per-opcode granularity is achieved by construction, not by probing.** The interpreter's opcode dispatch (`(*EVMInterpreter).Run`, `core/vm/interpreter.go:168`) reads the opcode as a loop-local byte and calls it through an indirect function-table lookup (`operation.execute(...)` at interpreter.go:322, table lookup at :255) — there is no stable per-opcode symbol or argument to uprobe. Instead: craft bytecode dominated by one repeated opcode (§4), measure the *transaction's* wall-clock time, divide by iteration count. `per-tx time ÷ opcode count ≈ per-opcode time`. This is the disciplined substitute the `/ebpf` skill's overhead-bound and Go-uprobe-hazard rules point you toward — don't try to fight the interpreter's dispatch shape. + +--- + +## 2. The privileged-attach gate — read before running `kubectl debug` + +Any eBPF uprobe, `offcputime`, or CPU `profile` run against `seid` requires a **privileged kernel capability grant** (`CAP_BPF`/`CAP_SYS_ADMIN`/`CAP_PERFMON`) on a pod in **harbor — a shared EKS cluster** running other engineers' chains and the `sei-k8s-controller` itself. This is root-on-node. The `/ebpf` skill's guardrail is unconditional here: **the gate is on the attach mechanism, not on how low-stakes the target is.** A personal, zero-stake, single-validator throwaway chain does not waive it — get explicit human + security sign-off before attaching anything, even a one-off `bpftrace -e` line. + +**What's actually true about `eng-` today (verified against the platform repo, not assumed):** + +- `eng-` namespaces carry **no Pod Security Standard label** (`clusters/harbor/engineers/base/namespace.yaml`) and harbor has **no cluster-wide PSS default** either — so the effective PSS level is Kubernetes' own default, **`privileged`, enforced nowhere**. Nothing in-namespace blocks a privileged pod or a `kubectl debug --profile=sysadmin` ephemeral container today. +- Compare this to **prod**: `clusters/prod/walle/podcomposition-policy.yaml` explicitly blocks ephemeral containers as a `kubectl debug` escape vector, on top of a `restricted` PSS label. Harbor's `eng-` has no equivalent. **This is an unhardened gap, not a sanctioned path** — it's the kind of thing platform closes the moment someone notices, which would break this runbook retroactively if you built a habit of relying on it silently. +- **Shared-node blast radius is real independent of PSS.** Engineer workloads bin-pack onto harbor's untainted default Karpenter NodePool — a `hostPID` pod sees *every* process on that node, including other engineers' `seid` processes and the Cilium agent, not just your own. + +**Do this, in order, before attaching anything:** + +1. **Get sign-off first.** Surface the ask (what you're attaching, to what, on which pod, for how long) to the platform/security owner before running anything privileged. Don't proceed on "PSS doesn't block it" alone. +2. **Pin to a dedicated node.** Add a `nodeSelector`/toleration so ``'s pod (and nothing else) lands on a node you're not sharing with other tenants, for the duration of the experiment. This contains the blast radius even though PSS wouldn't otherwise stop it. +3. **Scope the capability grant to the minimum, not `hostPID`.** Use `kubectl debug --target=seid -it -0 --image= --profile=sysadmin` — `--target=` joins **only that pod's** PID namespace, not the node's. Where possible, request `CAP_BPF`+`CAP_PERFMON` rather than the broader `sysadmin` profile — a same-pod uprobe/off-CPU/profile target doesn't need full `CAP_SYS_ADMIN` or `hostPID`. +4. **If this becomes recurring or a second engineer needs it**, that's the signal to stop improvising per-run and instead open a platform PR for a proper dedicated profiling namespace with its own restricted-by-default PSS + a narrow, auditable exception — mirror `clusters/harbor/admission/tenant-hostname-policy.yaml`'s shape for the exception and `clusters/dev/sei-omnigent/sandbox-namespace.yaml` for the relaxed-but-still-restricted namespace pattern. `clusters/harbor/chaos-mesh/chaos-mesh.yaml` is harbor's existing precedent for an elevated-privilege, namespace-scoped, opt-in-annotation-gated workload — same shape, different capability. **This is a platform decision, not something to self-approve** — this runbook stops at "here's the gap and the mitigation," not "here's the policy." + +--- + +## 3. Standing up the personal chain + +You need **two nodes**, not one: a validator's EVM JSON-RPC surface is disabled (`ModeValidator disables EVM` — see the `/harbor-dev` skill's `references/harbor-cluster.md`), so you can't submit or query EVM transactions against the validator directly, even though it does execute every transaction for consensus. Stand up a single-validator genesis chain plus one RPC follower via the `/harbor-dev` skill's render→PR→Flux flow (`seictl network`/`seictl node` — see its `references/ephemeral-chain-flow.md` and `references/seictl-cli.md` for the full mechanics; only the commands unique to this use case are below): + +```sh +# Single-validator genesis chain — --replicas 1 is the load-bearing flag (default is 4) +seictl network apply --preset genesis-chain \ + --chain-id --image --replicas 1 \ + -n eng- --dry-run + +# One RPC follower, peered to the genesis chain — this is your submission + query + instrumentation target +seictl node apply --preset rpc \ + --chain-id --network \ + -n eng- --dry-run +``` + +Render → PR against `harbor-engineering-workspace` → merge → `seictl network watch --until=Ready` → `seictl node watch --until=Running` — this is the standard `/harbor-dev` flow, not repeated here. + +**Instrument ``, not the validator.** You're already pointing `seid tx evm ... --evm-rpc=` at it for submission and `eth_getTransactionReceipt` for `gasUsed` — keeping the probe target and the query target in the same pod means one `kubectl debug --target=seid` session covers everything, with no cross-node clock correlation to worry about. The validator executes the same state transition for consensus, but there's no operational reason to split targets. + +--- + +## 4. Crafting opcode-isolated bytecode + +**The differential-loop technique** isolates a single opcode's marginal cost without needing per-opcode instrumentation: write two bytecode variants that are identical except for the opcode under test. + +- **Variant A (opcode-under-test):** a loop body of `JUMPDEST`, the target `` (with whatever operands it needs pre-pushed), then the loop-decrement/`JUMPI` back to `JUMPDEST`. +- **Variant B (baseline):** identical loop scaffold, but the target opcode replaced with a cheap no-op pair (e.g. `PUSH1 0x00` / `POP`) of known, separately-measured cost. + +`(wall-clock(A) − wall-clock(B)) ÷ iteration count ≈ per-opcode marginal wall-clock cost`, and the same subtraction on `gasUsed` gives you the marginal *gas* cost to compare it against. This cancels the loop-overhead opcodes (`PUSH`/`DUP`/`SUB`/`JUMPI`) and the per-tx fixed overhead (§1) from both sides symmetrically. + +Deploy and invoke via the `seid` CLI directly against `` (no sei-chain code involved — these are standard `seid tx evm` subcommands, see `seid tx evm --help` for the full flag set): + +```sh +seid tx evm deploy /path/to/-loop.bin \ + --from= --chain-id= --evm-rpc=http://.eng-.svc:8545 \ + --gas-limit= --gas-fee-cap= + +seid tx evm call-contract \ + --from= --value=0 --chain-id= --evm-rpc=http://.eng-.svc:8545 +``` + +Pick a large-but-fixed iteration count up front (large enough that fixed overhead is a small fraction of total gas/time; the EVM code-size cap bounds how much you can unroll instead of loop, so loop rather than unroll for counts beyond a few hundred). Run each variant multiple times and take the distribution, not a single sample — wall-clock has real variance from GC pauses, Pebble compaction, and scheduling noise that gas cost is blind to. + +--- + +## 5. Submission + correlation protocol + +**Submit exactly one transaction, wait for its block to finalize and its receipt to confirm, before submitting the next.** This is the single biggest simplification available to you: with sequential, no-overlap submission, a block's execution window contains exactly the transaction(s) you just sent, in an order you know. It also directly sidesteps the OCC re-execution confound from §1 — nothing is contending for the same storage slot across concurrent transactions if there's only ever one in flight. + +Log client-side, per transaction: `{submit_ts, tx_hash, block_height, gasUsed (from the receipt)}`. Join this against whatever the probe emits (§6) by time-ordering — you do not need to decode transaction data inside the probe itself. + +**One real gotcha:** `StateTransition.Execute` (the function you're timing, §6) runs more than once per submitted transaction — once during `eth_estimateGas`/`CheckTx` simulation, and again at actual `FinalizeBlock` delivery. Expect **2–3 invocations per tx**, not one. Disambiguate by taking the invocation whose timestamp falls inside the block-production window for the block your tx landed in (correlate against the block's timestamp from `/status` or the block header), or simply the *last* invocation observed before the receipt becomes queryable. + +--- + +## 6. Instrumentation — cheapest and least-privileged first + +Don't reach for eBPF as step one. Try these in order; stop at the first tier that gives you a usable signal. + +**Tier 0 — free, no probe at all.** `gasUsed` from the receipt and `eth_estimateGas` are already there. Before instrumenting anything, record the live gas parameters and software version so the eventual comparison means something: confirm the value of any governance-mutable param you're testing against (e.g. `SeiSstoreSetGasEIP2200`, `x/evm/types/params.go:43`) via whatever query path `seid query evm --help` / `seid query params --help` currently exposes on the image you're running — don't assume a CLI shape that may have changed. Record the resolved image digest alongside it. + +**Tier 1 — Cosmos SDK's own pprof, no elevated privileges.** The `rpc` preset's `spec.overrides`/`--set` surface can set `network.rpc.pprof_laddr` in the rendered `config.toml` (see the preset's example override in `/harbor-dev`'s `references/seictl-cli.md`). Port-forward and pull a CPU profile with plain `go tool pprof` — zero Kubernetes privilege needed beyond what you already have in your own namespace. **Verify empirically whether this surfaces EVM-interpreter frames at all** — `net/http/pprof` in this codebase is only explicitly imported by the `ethreplay` subcommand, not the running validator/full-node path, so whether `pprof_laddr` gives you a populated `net/http/pprof` mux at all (via a transitive cosmos-sdk import) needs to be confirmed against the actual running pod, not assumed from either agent's read of `seid`'s own `main.go`. If it's empty or unhelpful, move to Tier 2 — this was a cheap thing to rule out first. + +**Tier 2 — eBPF (the gated tier; §2 must be cleared first).** + +Resolve real symbols before writing any probe — do not assume upstream go-ethereum symbol names. `sei-chain`'s `go.mod` pulls go-ethereum via a `replace` directive to a Sei fork; **Go's `replace` preserves the original import path in compiled symbol names**, so the binary's symbols are still `github.com/ethereum/go-ethereum/...`, not a `sei-protocol/...` path. Confirm on the actual binary, not from source alone: + +```sh +kubectl -n eng- debug --target=seid -it -0 --image= --profile=sysadmin -- \ + go tool nm /proc//root/usr/bin/seid | grep -E 'StateTransition\)\.Execute|EVMInterpreter\)\.Run' +``` + +The two real boundaries (from ``, confirm line numbers haven't drifted before citing them to anyone): + +- **Per-tx:** `github.com/ethereum/go-ethereum/core.(*StateTransition).Execute` (`core/state_transition.go`) — Sei's `x/evm/keeper/msg_server.go` (`applyEVMMessage`) calls this directly; don't probe the upstream `ApplyMessage` wrapper, Sei bypasses it. +- **Per-opcode dispatch loop (not individually probeable — see §1):** `github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run`. + +**Prefer `offcputime` and a PID-filtered on-CPU `profile` over raw entry/return timing.** This is the actual highest-value signal for your question — whether `Execute`'s wall-clock time is CPU-bound (interpreter work, i.e. what gas is supposed to meter) or blocked on Pebble/state I/O (which gas categorically does not meter) is a more honest answer than a single duration number, and both are map-aggregated (near-zero overhead at the ~1 tx/block rate this protocol produces): + +``` +# on-CPU attribution, PID-filtered +bpftrace -e 'profile:hz:49 /pid == / { @[ustack] = count(); }' + +# off-CPU (blocked) time, PID-filtered — needs the offcputime tool, not a bare bpftrace one-liner +offcputime -p -f 30 +``` + +**Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only, on this throwaway chain, with two hazards you must state plainly if you use it:** (1) Go's stack-copying runtime and `uretprobe`'s return-address rewrite can interact badly — this is a known divergence, not a hypothetical; (2) a goroutine can migrate OS threads mid-call on a blocking I/O read, which breaks a `tid`-keyed entry/return pairing even at concurrency 1. The `/ebpf` skill's "never on a validator/consensus-critical node" guardrail is weaker here specifically *because* this is a zero-stake personal chain where a crash costs you a re-deploy, not a slashing event — but that weakening applies only to the crash-tolerance judgment call, not to the §2 attach-privilege gate, which is unconditional. + +``` +uprobe:/proc//root/usr/bin/seid:"github.com/ethereum/go-ethereum/core.(*StateTransition).Execute" +{ @start[tid] = nsecs; } +uretprobe:/proc//root/usr/bin/seid:"github.com/ethereum/go-ethereum/core.(*StateTransition).Execute" +/@start[tid]/ +{ @dur = hist(nsecs - @start[tid]); delete(@start[tid]); } +``` + +--- + +## 7. Reading results + +For each opcode under test, you should have: the differential wall-clock time (§4) at the tier-2 granularity you reached, the differential `gasUsed`, and the live gas parameter value that produced it (§6 tier 0). Report them as a ratio (time-per-gas-unit) per opcode, not a single aggregate — the whole point is to see *which* opcodes diverge, and by how much, not to produce one number for "the EVM." + +**Before concluding anything, re-check against §1:** if the opcode you measured is `SSTORE`, you already know the gas side is a governance-set number, not a compute-derived one — a divergence there confirms a known fact, it isn't a finding. A divergence on `KECCAK256` or pure arithmetic is the genuine signal this experiment exists to find. A precompile's numbers belong in their own table, never blended with interpreted-opcode results. + +--- + +## 8. Failure modes (quick reference) + +| Symptom | Cause | Fix | +|---|---|---| +| `seid tx evm` calls succeed but you can't read `gasUsed` / can't reach `--evm-rpc` | pointed at the validator, not the RPC follower | Validators don't serve EVM JSON-RPC (§3); target ``'s `evmJsonRpc` endpoint. | +| Probe never fires / symbol not found | assumed upstream go-ethereum's import path instead of resolving on the actual binary | Re-run the `go tool nm` resolution in §6 against `/proc//root/usr/bin/seid` — the `replace` directive keeps the upstream path, but don't take that on faith either. | +| Wall-clock measurements wildly inconsistent run-to-run | overlapping submissions triggering OCC re-execution (§1), or measuring a re-execution as if it were a clean single pass | Confirm strictly sequential one-tx-per-block submission (§5); check for more than one `Execute` invocation per tx and disambiguate by block-production window. | +| `seid` pod crashes or misbehaves after attaching a `uretprobe` | the Go-runtime/`uretprobe` interaction hazard (§6) | Expected risk on this tier — redeploy the personal chain (it's throwaway) and prefer `offcputime`/`profile` for anything beyond a one-off rehearsal. | +| "Gas doesn't match compute" conclusion on `SSTORE` | tested a governance-mutable param as if it were compute-derived (§1) | Reframe as "confirms the known SSTORE gas/upstream divergence," not a new finding; re-run on `KECCAK256`/arithmetic for the real test. | +| Can't get sign-off / told to stop before attaching anything privileged | §2's gate working as intended | This is the platform team's call, not something to route around — escalate the ask, don't self-approve a scoped-down version of the same attach. | + +--- + +## 9. Pre-flight checklist + +- [ ] Personal single-validator + RPC-follower chain up via `/harbor-dev` (§3); confirmed you're pointed at ``, not the validator, for submission/query. +- [ ] Recorded the live gas parameter values + image digest you're testing against (§6 tier 0) — the comparison is meaningless without this. +- [ ] Differential bytecode pair built for the target opcode (§4), with a fixed, high-enough iteration count. +- [ ] Sequential one-tx-per-block submission harness ready, logging `{submit_ts, tx_hash, block_height, gasUsed}` (§5). +- [ ] Tier 0 and Tier 1 (pprof) tried and found insufficient before reaching for eBPF. +- [ ] **Sign-off obtained** for any privileged attach, workload pinned to a dedicated node, capabilities scoped to `--target=seid` rather than `hostPID` (§2) — before running anything in §6 tier 2. +- [ ] Symbols resolved against the actual running binary, not assumed from source reading alone (§6). + +--- + +## References + +- Sei-EVM gas/precompile specifics: `/evm` skill, `references/sei-evm-profile.md`, `kit-evm-parity-gas.md`, `kit-sei-precompiles.md`; `x/evm/types/params.go` in `sei-chain` for the governance-mutable param set. +- eBPF method, overhead-bounding, and the Go-uprobe/uretprobe divergences: `/ebpf` skill, `references/pack-perf-methodology.md`. +- Harbor chain spin-up mechanics: `/harbor-dev` skill, `references/ephemeral-chain-flow.md`, `references/seictl-cli.md`, `references/harbor-cluster.md`. +- Harbor `eng-` Pod Security / admission posture: `sei-protocol/platform` — `clusters/harbor/engineers/base/namespace.yaml`, `clusters/prod/walle/{namespace,podcomposition-policy}.yaml` (the posture to eventually mirror), `clusters/dev/sei-omnigent/sandbox-namespace.yaml`, `clusters/harbor/chaos-mesh/chaos-mesh.yaml`, `clusters/harbor/admission/tenant-hostname-policy.yaml`. +- Execution entry points: `sei-chain` — `x/evm/keeper/msg_server.go` (`applyEVMMessage`, `EVMTransaction`); the vendored go-ethereum fork's `core/state_transition.go` (`StateTransition.Execute`) and `core/vm/interpreter.go` (`EVMInterpreter.Run`) — resolve the fork version via `go.mod`, don't hardcode it here. From 6140dd358e86531ca84c272ff40411e145052ae4 Mon Sep 17 00:00:00 2001 From: bdchatham Date: Tue, 7 Jul 2026 14:53:14 -0700 Subject: [PATCH 2/2] docs(runbooks): rework gas-vs-execution-time runbook around real replayer/archive nodes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the from-scratch synthetic genesis chain with the actual infra the gas-repricing project needs: a replayer-mode SeiNode restoring state at Y and replaying forward through block X (functionality 1), and an archive-backed path for arbitrary bytecode against state at Y (functionality 2). Confirms hardware parity with prod validators is already automatic via NodepoolForMode/nodepoolName/nodepoolArchive, cited against the controller source and both clusters' Karpenter config. Peer-reviewed with the systems-engineer agent, which traced Sei's OCC scheduler in sei-cosmos and found that per-tx timing correlation inside a replayed multi-tx block breaks under parallel/re-executed scheduling unless OCC is disabled for the measurement run (new §3a). Also fixed: Path A can't get eBPF timing off the shared archive node (not your node to instrument), and block-window detection must be continuous/post-hoc rather than reactive given ~25ms block-application windows during replay. Co-Authored-By: Claude Sonnet 5 --- .agent/runbooks/README.md | 2 +- .../benchmarking-evm-gas-vs-execution-time.md | 218 ++++++++++-------- 2 files changed, 126 insertions(+), 94 deletions(-) diff --git a/.agent/runbooks/README.md b/.agent/runbooks/README.md index 652ea04..673a442 100644 --- a/.agent/runbooks/README.md +++ b/.agent/runbooks/README.md @@ -32,7 +32,7 @@ Agents fetch this `README.md` to discover what's available, then `WebFetch` the | [`operating-archive-node-byov.md`](operating-archive-node-byov.md) | Required volume contents, PV/PVC spec, SeiNode/SeiNetwork spec, controller validation surface, and EBS-swap cutover sequence for archive nodes using the bring-your-own-volume (`dataVolume.import`) path. | Bringing up an archive node from a pre-populated EBS; swapping the underlying volume of an existing archive PV; debugging `ImportPVCReady=False`; receipt-store pruning concerns. | | [`migrating-validator-to-byo-secrets.md`](migrating-validator-to-byo-secrets.md) | Cutting a live validator from a legacy host onto the platform carrying its consensus identity via Secrets (`signingKey`/`nodeKey`): what migrates, SeiNetwork spec, controller validation surface, the stop-before-start double-sign discipline + layered equivocation defenses, cutover/rollback sequence, and dry-run gotchas. | Migrating an existing validator (e.g. arctic-1 node-19) off EC2 onto K8s; any cutover where a consensus key changes hosts; understanding the `replicas:1` CEL guard or the double-sign alerts. | | [`validating-flatkv-memiavl-parity-via-sharded-replay.md`](validating-flatkv-memiavl-parity-via-sharded-replay.md) | flatKV↔memIAVL storage-engine parity validation by differential historical replay: the flatKV+memIAVL replay-pair topology (same binary, same snapshot, blocks from a shared archive), the correctness gates (compare pair-not-archive; verify migration complete so flatKV reads are genuine; `historical_replay` build for pre-v6.5 txs), the seictl shadow comparator, result aggregation + Notion report, and the fan-out to 50+ shards. | Standing up a flatKV-vs-memIAVL correctness validation on harbor; driving a sharded replay campaign; debugging why a replay node is stuck or a comparison reads all-indeterminate/vacuous; understanding `migrate_evm` vs `memiavl_only`/`evm_migrated`/`flatkv_only` read routing. | -| [`benchmarking-evm-gas-vs-execution-time.md`](benchmarking-evm-gas-vs-execution-time.md) | Testing whether Sei's EVM gas schedule tracks real compute: standing up a personal single-validator + RPC-follower chain, the confounds that invalidate a naive comparison (governance-mutable `SSTORE` gas, precompiles, OCC re-execution), the differential-loop bytecode-crafting technique for per-opcode isolation, the privileged-attach gate for eBPF on harbor's `eng-` namespaces (PSS posture, scoped `kubectl debug --target=`, dedicated-node pinning), and concrete probe design against real `seid`/go-ethereum symbols (why per-opcode uprobing doesn't work, why `offcputime`/`profile` beat raw uprobe timing). | Designing a gas-vs-compute benchmark; deciding whether an opcode's gas cost is a fair compute proxy before trusting a result; attaching any eBPF probe to a harbor pod; resolving `seid`'s real symbol names before writing a probe. | +| [`benchmarking-evm-gas-vs-execution-time.md`](benchmarking-evm-gas-vs-execution-time.md) | Testing whether Sei's EVM gas schedule tracks real compute, for both (1) replaying a real historical block against restored state and (2) executing arbitrary bytecode against a specific historical state: why hardware parity with prod validators is automatic (`NodepoolForMode`, no engineer config needed), the replayer-node mechanism for functionality 1 and its real gap (gas is captured, execution time is not), the unverified `eth_call`-state-override dependency for functionality 2 and its private-writable-fork fallback, the confounds that invalidate a naive comparison (governance-mutable `SSTORE` gas, precompiles, OCC re-execution/determinism), the differential-loop bytecode-crafting technique for per-opcode isolation, the privileged-attach gate for eBPF on harbor's `eng-` namespaces (PSS posture, scoped `kubectl debug --target=`), and concrete probe design against real `seid`/go-ethereum symbols (why per-opcode uprobing doesn't work, why `offcputime`/`profile` beat raw uprobe timing). | Designing a gas-vs-compute benchmark; replaying a historical block for timing rather than storage-parity; deciding whether an opcode's gas cost is a fair compute proxy before trusting a result; attaching any eBPF probe to a harbor pod; resolving `seid`'s real symbol names before writing a probe. | ## Adding a new runbook diff --git a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md index 6362a8f..cbceee7 100644 --- a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md +++ b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md @@ -1,10 +1,10 @@ -# Benchmarking EVM Gas vs. Real Execution Time on a Personal Harbor Chain +# Benchmarking EVM Gas vs. Real Execution Time on Sei -**Audience:** engineers (and their agents) investigating whether Sei's EVM gas schedule actually tracks real compute cost — "does `gasUsed` correspond to wall-clock execution time, or has it drifted from what the interpreter actually does." -**Scope:** standing up a personal single-validator + RPC-follower chain in your `eng-` harbor namespace, crafting opcode-isolated bytecode, the privileged-attach gate for eBPF instrumentation, concrete probe design against real `seid`/go-ethereum symbols, the submission/correlation protocol, and the confounds that will produce a false result if ignored. **No sei-chain source changes anywhere in this procedure** — everything here is external client traffic + kernel/process-level observation of a binary you don't touch. -**Not in scope:** *why* Sei's gas schedule has the specific values it does (that's a governance/design question — if this benchmarking effort produces a case for changing a gas param, capture *that* as a `/design` doc, not here). Modifying gas costs, submitting the result on-chain, or running any of this against a shared/long-lived/production chain — this is a throwaway personal chain, full stop. Full onboarding to harbor (see the `/harbor-dev` skill and its `references/preflight.md`). +**Audience:** engineers (and their agents) on the gas-repricing project investigating whether Sei's EVM gas schedule tracks real compute cost. Covers both target functionalities: (1) replaying a real historical block against restored state and measuring gas + execution time per tx, and (2) executing arbitrary bytecode against a specific historical state and measuring the same. +**Scope:** standing up a replayer node (functionality 1) and an archive-backed environment (functionality 2) on harbor with confirmed prod-matching hardware, the privileged-attach gate for eBPF instrumentation, concrete probe design against real `seid`/go-ethereum symbols, and the confounds that will produce a false result if ignored. **No sei-chain source changes anywhere in this procedure** — everything here is external client traffic + kernel/process-level observation of a binary you don't touch. +**Not in scope:** *why* Sei's gas schedule has the specific values it does (a governance/design question — capture a resulting change proposal as a `/design` doc, not here). Running any of this against a shared/long-lived/production chain, or against the canonical shared archive node in a way that mutates its state. -> **Read this whole section before running anything.** Three independent traps make the naive version of this experiment **silently wrong**: (1) at least one opcode's gas cost (`SSTORE`) is a **governance-mutable Sei chain param that already diverges from upstream go-ethereum** — using it as your "does gas match compute" baseline tests a value that was hand-tuned away from compute cost *by design*, not by accident; (2) per-opcode timing via a uprobe **is not achievable** on this binary — the interpreter's opcode dispatch is an indirect call with no per-opcode symbol, and Go's runtime makes `uretprobe`-based entry/return timing actively hazardous; (3) attaching any eBPF probe to a `seid` pod on harbor requires a **privileged kernel capability grant on a shared EKS cluster** — nothing in harbor's current `eng-` namespace configuration technically blocks this, but that is an unhardened gap, not a green light. §1, §2, and §4 exist to keep you out of these three holes. +> **Read this whole section before running anything.** Four things make the naive version of this experiment **silently wrong**: (1) at least one opcode's gas cost (`SSTORE`) is a **governance-mutable Sei chain param that already diverges from upstream go-ethereum** — using it as your "does gas match compute" baseline tests a value that was hand-tuned away from compute cost *by design*; (2) per-opcode timing via a uprobe **is not achievable** on this binary — the interpreter's opcode dispatch is an indirect call with no per-opcode symbol; (3) attaching any eBPF probe to a `seid` pod on harbor requires a **privileged kernel capability grant on a shared EKS cluster** — nothing in harbor's current namespace configuration technically blocks this, but that is an unhardened gap, not a green light; (4) **arbitrary-bytecode-against-historical-state (functionality 2) has an unverified RPC dependency** — confirm it before you build a campaign on top of an assumption. §1, §4, §5, and §6 exist to keep you out of these holes. --- @@ -13,144 +13,166 @@ | Placeholder | Meaning | |---|---| | `` | your harbor engineer alias; namespace is `eng-`. | -| `` | a fresh chain-id you choose for this experiment (see the `/harbor-dev` naming rule — tie it to a Linear ticket or your alias, never reuse). Also the `SeiNetwork` name. | -| `` | `-rpc-0` — the single RPC follower (§3). | -| `` / `` | the sei-chain image you're benchmarking; resolved per `references/image-resolution.md` in the `/harbor-dev` skill. | -| `` | the single EVM opcode under test for a given run (§4). | -| `` | the PID of the `seid` process inside ``'s pod — resolve fresh each pod restart. | -| `` | your local `sei-chain` clone, used to resolve real symbol names before writing any probe (§6). | +| `` | the historical height whose post-block state you're restoring/executing against. | +| `` | the historical block (functionality 1 only) whose transactions you're re-executing and timing; `X > Y`. | +| `` | the `SeiNode` in `replayer` mode restoring to `` and replaying forward (§3). | +| `` | the archive-mode `SeiNode` or existing shared archive infra used for functionality 2 (§4). | +| `` | the single EVM opcode under test for a given functionality-2 run (§5). | +| `` | the PID of the `seid` process inside the target pod — resolve fresh each pod restart. | +| `` | your local `sei-chain` clone, used to resolve real symbol names before writing any probe (§7). | --- ## 1. Mental model — what this can and cannot prove (read first) -The question is simple: for a given opcode, does the gas charged scale with the real wall-clock cost of executing it? The honest answer requires knowing, going in, which parts of the gas schedule were **derived from measured compute** and which were **set by other means** (governance, upstream inheritance, a flat precompile fee) — comparing against the latter and reporting "gas doesn't match compute" would be reporting a known, intentional divergence as if it were a finding. +The question: for a given opcode, does the gas charged scale with the real wall-clock cost of executing it? The honest answer requires knowing which parts of the gas schedule were **derived from measured compute** vs. **set by other means** (governance, upstream inheritance, a flat precompile fee). **What's a fair target vs. a confound:** | Category | Examples | Why | |---|---|---| -| **Primary target — interpreter-local, no state backend** | `KECCAK256`, pure arithmetic (`ADD`/`MUL`/`PUSH`/`DUP`/…) | Executes entirely in the go-ethereum interpreter loop with no I/O; the cleanest "compute-bound opcode" available. | -| **Secondary target — real, but control for it** | `SLOAD`/`SSTORE` | **`SSTORE`'s set-gas is a governance-mutable Sei chain param (`SeiSstoreSetGasEIP2200`, `x/evm/types/params.go:43`) that already diverges from the upstream go-ethereum constant** — you are not testing "does gas match compute," you're testing "does gas match a number governance picked." If you test it anyway (worth doing as its own category), you must also control for IAVL-tree-depth / cache-warmth / cold-vs-warm-slot variance (Sei's state backend is memIAVL/flatKV, not an MPT) — separate cold-slot and warm-slot runs, and don't compare across them. | -| **Separate, expected-to-diverge category — do not average into "opcode gas"** | Sei precompiles (staking, distribution, oracle, etc. — see `/evm` skill's precompile kit for the current address table) | Precompile "execution" is a Go-native Cosmos-module call outside the interpreter entirely; its gas is a governance-tunable flat-ish fee structurally decoupled from whatever compute it triggers. Report it, but don't blend it with interpreted-opcode results. | +| **Primary target — interpreter-local, no state backend** | `KECCAK256`, pure arithmetic (`ADD`/`MUL`/`PUSH`/`DUP`/…) | Executes entirely in the interpreter loop with no I/O — the cleanest compute-bound signal. | +| **Secondary target — real, but control for it** | `SLOAD`/`SSTORE` | **`SSTORE`'s set-gas is a governance-mutable Sei chain param (`SeiSstoreSetGasEIP2200`, `x/evm/types/params.go:43`, code default `20000`, governance-set higher on mainnet — query it live, never hard-code either value)** — a divergence here confirms a known, intentional gap, not a finding. If you test it anyway, separate cold-slot and warm-slot runs; Sei's state backend is memIAVL/flatKV, not an MPT, and cache-warmth dominates. | +| **Separate, expected-to-diverge category — never blend into "opcode gas"** | Sei precompiles (staking, distribution, oracle, etc. — see `/evm` skill's `kit-sei-precompiles.md`) | Precompile execution is a Go-native Cosmos-module call outside the interpreter entirely; gas is a governance-tunable, structurally decoupled fee. | -**Two more confounds that apply to every run, regardless of opcode:** +**Confounds that apply regardless of functionality:** -- **Per-tx fixed overhead** (signature verification, the Cosmos ante handler, the EVM↔Cosmos StateDB bridge, usei↔wei decimal conversion) is real wall-clock time attributable to *no* opcode. It doesn't disappear — it dilutes your measurement unless your loop count is high enough to make it a small fraction of total time. -- **Optimistic concurrent execution (OCC).** Sei's parallel tx execution can serialize and **re-execute** a transaction on hot-slot contention. A re-executed tx's wall-clock time is not a clean single-pass measurement. §5's one-tx-per-block sequential-submission protocol sidesteps this by construction — don't skip it to save time. +- **Per-tx fixed overhead** (signature verification, the Cosmos ante handler, the EVM↔Cosmos StateDB bridge) is real wall-clock time attributable to *no* opcode — it dilutes a measurement unless your sample size / loop count makes it a small fraction of the total. +- **Optimistic concurrent execution (OCC) — this breaks naive per-tx correlation for functionality 1.** Sei applies a block's transactions through a **parallel scheduler**, not a serial loop: `DeliverTxBatch` → `tasks.NewScheduler(app.ConcurrencyWorkers(), …)` → `ProcessAll` spawns up to `ConcurrencyWorkers` goroutines (localnode default `concurrency-workers = 4`, `occ-enabled = true`; verified `app/abci.go:243`, `sei-cosmos/tasks/scheduler.go`), and re-executes any tx whose read/write set conflicts. **This same block-delivery path runs during replay** — a replayed historical block is applied by the OCC scheduler, not single-stepped. Consequences you must design around: + - **Functionality 2** (synthetic bytecode, §5) avoids all of this by construction — one tx per block means one task, no contention, no re-execution (§5). + - **Functionality 1** (replaying a real multi-tx block, §3) does **not** get clean per-tx timing for free: multiple `StateTransition.Execute` calls run concurrently on different OS threads, and a conflicting tx runs `Execute` more than once. An ordinal-by-call-count bracket (which is what §7's `tid`-keyed uprobe gives you) therefore maps to *neither* a specific tx *nor* a single execution — it silently produces **block-level aggregate timing, not per-tx timing**. See §3's correlation subsection for the fix (disable OCC for the measurement run) and its limits. + - **Determinism:** even setting correlation aside, OCC re-execution is not a deterministic function of the tx list alone (incarnation counts depend on scheduling), so repeated replays of the same `` can differ. If timing looks bimodal/inconsistent across repeated replays, suspect re-execution before suspecting your probe. -**Per-opcode granularity is achieved by construction, not by probing.** The interpreter's opcode dispatch (`(*EVMInterpreter).Run`, `core/vm/interpreter.go:168`) reads the opcode as a loop-local byte and calls it through an indirect function-table lookup (`operation.execute(...)` at interpreter.go:322, table lookup at :255) — there is no stable per-opcode symbol or argument to uprobe. Instead: craft bytecode dominated by one repeated opcode (§4), measure the *transaction's* wall-clock time, divide by iteration count. `per-tx time ÷ opcode count ≈ per-opcode time`. This is the disciplined substitute the `/ebpf` skill's overhead-bound and Go-uprobe-hazard rules point you toward — don't try to fight the interpreter's dispatch shape. +--- + +## 2. Hardware parity — already automatic, verify it rather than configure it + +**You do not need to request specific hardware.** `sei-k8s-controller` picks the Karpenter NodePool per node mode automatically: `internal/platform/platform.go`'s `NodepoolForMode()` — archive-mode nodes get `c.NodepoolArchive`, every other mode (including `replayer`) gets `c.NodepoolName` — and wires the matching toleration + required node affinity onto the pod spec (`internal/task/bootstrap_resources.go:203-227`, mirrored in `internal/noderesource/noderesource.go:454-479`). Both `clusters/harbor/sei-k8s-controller/controller-config.yaml` and `clusters/prod/sei-k8s-controller/controller-config.yaml` (in `sei-protocol/platform`) set the **identical** values: `nodepoolName: sei-node`, `nodepoolArchive: sei-archive`, `tolerationKey: sei.io/workload`. Harbor's `clusters/harbor/default/nodepool-{default,archive}.yaml` carry the **same** `r6i`/`r7i` instance-family, Nitro-hypervisor, on-demand requirements as `clusters/prod/default/nodepool-{default,archive}.yaml`. Net: a replayer or archive `SeiNode` in your `eng-` namespace lands on the same instance class as its prod counterpart, with zero engineer-side configuration. + +**Verify it anyway before trusting a timing result** — a Karpenter requirements drift between clusters would silently invalidate every measurement: + +```sh +NODE=$(kubectl -n eng- get pod -0 -o jsonpath='{.spec.nodeName}') +kubectl get node "$NODE" -o jsonpath='{.metadata.labels.node\.kubernetes\.io/instance-type}{"\n"}{.metadata.labels.karpenter\.sh/nodepool}{"\n"}' +``` + +Expect an `r6i.*`/`r7i.*` instance type and `nodepool=sei-node` (or `sei-archive` for an archive-mode node). If you see anything else, halt and check for a stale `controller-config.yaml` divergence between harbor and prod before proceeding — don't benchmark on it. --- -## 2. The privileged-attach gate — read before running `kubectl debug` +## 3. Functionality 1 — replay block `` against state after ``, measure gas + time -Any eBPF uprobe, `offcputime`, or CPU `profile` run against `seid` requires a **privileged kernel capability grant** (`CAP_BPF`/`CAP_SYS_ADMIN`/`CAP_PERFMON`) on a pod in **harbor — a shared EKS cluster** running other engineers' chains and the `sei-k8s-controller` itself. This is root-on-node. The `/ebpf` skill's guardrail is unconditional here: **the gate is on the attach mechanism, not on how low-stakes the target is.** A personal, zero-stake, single-validator throwaway chain does not waive it — get explicit human + security sign-off before attaching anything, even a one-off `bpftrace -e` line. +**The state-restore-and-forward-replay mechanism is real, documented tooling** — this is the same machinery behind `validating-flatkv-memiavl-parity-via-sharded-replay.md`, used here for timing instead of storage-parity comparison. You do **not** need that runbook's flatKV-vs-memIAVL differential pair; stand up a single replayer engine. -**What's actually true about `eng-` today (verified against the platform repo, not assumed):** +1. **Render a `replayer`-mode `SeiNode`** targeting ``: `spec.replayer.snapshot.s3.targetHeight: ` (per `validating-flatkv-memiavl-parity-via-sharded-replay.md:148`) — the seictl sidecar restores the highest snapshot `<= ` and syncs forward from there, peering a shared full-history archive over p2p as its block source. Render via the `/harbor-dev` skill's PR-based flow into `harbor-engineering-workspace`, same as any other `SeiNode`. +2. **Let it replay forward past ``.** There is no documented flag to halt exactly at a target height — the tooling (and its shadow-comparator sibling) is built for continuous mass replay (~40 blocks/s/node, paged in 100-block chunks for the parity use case). You don't need a halt, but note that you **cannot reactively bracket the `` window** — RPC height/logs report `` only *after* it commits (§3a). Instead, attach the probe *continuously across a replay span containing ``* and correlate afterward by height/tx; poll `.status`/RPC height or tail logs only to confirm `` has been applied and to bound the span. Letting the node continue replaying past `` is harmless. +3. **Gas is already there per-tx** — the replayed node has genuinely applied ``'s transactions; read `gasUsed` per tx the same way you would from any node (`seid query evm tx ` / `eth_getTransactionReceipt`), no comparator needed for a single-block read. +4. **Execution time is the actual gap.** Neither this replay mechanism nor its comparator captures timing anywhere — the comparator's three layers cover AppHash/`LastResultsHash`, per-tx `code`/`gasUsed`/`log`/`events`, and post-state storage/balance/code/nonce; there is no timing field in any of them. This is exactly what §7's eBPF instrumentation is for, bracketed around the height-`` transition you identified in step 2. -- `eng-` namespaces carry **no Pod Security Standard label** (`clusters/harbor/engineers/base/namespace.yaml`) and harbor has **no cluster-wide PSS default** either — so the effective PSS level is Kubernetes' own default, **`privileged`, enforced nowhere**. Nothing in-namespace blocks a privileged pod or a `kubectl debug --profile=sysadmin` ephemeral container today. -- Compare this to **prod**: `clusters/prod/walle/podcomposition-policy.yaml` explicitly blocks ephemeral containers as a `kubectl debug` escape vector, on top of a `restricted` PSS label. Harbor's `eng-` has no equivalent. **This is an unhardened gap, not a sanctioned path** — it's the kind of thing platform closes the moment someone notices, which would break this runbook retroactively if you built a habit of relying on it silently. -- **Shared-node blast radius is real independent of PSS.** Engineer workloads bin-pack onto harbor's untainted default Karpenter NodePool — a `hostPID` pod sees *every* process on that node, including other engineers' `seid` processes and the Cilium agent, not just your own. +### 3a. Per-tx correlation inside a multi-tx block — the real constraint (read before trusting any per-tx number) -**Do this, in order, before attaching anything:** +Bracketing `StateTransition.Execute` with a `tid`-keyed uprobe/uretprobe (§7) gives **block-level aggregate timing, not per-tx timing**, on any block with more than one tx — because the OCC scheduler runs several `Execute` calls concurrently on different threads and re-runs conflicting txs (§1). Two separate problems: (a) *which* invocation is *which* tx — `Execute()` takes **no arguments** (receiver-only; the tx lives in the `st.msg *Message` field), so there is no arg to read a hash from, and ordinal-by-call-count is ambiguous under concurrency + re-execution; (b) a `tid`-keyed entry/return pair can be split across threads by goroutine migration, worse under OCC's higher scheduling pressure. + +**Fix — serialize the measurement run by disabling OCC.** Render the replayer with `occ-enabled = false` and `concurrency-workers = 1` (app.toml keys; wired via `baseapp.SetOccEnabled`/`SetConcurrencyWorkers`, verified). The block-delivery path then takes its non-OCC serial branch (`!ctx.IsOCCEnabled()`) — one `Execute` at a time, in tx order, no conflict re-execution — so ordinal-by-call-count *does* map cleanly to the Nth tx in ``. **This does not corrupt the measurement you care about:** gas is charged per-tx independently of the parallel schedule, and a tx's own interpreter/state work is the same instruction stream serial or parallel; you are removing cross-tx cache/scheduler interference, not changing the tx's compute. Confirm the SeiNode/seictl spec actually exposes these app.toml keys before relying on it — **VERIFY**; if it doesn't, this is a `/harbor-dev` spec gap to close, not a workaround to skip. + - **If OCC cannot be disabled** on the replayer: state plainly that functionality 1 yields **block-level aggregate `Execute` time only** (sum over the block's txs + any re-executions), correlatable to the block's *total* `gasUsed`, not per-tx. Per-tx then requires a compiled CO-RE tool that reads `st.msg` from the receiver (pointer + resolved field offsets) and keys by goroutine id, not tid — fragile, version-pinned to the struct layout, and out of scope for a bpftrace one-liner. Don't claim per-tx numbers you can't attribute. + - **Determinism bonus:** serial application also removes the OCC re-execution non-determinism (§1), so repeated replays of `` are directly comparable. -1. **Get sign-off first.** Surface the ask (what you're attaching, to what, on which pod, for how long) to the platform/security owner before running anything privileged. Don't proceed on "PSS doesn't block it" alone. -2. **Pin to a dedicated node.** Add a `nodeSelector`/toleration so ``'s pod (and nothing else) lands on a node you're not sharing with other tenants, for the duration of the experiment. This contains the blast radius even though PSS wouldn't otherwise stop it. -3. **Scope the capability grant to the minimum, not `hostPID`.** Use `kubectl debug --target=seid -it -0 --image= --profile=sysadmin` — `--target=` joins **only that pod's** PID namespace, not the node's. Where possible, request `CAP_BPF`+`CAP_PERFMON` rather than the broader `sysadmin` profile — a same-pod uprobe/off-CPU/profile target doesn't need full `CAP_SYS_ADMIN` or `hostPID`. -4. **If this becomes recurring or a second engineer needs it**, that's the signal to stop improvising per-run and instead open a platform PR for a proper dedicated profiling namespace with its own restricted-by-default PSS + a narrow, auditable exception — mirror `clusters/harbor/admission/tenant-hostname-policy.yaml`'s shape for the exception and `clusters/dev/sei-omnigent/sandbox-namespace.yaml` for the relaxed-but-still-restricted namespace pattern. `clusters/harbor/chaos-mesh/chaos-mesh.yaml` is harbor's existing precedent for an elevated-privilege, namespace-scoped, opt-in-annotation-gated workload — same shape, different capability. **This is a platform decision, not something to self-approve** — this runbook stops at "here's the gap and the mitigation," not "here's the policy." +**Window detection is post-hoc, not reactive.** You cannot *react* to entering block ``: at ~40 blocks/s a block's application window is ~25 ms, and RPC height / logs only report `` *after* it has committed — by the time you observe it, the `Execute` calls are done. So the probe must be **attached and aggregating continuously across a replay span that contains ``**, then filtered afterward by height/tx. Under the disable-OCC serial path this is clean (aggregate per-tx in order, join to the tx list of `` after); with a fast continuous replay you also want to slow the arrival into `` (small `concurrency-workers`, or throttle the block source) so the window isn't lost in sampling noise. --- -## 3. Standing up the personal chain +## 4. Functionality 2 — execute arbitrary bytecode against state after ``, measure gas + time -You need **two nodes**, not one: a validator's EVM JSON-RPC surface is disabled (`ModeValidator disables EVM` — see the `/harbor-dev` skill's `references/harbor-cluster.md`), so you can't submit or query EVM transactions against the validator directly, even though it does execute every transaction for consensus. Stand up a single-validator genesis chain plus one RPC follower via the `/harbor-dev` skill's render→PR→Flux flow (`seictl network`/`seictl node` — see its `references/ephemeral-chain-flow.md` and `references/seictl-cli.md` for the full mechanics; only the commands unique to this use case are below): +**This has an unverified dependency — confirm it before building anything on top of it.** Two paths, in preference order: -```sh -# Single-validator genesis chain — --replicas 1 is the load-bearing flag (default is 4) -seictl network apply --preset genesis-chain \ - --chain-id --image --replicas 1 \ - -n eng- --dry-run - -# One RPC follower, peered to the genesis chain — this is your submission + query + instrumentation target -seictl node apply --preset rpc \ - --chain-id --network \ - -n eng- --dry-run -``` +### Path A (preferred) — read-only historical `eth_call` with a state override -Render → PR against `harbor-engineering-workspace` → merge → `seictl network watch --until=Ready` → `seictl node watch --until=Running` — this is the standard `/harbor-dev` flow, not repeated here. +If Sei's EVM JSON-RPC supports `eth_call` with a historical `blockNumber`/`blockTag` **and** a state-override object (to simulate the bytecode as if already deployed, without a real mutating tx), this is the clean answer *for gas and correctness*: point it at an archive node's RPC, at height ``, no real mutating tx. **`operating-archive-node-byov.md` says nothing about this** — it's pure PV/EBS provisioning — so this capability is not documented anywhere in this repo's runbooks. **Confirm empirically before relying on it:** a trivial smoke test (a state-overridden `eth_call` for a known simple computation at a known historical height, checked against a hand-computed expected result). -**Instrument ``, not the validator.** You're already pointing `seid tx evm ... --evm-rpc=` at it for submission and `eth_getTransactionReceipt` for `gasUsed` — keeping the probe target and the query target in the same pod means one `kubectl debug --target=seid` session covers everything, with no cross-node clock correlation to worry about. The validator executes the same state transition for consensus, but there's no operational reason to split targets. +**Path A does not solve the *timing* half on the shared archive node.** Execution time here comes from §6/§7 eBPF instrumentation, and you may not attach a privileged probe to the **canonical shared archive node** — that is the §6 gate and the shared-blast-radius rule, and it is *not* your node to instrument. So Path A splits: +- **Gas / correctness:** the state-overridden `eth_call` against the shared archive is fine (read-only, no mutation). +- **Timing:** either (i) accept only a client-side RPC round-trip latency as a coarse, network-contaminated proxy (not eBPF-clean — state this limitation, don't dress it as compute time), or (ii) run the *same* state-overridden `eth_call` against **your own private archive/replay node** (restored to `` as in Path B step 1) which you *are* cleared to instrument. Option (ii) recovers eBPF-grade timing but forfeits Path A's "no new infrastructure" benefit — at which point Path A and Path B converge on "a private node you own," and the only difference is read-only `eth_call` vs. a real submitted tx. Choose Path A(ii) over Path B when the state-override RPC works, to avoid mutating even your private node. ---- +### Path B (fallback) — private writable fork of state-at-`` -## 4. Crafting opcode-isolated bytecode +If Path A's RPC capability doesn't exist or doesn't support the injection you need, use the same state-restore mechanism as §3 but stop treating it as a passive replayer: restore a `SeiNode` to height `` (`spec.replayer.snapshot.s3.targetHeight: `, same as §3 step 1), then instead of continuing historical replay, submit your own synthetic signed EVM transactions against it (`seid tx evm deploy` / `call-contract`, §5's bytecode). This gives you real trie depth / cache-warmth from genuine historical state (unlike a from-scratch empty genesis chain) at the cost of standing up and mutating your own private copy — never do this against the canonical shared archive. -**The differential-loop technique** isolates a single opcode's marginal cost without needing per-opcode instrumentation: write two bytecode variants that are identical except for the opcode under test. +Either path, **instrument the same node you're submitting to** — no cross-node clock correlation needed. + +--- -- **Variant A (opcode-under-test):** a loop body of `JUMPDEST`, the target `` (with whatever operands it needs pre-pushed), then the loop-decrement/`JUMPI` back to `JUMPDEST`. -- **Variant B (baseline):** identical loop scaffold, but the target opcode replaced with a cheap no-op pair (e.g. `PUSH1 0x00` / `POP`) of known, separately-measured cost. +## 5. Crafting opcode-isolated bytecode (Path B / any synthetic-tx run) -`(wall-clock(A) − wall-clock(B)) ÷ iteration count ≈ per-opcode marginal wall-clock cost`, and the same subtraction on `gasUsed` gives you the marginal *gas* cost to compare it against. This cancels the loop-overhead opcodes (`PUSH`/`DUP`/`SUB`/`JUMPI`) and the per-tx fixed overhead (§1) from both sides symmetrically. +**The differential-loop technique** isolates a single opcode's marginal cost without per-opcode instrumentation: write two bytecode variants identical except for the opcode under test. -Deploy and invoke via the `seid` CLI directly against `` (no sei-chain code involved — these are standard `seid tx evm` subcommands, see `seid tx evm --help` for the full flag set): +- **Variant A (opcode-under-test):** a loop body of `JUMPDEST`, the target `` (with operands pre-pushed), then loop-decrement/`JUMPI` back to `JUMPDEST`. +- **Variant B (baseline):** identical scaffold, target opcode replaced with a cheap no-op pair (`PUSH1 0x00`/`POP`) of known cost. + +`(wall-clock(A) − wall-clock(B)) ÷ iteration count ≈ per-opcode marginal wall-clock cost`; the same subtraction on `gasUsed` gives the marginal gas cost to compare against it. This cancels loop-overhead opcodes and per-tx fixed overhead from both sides symmetrically. ```sh seid tx evm deploy /path/to/-loop.bin \ - --from= --chain-id= --evm-rpc=http://.eng-.svc:8545 \ + --from= --chain-id= --evm-rpc= \ --gas-limit= --gas-fee-cap= seid tx evm call-contract \ - --from= --value=0 --chain-id= --evm-rpc=http://.eng-.svc:8545 + --from= --value=0 --chain-id= --evm-rpc= ``` -Pick a large-but-fixed iteration count up front (large enough that fixed overhead is a small fraction of total gas/time; the EVM code-size cap bounds how much you can unroll instead of loop, so loop rather than unroll for counts beyond a few hundred). Run each variant multiple times and take the distribution, not a single sample — wall-clock has real variance from GC pauses, Pebble compaction, and scheduling noise that gas cost is blind to. +Pick a large-but-fixed iteration count up front (large enough that fixed overhead is a small fraction of total gas/time; loop rather than unroll past a few hundred iterations — code-size cap). Run each variant multiple times and take the distribution, not a single sample — wall-clock has real variance from GC pauses, compaction, and scheduling noise that gas is blind to. + +**Submission protocol (Path B only):** submit exactly one transaction, wait for its block to finalize and its receipt to confirm, before submitting the next. This sidesteps the OCC re-execution confound (§1) by construction — nothing contends for the same slot if only one tx is ever in flight. `StateTransition.Execute` (§7) runs more than once per submitted tx — once during `eth_estimateGas`/`CheckTx` simulation, again at `FinalizeBlock` delivery — expect 2-3 invocations, and disambiguate by taking the one whose timestamp falls inside the actual block-production window. --- -## 5. Submission + correlation protocol +## 6. The privileged-attach gate — read before running `kubectl debug` -**Submit exactly one transaction, wait for its block to finalize and its receipt to confirm, before submitting the next.** This is the single biggest simplification available to you: with sequential, no-overlap submission, a block's execution window contains exactly the transaction(s) you just sent, in an order you know. It also directly sidesteps the OCC re-execution confound from §1 — nothing is contending for the same storage slot across concurrent transactions if there's only ever one in flight. +Any eBPF uprobe, `offcputime`, or CPU `profile` run against `seid` requires a **privileged kernel capability grant** (`CAP_BPF`/`CAP_SYS_ADMIN`/`CAP_PERFMON`) on a pod in **harbor — a shared EKS cluster** running other engineers' chains and `sei-k8s-controller` itself. This is root-on-node. The `/ebpf` skill's guardrail is unconditional here: **the gate is on the attach mechanism, not on how low-stakes the target is.** A throwaway replayer/archive-fork node does not waive it. -Log client-side, per transaction: `{submit_ts, tx_hash, block_height, gasUsed (from the receipt)}`. Join this against whatever the probe emits (§6) by time-ordering — you do not need to decode transaction data inside the probe itself. +**What's actually true about `eng-` today (verified against the platform repo, not assumed):** `eng-` namespaces carry no Pod Security Standard label (`clusters/harbor/engineers/base/namespace.yaml`), and harbor has no cluster-wide PSS default either (`clusters/harbor/admission/` contains only an unrelated tenant-hostname policy) — the effective PSS level is Kubernetes' own default, `privileged`, enforced nowhere. Compare **prod**'s `walle` namespace: explicit `restricted` PSS + a `ValidatingAdmissionPolicy` blocking ephemeral containers outright. Harbor's `eng-` has no equivalent. **This is an unhardened gap, not a sanctioned path**, and shared-node blast radius is real independent of PSS — engineer workloads bin-pack onto harbor's Karpenter pools alongside other tenants' processes. + +**Do this, in order, before attaching anything:** -**One real gotcha:** `StateTransition.Execute` (the function you're timing, §6) runs more than once per submitted transaction — once during `eth_estimateGas`/`CheckTx` simulation, and again at actual `FinalizeBlock` delivery. Expect **2–3 invocations per tx**, not one. Disambiguate by taking the invocation whose timestamp falls inside the block-production window for the block your tx landed in (correlate against the block's timestamp from `/status` or the block header), or simply the *last* invocation observed before the receipt becomes queryable. +1. **Get sign-off first** — surface what you're attaching, to what, on which pod, for how long, to the platform/security owner before running anything privileged. +2. **Pin to a dedicated node** — for the duration of the experiment, don't share the node with other tenants (§2's hardware-parity node is already tainted/dedicated by `sei-node`/`sei-archive`'s own scheduling — confirm nothing else landed there). +3. **Scope the capability grant to the minimum**, not `hostPID`: `kubectl debug --target=seid -it -0 --image= --profile=sysadmin` — `--target=` joins only that pod's PID namespace. Prefer `CAP_BPF`+`CAP_PERFMON` over the broader `sysadmin` profile where possible. +4. **If this becomes recurring**, that's the signal to open a platform PR for a dedicated profiling namespace with its own restricted-by-default PSS + a narrow, auditable exception (mirror `clusters/harbor/admission/tenant-hostname-policy.yaml`'s exception shape; `clusters/harbor/chaos-mesh/chaos-mesh.yaml` is harbor's existing precedent for an elevated-privilege, opt-in-gated workload) — **a platform decision, not something to self-approve.** --- -## 6. Instrumentation — cheapest and least-privileged first +## 7. Instrumentation — cheapest and least-privileged first -Don't reach for eBPF as step one. Try these in order; stop at the first tier that gives you a usable signal. +Don't reach for eBPF as step one. -**Tier 0 — free, no probe at all.** `gasUsed` from the receipt and `eth_estimateGas` are already there. Before instrumenting anything, record the live gas parameters and software version so the eventual comparison means something: confirm the value of any governance-mutable param you're testing against (e.g. `SeiSstoreSetGasEIP2200`, `x/evm/types/params.go:43`) via whatever query path `seid query evm --help` / `seid query params --help` currently exposes on the image you're running — don't assume a CLI shape that may have changed. Record the resolved image digest alongside it. +**Tier 0 — free, no probe.** `gasUsed` from the receipt and `eth_estimateGas` are already there. Record the live gas parameters (e.g. `SeiSstoreSetGasEIP2200`, `x/evm/types/params.go:43`) and image digest before instrumenting anything — the comparison is meaningless without knowing what you're comparing against. -**Tier 1 — Cosmos SDK's own pprof, no elevated privileges.** The `rpc` preset's `spec.overrides`/`--set` surface can set `network.rpc.pprof_laddr` in the rendered `config.toml` (see the preset's example override in `/harbor-dev`'s `references/seictl-cli.md`). Port-forward and pull a CPU profile with plain `go tool pprof` — zero Kubernetes privilege needed beyond what you already have in your own namespace. **Verify empirically whether this surfaces EVM-interpreter frames at all** — `net/http/pprof` in this codebase is only explicitly imported by the `ethreplay` subcommand, not the running validator/full-node path, so whether `pprof_laddr` gives you a populated `net/http/pprof` mux at all (via a transitive cosmos-sdk import) needs to be confirmed against the actual running pod, not assumed from either agent's read of `seid`'s own `main.go`. If it's empty or unhelpful, move to Tier 2 — this was a cheap thing to rule out first. +**Tier 1 — Cosmos SDK's own pprof, no elevated privileges.** The rendered `config.toml`'s `pprof_laddr` gives `go tool pprof` access with zero Kubernetes privilege beyond what you already have. **Verify empirically whether this surfaces EVM-interpreter frames at all** on the running pod before relying on it — don't assume from a source read alone. If empty or unhelpful, move to Tier 2. -**Tier 2 — eBPF (the gated tier; §2 must be cleared first).** +**Tier 2 — eBPF (gated; §6 must be cleared first).** -Resolve real symbols before writing any probe — do not assume upstream go-ethereum symbol names. `sei-chain`'s `go.mod` pulls go-ethereum via a `replace` directive to a Sei fork; **Go's `replace` preserves the original import path in compiled symbol names**, so the binary's symbols are still `github.com/ethereum/go-ethereum/...`, not a `sei-protocol/...` path. Confirm on the actual binary, not from source alone: +Resolve real symbols before writing any probe — `sei-chain`'s `go.mod` pulls go-ethereum via a `replace` directive, which **preserves the original import path in compiled symbol names**, so the binary's symbols are still `github.com/ethereum/go-ethereum/...`. Confirm on the actual binary: ```sh -kubectl -n eng- debug --target=seid -it -0 --image= --profile=sysadmin -- \ +kubectl -n eng- debug --target=seid -it -0 --image= --profile=sysadmin -- \ go tool nm /proc//root/usr/bin/seid | grep -E 'StateTransition\)\.Execute|EVMInterpreter\)\.Run' ``` -The two real boundaries (from ``, confirm line numbers haven't drifted before citing them to anyone): +The two real boundaries (confirm line numbers against `` before citing them to anyone): -- **Per-tx:** `github.com/ethereum/go-ethereum/core.(*StateTransition).Execute` (`core/state_transition.go`) — Sei's `x/evm/keeper/msg_server.go` (`applyEVMMessage`) calls this directly; don't probe the upstream `ApplyMessage` wrapper, Sei bypasses it. -- **Per-opcode dispatch loop (not individually probeable — see §1):** `github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run`. +- **Per-tx bracket:** `github.com/ethereum/go-ethereum/core.(*StateTransition).Execute` (`core/state_transition.go`) — Sei's `x/evm/keeper/msg_server.go` (`applyEVMMessage`) calls this directly, bypassing the upstream `ApplyMessage` wrapper. +- **Per-opcode dispatch (not individually probeable):** `github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run` — indirect function-table dispatch, no stable per-opcode symbol. This is why §5 constructs isolation via bytecode rather than probing the loop. -**Prefer `offcputime` and a PID-filtered on-CPU `profile` over raw entry/return timing.** This is the actual highest-value signal for your question — whether `Execute`'s wall-clock time is CPU-bound (interpreter work, i.e. what gas is supposed to meter) or blocked on Pebble/state I/O (which gas categorically does not meter) is a more honest answer than a single duration number, and both are map-aggregated (near-zero overhead at the ~1 tx/block rate this protocol produces): +**Prefer `offcputime` and a PID-filtered on-CPU `profile` over raw entry/return timing** — the actual highest-value signal is whether `Execute`'s wall-clock time is CPU-bound (interpreter work, what gas is supposed to meter) or blocked on state I/O (which gas categorically does not meter). Both are map-aggregated, so overhead is event-rate-bound not drain-bound — but "near-zero" holds only for **functionality 2's low tx rate**. **On a functionality-1 replayer running ~40 blocks/s with heavy state I/O, the context-switch rate is high, and `offcputime` overhead scales with context-switch rate** (`/ebpf` skill's `pack-perf-methodology.md` §7 caveat) — it is bounded but no longer negligible, and it perturbs the very replay you're timing. Per guardrail 3 (which holds even off-prod, because a perturbing probe corrupts a *measurement*): **A/B the replay throughput (blocks/s) with the probe attached vs. detached before trusting any number**; if attaching the probe moves replay throughput, the timing is contaminated — narrow the window, drop the sampling rate, or serialize (§3a) so the rate is low. Note also that during replay there is **no `CheckTx`/`eth_estimateGas` path** (blocks arrive already-finalized over p2p), so — unlike §5's submission flow — `Execute` fires once per tx *per OCC incarnation*, not 2–3× per tx: ``` # on-CPU attribution, PID-filtered bpftrace -e 'profile:hz:49 /pid == / { @[ustack] = count(); }' -# off-CPU (blocked) time, PID-filtered — needs the offcputime tool, not a bare bpftrace one-liner +# off-CPU (blocked) time, PID-filtered offcputime -p -f 30 ``` -**Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only, on this throwaway chain, with two hazards you must state plainly if you use it:** (1) Go's stack-copying runtime and `uretprobe`'s return-address rewrite can interact badly — this is a known divergence, not a hypothetical; (2) a goroutine can migrate OS threads mid-call on a blocking I/O read, which breaks a `tid`-keyed entry/return pairing even at concurrency 1. The `/ebpf` skill's "never on a validator/consensus-critical node" guardrail is weaker here specifically *because* this is a zero-stake personal chain where a crash costs you a re-deploy, not a slashing event — but that weakening applies only to the crash-tolerance judgment call, not to the §2 attach-privilege gate, which is unconditional. +**Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only**, with two hazards to state plainly if you use it: (1) Go's stack-copying runtime and `uretprobe`'s return-address rewrite can interact badly — a known divergence, not hypothetical; (2) a goroutine can migrate OS threads mid-call on a blocking I/O read, breaking a `tid`-keyed entry/return pairing even at concurrency 1. ``` uprobe:/proc//root/usr/bin/seid:"github.com/ethereum/go-ethereum/core.(*StateTransition).Execute" @@ -162,43 +184,53 @@ uretprobe:/proc//root/usr/bin/seid:"github.com/ethereum/go-ethereum/co --- -## 7. Reading results +## 8. Reading results -For each opcode under test, you should have: the differential wall-clock time (§4) at the tier-2 granularity you reached, the differential `gasUsed`, and the live gas parameter value that produced it (§6 tier 0). Report them as a ratio (time-per-gas-unit) per opcode, not a single aggregate — the whole point is to see *which* opcodes diverge, and by how much, not to produce one number for "the EVM." +For each opcode/block under test: the differential (functionality 2) or bracketed (functionality 1) wall-clock time at the tier reached, the `gasUsed`, and the live gas parameter value behind it (Tier 0). Report as a ratio (time-per-gas-unit) per opcode/tx, not a single aggregate. -**Before concluding anything, re-check against §1:** if the opcode you measured is `SSTORE`, you already know the gas side is a governance-set number, not a compute-derived one — a divergence there confirms a known fact, it isn't a finding. A divergence on `KECCAK256` or pure arithmetic is the genuine signal this experiment exists to find. A precompile's numbers belong in their own table, never blended with interpreted-opcode results. +**Before concluding anything, re-check against §1:** a divergence on `SSTORE` confirms a known governance-set gap, not a finding; the real signal is `KECCAK256`/arithmetic. Precompile numbers belong in their own table, never blended with interpreted-opcode results. For functionality 1, a bimodal or inconsistent time distribution across repeated replays of the same `` should raise the OCC-determinism assumption (§1) before any other explanation. --- -## 8. Failure modes (quick reference) +## 9. Failure modes (quick reference) | Symptom | Cause | Fix | |---|---|---| -| `seid tx evm` calls succeed but you can't read `gasUsed` / can't reach `--evm-rpc` | pointed at the validator, not the RPC follower | Validators don't serve EVM JSON-RPC (§3); target ``'s `evmJsonRpc` endpoint. | -| Probe never fires / symbol not found | assumed upstream go-ethereum's import path instead of resolving on the actual binary | Re-run the `go tool nm` resolution in §6 against `/proc//root/usr/bin/seid` — the `replace` directive keeps the upstream path, but don't take that on faith either. | -| Wall-clock measurements wildly inconsistent run-to-run | overlapping submissions triggering OCC re-execution (§1), or measuring a re-execution as if it were a clean single pass | Confirm strictly sequential one-tx-per-block submission (§5); check for more than one `Execute` invocation per tx and disambiguate by block-production window. | -| `seid` pod crashes or misbehaves after attaching a `uretprobe` | the Go-runtime/`uretprobe` interaction hazard (§6) | Expected risk on this tier — redeploy the personal chain (it's throwaway) and prefer `offcputime`/`profile` for anything beyond a one-off rehearsal. | -| "Gas doesn't match compute" conclusion on `SSTORE` | tested a governance-mutable param as if it were compute-derived (§1) | Reframe as "confirms the known SSTORE gas/upstream divergence," not a new finding; re-run on `KECCAK256`/arithmetic for the real test. | -| Can't get sign-off / told to stop before attaching anything privileged | §2's gate working as intended | This is the platform team's call, not something to route around — escalate the ask, don't self-approve a scoped-down version of the same attach. | +| `kubectl describe node` shows an instance type outside `r6i`/`r7i` | `controller-config.yaml` or NodePool requirements drifted between harbor and prod | Halt — the timing result won't reflect prod hardware. Diff `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml` and `clusters/{harbor,prod}/default/nodepool-*.yaml` before proceeding. | +| Can't get a clean read on block ``'s gas/results | Tried to force the replayer to halt exactly at `` instead of reading its result once the height passed | Let replay continue past ``; query ``'s results directly via RPC once the height has been applied (§3). | +| `eth_call` with a state override returns an error or silently ignores the override | Path A's RPC capability (§4) isn't actually supported on this `seid` version | Fall back to Path B — restore a private writable fork of state-at-`` and submit real signed txs. | +| Probe never fires / symbol not found | Assumed upstream go-ethereum's import path instead of resolving on the actual binary | Re-run the `go tool nm` resolution (§7) against `/proc//root/usr/bin/seid`. | +| Wall-clock measurements wildly inconsistent run-to-run | OCC re-execution (§1) — overlapping submissions (functionality 2) or parallel/re-executed scheduling on replay (functionality 1) | Functionality 2: strictly sequential one-tx-per-block. Functionality 1: render the replayer with `occ-enabled = false`, `concurrency-workers = 1` (§3a) for serial, deterministic, per-tx-correlatable application. | +| Functionality-1 per-tx timing doesn't line up with per-tx `gasUsed` | Bracketed a multi-tx block under OCC — got block-level aggregate, not per-tx (§1, §3a) | Disable OCC on the replayer (§3a) so ordinal-by-call-count maps to the Nth tx; if OCC can't be disabled, report block-level aggregate vs. block-total gas only. | +| Attaching the probe changes replay throughput (blocks/s) | High context-switch rate on a 40 blocks/s replayer makes `offcputime`/probe overhead non-negligible (§7, guardrail 3) | A/B throughput probe-on vs. probe-off; narrow the window, lower sampling rate, or serialize (§3a) until throughput is unperturbed. | +| Path A gives gas but no clean execution time | Tried to eBPF-attach the shared archive node (forbidden, §4/§6) | Run the state-overridden `eth_call` against your own private restore node (Path A(ii), §4), or fall back to Path B. | +| `seid` pod crashes after attaching a `uretprobe` | The Go-runtime/`uretprobe` interaction hazard (§7) | Expected risk on a throwaway node — redeploy and prefer `offcputime`/`profile` beyond a one-off rehearsal. | +| "Gas doesn't match compute" conclusion on `SSTORE` | Tested a governance-mutable param as if compute-derived (§1) | Reframe as confirming the known divergence; re-run on `KECCAK256`/arithmetic for the real test. | +| Can't get sign-off / told to stop before attaching anything privileged | §6's gate working as intended | Platform team's call — escalate, don't self-approve a scoped-down version of the same attach. | --- -## 9. Pre-flight checklist +## 10. Pre-flight checklist -- [ ] Personal single-validator + RPC-follower chain up via `/harbor-dev` (§3); confirmed you're pointed at ``, not the validator, for submission/query. -- [ ] Recorded the live gas parameter values + image digest you're testing against (§6 tier 0) — the comparison is meaningless without this. -- [ ] Differential bytecode pair built for the target opcode (§4), with a fixed, high-enough iteration count. -- [ ] Sequential one-tx-per-block submission harness ready, logging `{submit_ts, tx_hash, block_height, gasUsed}` (§5). +- [ ] Confirmed the target node landed on `r6i`/`r7i` via `sei-node`/`sei-archive` (§2) — don't trust the default without checking. +- [ ] Recorded live gas parameter values + image digest (§7 Tier 0). +- [ ] **Functionality 1:** replayer node restoring to `` and replaying forward (§3); confirmed per-tx `gasUsed` for `` is readable post-replay. +- [ ] **Functionality 1 (per-tx correlation):** if `` has >1 tx, replayer rendered with `occ-enabled = false` + `concurrency-workers = 1` (§3a) so serial ordinal correlation holds — or the report is explicitly scoped to block-level aggregate timing. Probe attached *continuously across a span containing ``* (window detection is post-hoc, not reactive — §3a), not started reactively on the height transition. +- [ ] **Functionality 2:** confirmed whether Path A's `eth_call` state-override capability actually exists on this `seid` version (§4) before committing to it; if not, Path B's private writable fork is staged instead. +- [ ] (Path B only) Differential bytecode pair built (§5), fixed high-enough iteration count, sequential one-tx-per-block submission harness logging `{submit_ts, tx_hash, block_height, gasUsed}`. - [ ] Tier 0 and Tier 1 (pprof) tried and found insufficient before reaching for eBPF. -- [ ] **Sign-off obtained** for any privileged attach, workload pinned to a dedicated node, capabilities scoped to `--target=seid` rather than `hostPID` (§2) — before running anything in §6 tier 2. -- [ ] Symbols resolved against the actual running binary, not assumed from source reading alone (§6). +- [ ] **Sign-off obtained** for any privileged attach, capabilities scoped to `--target=seid` rather than `hostPID` (§6) — before running anything in §7 Tier 2. +- [ ] Symbols resolved against the actual running binary, not assumed from source alone (§7). --- ## References -- Sei-EVM gas/precompile specifics: `/evm` skill, `references/sei-evm-profile.md`, `kit-evm-parity-gas.md`, `kit-sei-precompiles.md`; `x/evm/types/params.go` in `sei-chain` for the governance-mutable param set. -- eBPF method, overhead-bounding, and the Go-uprobe/uretprobe divergences: `/ebpf` skill, `references/pack-perf-methodology.md`. +- Sei-EVM gas/precompile specifics: `/evm` skill — `kit-evm-parity-gas.md`, `kit-sei-precompiles.md`; `x/evm/types/params.go` in `sei-chain` for the governance-mutable param set. +- eBPF method, overhead-bounding, Go-uprobe/uretprobe divergences: `/ebpf` skill, `references/pack-perf-methodology.md`. - Harbor chain spin-up mechanics: `/harbor-dev` skill, `references/ephemeral-chain-flow.md`, `references/seictl-cli.md`, `references/harbor-cluster.md`. -- Harbor `eng-` Pod Security / admission posture: `sei-protocol/platform` — `clusters/harbor/engineers/base/namespace.yaml`, `clusters/prod/walle/{namespace,podcomposition-policy}.yaml` (the posture to eventually mirror), `clusters/dev/sei-omnigent/sandbox-namespace.yaml`, `clusters/harbor/chaos-mesh/chaos-mesh.yaml`, `clusters/harbor/admission/tenant-hostname-policy.yaml`. -- Execution entry points: `sei-chain` — `x/evm/keeper/msg_server.go` (`applyEVMMessage`, `EVMTransaction`); the vendored go-ethereum fork's `core/state_transition.go` (`StateTransition.Execute`) and `core/vm/interpreter.go` (`EVMInterpreter.Run`) — resolve the fork version via `go.mod`, don't hardcode it here. +- Replay mechanism (functionality 1): `validating-flatkv-memiavl-parity-via-sharded-replay.md` (this directory) — the `replayer.snapshot.s3.targetHeight` restore, the shadow comparator's layer definitions (gas yes, timing no). +- Archive-node provisioning: `operating-archive-node-byov.md` (this directory) — infra only; does not document RPC query semantics, hence §4's Path A verification requirement. +- Hardware-parity mechanism: `sei-k8s-controller` — `internal/platform/platform.go` (`NodepoolForMode`), `internal/task/bootstrap_resources.go`, `internal/noderesource/noderesource.go`; `sei-protocol/platform` — `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml`, `clusters/{harbor,prod}/default/nodepool-{default,archive}.yaml`. +- Harbor `eng-` Pod Security / admission posture: `sei-protocol/platform` — `clusters/harbor/engineers/base/namespace.yaml`, `clusters/prod/walle/{namespace,podcomposition-policy}.yaml` (the posture to eventually mirror), `clusters/harbor/chaos-mesh/chaos-mesh.yaml`, `clusters/harbor/admission/tenant-hostname-policy.yaml`. +- Execution entry points: `sei-chain` — `x/evm/keeper/msg_server.go` (`applyEVMMessage`); the vendored go-ethereum fork's `core/state_transition.go` (`StateTransition.Execute`) and `core/vm/interpreter.go` (`EVMInterpreter.Run`) — resolve the fork version via `go.mod`, don't hardcode it here.