Skip to content

feat(node): spec-declared state-sync rpc-servers; hold STS while gate blocks plan#456

Merged
bdchatham merged 2 commits into
mainfrom
feat/statesync-explicit-rpc-servers
Jul 8, 2026
Merged

feat(node): spec-declared state-sync rpc-servers; hold STS while gate blocks plan#456
bdchatham merged 2 commits into
mainfrom
feat/statesync-explicit-rpc-servers

Conversation

@bdchatham

Copy link
Copy Markdown
Collaborator

Summary

  • spec.fullNode.snapshot.rpcServers — a spec-declared CometBFT witness set that replaces the controller-level canonical-syncer registry for that node, so self-serve chains (eng namespaces) can use state-sync bootstrap without per-chain platform config. Admission enforces the CometBFT floor (minItems=2), uniqueness (listType=set), and bare host:port shape; the reconcile gate keeps a live fail-closed floor that falls through to the registry path (CRD-downgrade backstop).
  • Hold initial StatefulSet creation while the state-sync gate suppresses the init plan. EnsureDataPVC (a plan task) is the sole creator of the data PVC the pod mounts by direct claimName, so an STS created during the block stranded a Pending pod on a claim nothing creates. The hold is create-only (StateSyncBlocksPlan && Status.StatefulSet == nil): Running nodes and existing STSes are never touched, so a transient registry blip cannot stall an image roll.
  • NoSyncersConfigured condition message now names both remediations (registry entry or spec rpcServers); new envtest suite exercises the admission rules against a real apiserver; fixed a stale impostor-branch comment in statefulset.go that contradicted noderesource/sync.go.

Motivating incident

eng-yiming/yiming-migration-mock-4-rpc-1 (2026-07-08): an RPC node added to a self-owned chain with stateSync: {} fail-closed on the empty registry (StateSyncReady=False/NoSyncersConfigured, correct) — but the ungated STS had already created a pod that sat Pending forever on data-...-rpc-1 not found, with the real cause buried in a SeiNode condition. Both halves are fixed: the config gap gets a self-serve field, and a blocked node now holds the STS so the only signal is the accurate one.

Design decisions

  • Replace, not merge (operator decision): a spec-declared set fully bypasses the registry, keeping minItems=2 enforceable at admission. Changing to merge later is a breaking semantic change on a served field — route through a new API version.
  • Field lives on SnapshotSource, not StateSyncSource: s3-restore applies via CometBFT state-sync and verifies against the same witnesses, so both variants need it (same reasoning as trustPeriod).
  • Spec items are normalized by sort+dedup only — never through the registry parser's comma-split, which would explode an admission-atomic item into unvalidated fragments.
  • Witnesses are light-client rpc-servers, not snapshot providers: chunks arrive over p2p from snapshot-serving peers, and at least one snapshot must already exist (see field doc).

Review evidence

Cross-reviewed by five independent lenses (systems, kubernetes, sei-network as assigned dissenter, platform, idiomatic) — unanimous after a resolution round; ledger: bdchatham-designs/designs/sei-k8s-controller-stack/xreview/feat-statesync-explicit-rpc-servers.md. The dissenter's headline objection (validators can't serve RPC witnesses) was empirically refuted against the live motivating chain (validator answers /status on 26657).

Accepted risk (tracked): a wrong-chain or dead spec-declared witness passes shape-only admission and fails at runtime in the sidecar — the sidecar does not yet assert the witness's chain-id. Eliminating fix is PLT-793 (seictl: chain-id assertion in trust-point queries + pad-to-1 hardening).

Rollout notes

  • CRD applies before the controller via config/default ordering (Flux); skew windows fail closed to the registry.
  • Retire a chain's registry entry only after every cell running that chain's nodes is pinned to a controller ?ref= that includes this change.

🤖 Generated with Claude Code

bdchatham and others added 2 commits July 7, 2026 19:29
… blocks plan

Add SnapshotSource.rpcServers: a spec-declared witness set that replaces the
controller-level canonical-syncer registry for that node, so self-serve chains
(eng namespaces) can use state-sync bootstrap without per-chain platform
config. Admission enforces the CometBFT floor (minItems=2), uniqueness
(listType=set), and bare host:port shape; the gate keeps a live fail-closed
floor that falls through to the registry path.

Also hold initial StatefulSet creation while the state-sync gate suppresses
the init plan: EnsureDataPVC is the only creator of the data PVC the pod
mounts by claimName, so an STS created during the block stranded a Pending
pod on a claim nothing creates (eng-yiming/yiming-migration-mock-4-rpc-1,
2026-07-08: StateSyncBlocked/NoSyncersConfigured with a pod unschedulable on
a nonexistent PVC). Only initial creation is held — a Running node's STS
keeps syncing through transient syncer-source errors.

The NoSyncersConfigured message now names both remediations. New envtest
suite exercises the admission rules against a real apiserver.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…comment

Cross-review findings: the rpcServers field doc conflated light-client
witnesses with p2p snapshot delivery, overstated what admission validation
guarantees (shape only — chain membership, reachability, and snapshot
existence are runtime concerns surfaced by the sidecar), and
statefulset.go's impostor paragraph contradicted the actual
Status.StatefulSet clearing in noderesource/sync.go.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Jul 8, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes node bootstrap ordering and introduces a new CRD field that affects state-sync trust configuration; wrong witnesses still pass admission and fail at runtime in the sidecar.

Overview
Adds spec.fullNode.snapshot.rpcServers (and the same on replayer/validator snapshot blocks) so operators can declare CometBFT light-client witnesses on the SeiNode instead of relying on the controller canonical-syncer registry. When at least two unique host:port entries are present, the state-sync gate takes them as ResolvedStateSyncers and skips the registry; otherwise behavior falls back to the registry with an updated NoSyncersConfigured message. CRD admission enforces minItems=2, set uniqueness, and endpoint shape; a new envtest suite exercises those rules against a real apiserver, and make test-integration includes ./internal/controller/node/envtest/....

Pre-Running nodes no longer get an initial StatefulSet while StateSyncBlocksPlan is true and status.statefulSet is unset, so a Pending pod is not created before the init plan (and EnsureDataPVC) can run. Running nodes and existing StatefulSets are unchanged. StateSyncBlocksPlan is exported from the planner so the controller shares the same predicate as plan suppression.

Planner/docs and configureStateSyncTask comments now describe witnesses from spec or registry; impostor-branch commentary in statefulset.go matches clearing Status.StatefulSet on UID mismatch.

Reviewed by Cursor Bugbot for commit 8a70fde. Bugbot is set up for automated code reviews on this repo. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant