feat: Lock-free apply_block refactor

[sergerad]

Summary

Closes #1539.
Closes #1853.
Closes #2414.

Makes the store's block-write path lock-free for readers, and makes the resulting read-consistency rules type-enforced. Reads previously contended on RwLocks over the in-memory trees (and blocked during apply_block's DB-commit window); they now load an immutable snapshot via ArcSwap and are never blocked by writes. All reads flow through a request-scoped StateView, so a query combining tree and DB data at different chain heights is no longer expressible.

                                   LoadedState::start()
                                           │ spawns worker task, hands out one of each
             ┌─────────────────────┬───────┴────────────┬─────────────────────┐
             ▼                     ▼                    ▼                     ▼
      ┌────────────┐        ┌─────────────┐      ┌─────────────┐      ╔══════════════╗
      │ Arc<State> │        │ BlockWriter │      │ ProofWriter │      ║  WriteWorker ║
      │ (read-only,│        │ (write cap, │      │ (write cap, │      ║ (tokio task) ║
      │  shared)   │        │  1 holder)  │      │  1 holder)  │      ╚══════════════╝
      └────────────┘        └─────────────┘      └─────────────┘        owns MUTABLE
             │                     │                    │               trees: nullifier,
             │       apply_block() │      apply_proof() │               account, MMR,
             │                     │                    │               forest
             │         WriteRequest│                    ├─ commit proof       │
             │             mpsc(1) │                    │  to block store     │ per committed
             │                     ▼                    │                     │ block: builds
             │                     ═══════════▶ ════════╪═══════════▶         │ + publishes
             │                                          │                     ▼
             │                                          │              ┌────────────────┐
      State fields                                      │              │ StateSnapshot  │
      ┌───────────────────────────────────────┐         │              │  (immutable,   │
      │ latest_snapshot: Arc<ArcSwap<─────────── swap on commit ──────▶│  per block N)  │
      │                        StateSnapshot>>│         │              │  tree READER   │
      │ committed_tip_tx: watch ◀── fired by worker     │              │  views + MMR   │
      │ proven_tip:       watch ◀── advanced by ────────┘              │ + SnapshotGuard│
      │ db, block_store, block/proof caches   │                        └────────────────┘
      └───────────────────────────────────────┘                                ▲
             │                                                                 │ pins ONE
             │ view()  (wait-free ArcSwap load, one per request)               │ generation
             ▼                                                                 │
      ┌─────────────────────────────────────────────┐                          │
      │ StateView  { snapshot: Arc<StateSnapshot> ──┼──────────────────────────┘
      │              db:       Arc<Db>            } │
      │  • tip() = snapshot height                  │   ALL reads live here:
      │  • DB queries scoped by tip                 │   get_account, get_*_inputs,
      │    (ScopedBlockNum / ScopedBlockRange)      │   sync_*, get_block_header, …
      │  • trees only via block_in_place helpers    │
      └─────────────────────────────────────────────┘

  Writes flow LEFT→RIGHT (capability → worker → snapshot); reads flow DOWN (State → view →
  pinned snapshot + tip-scoped DB). Live tips (committed/proven) bypass snapshots via watch
  channels. Readers never block writes; a view keeps serving block N while the worker
  publishes N+1.

Why:

• Read endpoints (sync, account/nullifier proofs, chain tip) no longer stall while a block is applied.
• Removes the fragile cross-task lock choreography in apply_block (oneshot handshakes between the DB task and the in-memory update).
• Snapshot-scoped reads were previously enforced only by convention — several paths read the tip and the data from different snapshots. StateView makes the scoping structural instead.

How:

Lock-free write path (state/writer/)

• A single WriteWorker task owns the mutable nullifier tree, account tree, blockchain MMR, and account-state forest, processing blocks serially from an mpsc channel — no locks. In-flight writes always complete; shutdown is only observed between requests.
• After each DB commit, the worker builds an immutable StateSnapshot (trees backed by read-only RocksDB snapshot views) and publishes it atomically via ArcSwap, so readers keep a consistent frozen view while the next block commits.
• Db::apply_block is now a plain transaction — the oneshot allow_acquire/acquire_done synchronization is removed.

Write capabilities (state/lifecycle.rs)

• LoadedState::start spawns the worker and returns the read-only Arc<State> plus non-cloneable BlockWriter/ProofWriter capabilities and a WriterTask handle, statically limiting each write path to one task. The capabilities expose no read access; tasks that read and write get Arc<State> alongside their capability.
• BlockWriter::stop drains and joins the worker so tree storage is released deterministically before the data directory is re-opened or deleted (used by recover and stress-test seeding).

Type-enforced reads (state/view/)

• All tree and DB reads live on StateView, pinned to one snapshot per request (State::view()). DB queries are scoped by the view's tip internally; callers cannot supply their own. RocksDB-backed trees are only reachable through block_in_place helpers; snapshot fields are only visible inside the view module.
• Tip-scoped Db queries require view-issued proof types (ScopedBlockNum / ScopedBlockRange), constructible only by a StateView after validating the bound against its tip — extending the enforcement to the DB boundary itself.
• Range-scoped sync queries validate range.end() <= tip themselves via a new RangeBeyondTip error (same InvalidArgument response as before); the RPC layer's range_bounds_check is deleted and pagination's chain_tip is now the tip the query actually ran against.
• Fixes paths that previously took two snapshots per request (get_account, the block producer's get_tx_inputs); sync_chain_mmr clamps the proven tip to the view's tip.

Live tips (state/tip.rs)

• State::committed_tip() / proven_tip() read the watch channels their writers publish to (mirroring subscribe_committed_tip / subscribe_proven_tip); the Finality enum is removed. The committed tip is published after the snapshot, so it never reports a block a fresh view cannot serve.

Observability: SnapshotGuard tracks live snapshot generations and lifetimes; warns when a snapshot outlives 10s or more than 4 generations are pinned (a leaked/slow reader pins a RocksDB snapshot).

Supporting changes: read-only reader() views for AccountStateForest / AccountTreeWithHistory (relaxed to BackendReader/SmtStorageReader bounds); state module restructured into view/ (read endpoints) and writer/ (worker + capabilities); new tracing field names allowlisted.

Changelog

[[entry]]
scope       = "node"
impact      = "changed"
description = "Store reads are lock-free: readers use atomically published in-memory snapshots and are no longer blocked while blocks are applied."

[sergerad]

Much of the diffs here are a move of impl but partial / involving a split to other files. The only changes should be w.r.t BlockNumber -> ScopedBlockNumber and ScopedBlockRange.

[sergerad]

This up_to_block bound may technically not be required but I think it might still be a good idea to use it for any path that involves multiple DB calls - some of which are block bound sensitive. If anything, to catch / not be effected by desync-like bugs we would not expect.