diff --git a/Cargo.lock b/Cargo.lock index 210708f..a8343d1 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2281,7 +2281,7 @@ dependencies = [ [[package]] name = "dig-node-service" -version = "0.75.0" +version = "0.75.1" dependencies = [ "async-trait", "axum", diff --git a/Cargo.toml b/Cargo.toml index d1e9652..2e90fea 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -32,7 +32,7 @@ edition = "2021" # the ROOT manifest (`[workspace.package].version`), so it MUST be set here for a # release to fire (§3.6). The library crates (dig-node-core/dig-runtime/dig-wallet) # keep their own independent versions — only the released binary tracks the workspace version. -version = "0.75.0" +version = "0.75.1" # Release hardening, matching digstore: keep integer-overflow checks ON in release. # The node parses untrusted serialized input and does offset/length arithmetic over diff --git a/DEVELOPMENT_LOG.md b/DEVELOPMENT_LOG.md index 1da42d0..6a73e37 100644 --- a/DEVELOPMENT_LOG.md +++ b/DEVELOPMENT_LOG.md @@ -4,6 +4,27 @@ High-signal realizations from debugging/development: non-obvious cross-system co sharp edges, and gotchas. Concise durable facts with context — NOT a change diary. See `CLAUDE.md` §4.5 for the maintenance contract (a curator periodically re-verifies + prunes). +## §21 backfill and the #1576 reshare are TWO transports for the same capsule — one gate, not two (#1614) + +A read miss can pull the SAME `(store, root)` whole `.dig` down two independent legs, and for a long +time each had its OWN in-flight set, blind to the other: + +- **Leg A — §21 backfill** (`maybe_backfill_capsule` → `gap_fill_generation` → `cache_fetch_and_cache`): + the authenticated whole-store sync from the RPC upstream. This is the PEERLESS-network fallback — it + can acquire a capsule when NO peer serves it, so it must never be suppressed away. +- **Leg B — #1576 reshare warm** (`spawn_capsule_reshare` → `CapsuleWarmer::warm`): the P2P pull that + makes a reader a discoverable HOLDER. It has NO upstream fallback — with no providers it just refuses. + +Because Leg A used `Node::backfilling` (a `HashSet`) and Leg B used a separate `WarmRegistry`, +a single read fired BOTH — 2× bandwidth/disk/CPU for one artifact. The fix is one shared single-flight +gate: `Node::capsule_acquisition` is the ONE `Arc`, and the reshare warmer is wired with a +CLONE of that same Arc (`wire_capsule_reshare(..., node.capsule_acquisition_gate())`), so both legs +test-and-set one registry keyed `"{store}:{root}"`. The keys were ALREADY byte-identical across the two +sites (`CapsuleKey::Display` == `maybe_backfill`'s `format!` == `WarmRegistry`'s key), which is what made +the collapse race-free — one mutex, one key space. Load-bearing ordering: each leg's origin/config/held +gates run BEFORE it claims the gate, so a gated-out read (a Peer origin, a cross-site read, an already-held +capsule) never consumes a slot. Keep BOTH legs — dropping Leg A would lose the peerless fallback. + ## Root has NO systemd `--user` bus, so a user-scope service install is impossible under `sudo` (#526) `systemctl --user` (and everything `service-manager`'s systemd backend does at `ServiceLevel::User`) diff --git a/SPEC.md b/SPEC.md index be79bfc..a542f03 100644 --- a/SPEC.md +++ b/SPEC.md @@ -2313,8 +2313,11 @@ a root OTHER than the confirmed root MUST be rejected (never cached or served). *actively seeks other nodes to pull missing generations* rather than only reacting to reads. (b) **Backfill-on-miss** — when a read is satisfied from another node or the upstream rather than from local disk, the node background-backfills the whole capsule so the NEXT read of that resource is - served locally (deduplicated: a backfill already in flight for `store:root` is not started twice). - Enabled by default; toggle with the `DIG_NODE_BACKFILL_ON_MISS` environment variable. + served locally (deduplicated: a backfill already in flight for `store:root` is not started twice). This + dedup spans BOTH whole-capsule transports: the §21 backfill here and the §21.3 P2P reshare warm claim + ONE shared single-flight gate keyed `(store, root)`, so a read starts at most one whole-capsule pull no + matter which leg wins, and the shared cap bounds concurrent distinct-generation acquisitions across the + two legs together. Enabled by default; toggle with the `DIG_NODE_BACKFILL_ON_MISS` environment variable. - **Fail-closed.** Gap-fill never pulls against an unconfirmable root (the §14.2 decision gates it). - **Verification invariant.** Every served module is verified against the chain-anchored root at SERVE, no matter how it arrived — a client read, a §21 whole-store sync, or a proactive/backfill gap-fill. @@ -3913,9 +3916,17 @@ read's latency is user-facing. Serving a module range is paced by the SAME FCFS `dig.fetchRange` uses (§17) — a whole-capsule pull is the largest thing a node serves, so exempting it would leave the biggest transfer as the one path able to starve every other peer. -At most ONE warm per generation runs at a time, so a burst of reads across a capsule's resources cannot -start N concurrent pulls of the same module. A store-granularity read starts no warm: it does not name -WHICH generation to pull, and guessing would reshare a capsule nobody asked for. +A single read triggers AT MOST ONE whole-capsule acquisition for a `(store, root)` generation, regardless +of tier. Two transports can pull the SAME capsule down — the §21 authenticated whole-store backfill +(`maybe_backfill_capsule` → `gap_fill_generation`, the peerless-network fallback that acquires from the +RPC upstream when no peer serves) and the §21.3 P2P reshare warm — and they are two routes to the same +artifact, so they claim ONE shared single-flight gate keyed `(store, root)`. Whichever leg claims the key +first runs the pull; the other, and any further read of the same not-yet-held capsule, is refused. A +burst of reads across a capsule's resources therefore cannot start N concurrent pulls of the same module, +and the two legs cannot double-pull it between them. The shared gate ALSO bounds how many DISTINCT +generations may acquire concurrently (a fixed cap across BOTH legs, not each in isolation): claims beyond +the cap are SKIPPED, not queued — the next read simply tries again. A store-granularity read starts no +warm: it does not name WHICH generation to pull, and guessing would reshare a capsule nobody asked for. ### 21.5. Dial path (MUST) @@ -3943,7 +3954,9 @@ A read can trigger background legs that spend this node's bandwidth and disk and advertises network-wide: the whole-capsule warm (`maybe_backfill_capsule`, §21.4) and the reshare pull plus holder-announce (§21.3). Those legs MUST be reachable only from a read this node's OWN OPERATOR made. A read that arrived from the network is SERVED normally and effects NOTHING: it starts no warm, no -reshare, no promotion, and no announce. +reshare, no promotion, and no announce. These two legs share ONE single-flight acquisition gate (§21.4), +so the origin/config/held gates on EACH leg run BEFORE it can claim that gate — a gated-out read never +consumes a slot, and the two legs dedup against each other rather than each pulling the capsule. The rule is not optional hardening. Every peer-facing read surface is unauthenticated in the sense that matters here — any well-formed self-signed mTLS leaf is accepted (§13), and the plaintext `/s/` surface diff --git a/crates/dig-node-core/src/download.rs b/crates/dig-node-core/src/download.rs index 1258e1b..167fafb 100644 --- a/crates/dig-node-core/src/download.rs +++ b/crates/dig-node-core/src/download.rs @@ -1184,6 +1184,10 @@ impl NodeContent { anchor_resolver: Arc, announce: Arc, cache_dir: &Path, + // The node's SHARED single-flight acquisition gate (#1614). Passed in — NOT freshly created — + // so this reshare warm claims the SAME registry the §21 backfill leg does; the two transports + // for one capsule then dedup against each other instead of each pulling the whole `.dig`. + capsule_acquisition: Arc, ) { let transport = Arc::new(crate::seams::dig_peer::NatModuleTransport::new( node_cert, @@ -1205,7 +1209,7 @@ impl NodeContent { cache_dir: cache_dir.to_path_buf(), }, announce, - Arc::new(crate::seams::dig_peer::WarmRegistry::new()), + capsule_acquisition, dig_download::ModuleDownloadConfig::default(), )); } diff --git a/crates/dig-node-core/src/lib.rs b/crates/dig-node-core/src/lib.rs index 5c89e5b..c85cd25 100644 --- a/crates/dig-node-core/src/lib.rs +++ b/crates/dig-node-core/src/lib.rs @@ -314,10 +314,17 @@ pub struct Node { /// (the browser is a consumer with no DHT), where an inventory-change refresh is a no-op. Kept off /// the `Node` struct's DHT-handle dependency (the node stays FFI-safe) by taking a boxed async hook. inventory_refresher: OnceLock, - /// Capsules whose background backfill (§14.3) is currently in flight, keyed `store_hex:root_hex`, - /// so a burst of resource reads for the same not-yet-held store spawns ONE whole-`.dig` pull, not - /// one per read. An entry is inserted before the pull spawns and removed when it finishes. - backfilling: std::sync::Mutex>, + /// The ONE single-flight gate every whole-capsule acquisition claims against, keyed + /// `store_hex:root_hex` (#1614). A read miss can pull the same `(store, root)` capsule down TWO + /// transports — the §21 authenticated whole-store backfill ([`Node::maybe_backfill_capsule`] → + /// [`Node::gap_fill_generation`]) and the #1576 P2P reshare warm ([`crate::seams::dig_peer::CapsuleWarmer`]). + /// They are two transports for the SAME artifact, so they share this gate: whichever leg claims the + /// key first runs the pull, the other refuses, and a burst of resource reads across one not-yet-held + /// store starts exactly ONE whole-`.dig` pull. The reshare warmer is wired with a clone of this same + /// `Arc` ([`crate::download::NodeContent::wire_capsule_reshare`]), so both legs test-and-set one + /// registry. The registry's distinct-generation cap ([`crate::seams::dig_peer::DEFAULT_MAX_CONCURRENT_WARMS`]) + /// therefore bounds concurrent acquisitions across BOTH legs, not each in isolation. + capsule_acquisition: Arc, /// A WEAK self-reference, installed by the standalone peer-network bring-up (which holds the /// `Arc`), so a `&self` read handler can spawn a detached background task that needs an owned /// `Arc` — the capsule backfill (§14.3). `Weak` (not `Arc`) so the node's refcount is @@ -2520,6 +2527,14 @@ fn decode_rk(hex_str: &str) -> Result<[u8; 32], ()> { } impl Node { + /// The node's ONE whole-capsule single-flight gate (#1614), shared by the §21 backfill leg + /// ([`Node::maybe_backfill_capsule`]) and the #1576 reshare warm. Handed to + /// [`crate::download::NodeContent::wire_capsule_reshare`] so both legs claim the same registry and a + /// read triggers at most one whole-capsule acquisition. + pub(crate) fn capsule_acquisition_gate(&self) -> Arc { + self.capsule_acquisition.clone() + } + /// Build a node from the environment (cache dir/cap, §21 identity, upstream). /// Used by both the standalone bin's [`run`] and the in-process `dig-runtime`. pub fn from_env() -> Arc { @@ -2556,7 +2571,7 @@ impl Node { p2p_content: OnceLock::new(), content_cache: std::sync::Mutex::new(ContentCache::default()), inventory_refresher: OnceLock::new(), - backfilling: std::sync::Mutex::new(std::collections::HashSet::new()), + capsule_acquisition: Arc::new(crate::seams::dig_peer::WarmRegistry::new()), verification_ledger: verification_ledger::VerificationLedger::new(), self_ref: OnceLock::new(), gossip: OnceLock::new(), @@ -2652,7 +2667,7 @@ pub(crate) mod test_support { p2p_content: OnceLock::new(), content_cache: std::sync::Mutex::new(ContentCache::default()), inventory_refresher: OnceLock::new(), - backfilling: std::sync::Mutex::new(std::collections::HashSet::new()), + capsule_acquisition: Arc::new(crate::seams::dig_peer::WarmRegistry::new()), verification_ledger: verification_ledger::VerificationLedger::new(), self_ref: OnceLock::new(), gossip: OnceLock::new(), @@ -3007,7 +3022,7 @@ mod tests { p2p_content: OnceLock::new(), content_cache: std::sync::Mutex::new(ContentCache::default()), inventory_refresher: OnceLock::new(), - backfilling: std::sync::Mutex::new(std::collections::HashSet::new()), + capsule_acquisition: Arc::new(crate::seams::dig_peer::WarmRegistry::new()), verification_ledger: verification_ledger::VerificationLedger::new(), self_ref: OnceLock::new(), gossip: OnceLock::new(), @@ -3088,7 +3103,7 @@ mod tests { p2p_content: OnceLock::new(), content_cache: std::sync::Mutex::new(ContentCache::default()), inventory_refresher: OnceLock::new(), - backfilling: std::sync::Mutex::new(std::collections::HashSet::new()), + capsule_acquisition: Arc::new(crate::seams::dig_peer::WarmRegistry::new()), verification_ledger: verification_ledger::VerificationLedger::new(), self_ref: OnceLock::new(), gossip: OnceLock::new(), @@ -3134,7 +3149,7 @@ mod tests { p2p_content: OnceLock::new(), content_cache: std::sync::Mutex::new(ContentCache::default()), inventory_refresher: OnceLock::new(), - backfilling: std::sync::Mutex::new(std::collections::HashSet::new()), + capsule_acquisition: Arc::new(crate::seams::dig_peer::WarmRegistry::new()), verification_ledger: verification_ledger::VerificationLedger::new(), self_ref: OnceLock::new(), gossip: OnceLock::new(), @@ -3221,7 +3236,7 @@ mod tests { p2p_content: OnceLock::new(), content_cache: std::sync::Mutex::new(ContentCache::default()), inventory_refresher: OnceLock::new(), - backfilling: std::sync::Mutex::new(std::collections::HashSet::new()), + capsule_acquisition: Arc::new(crate::seams::dig_peer::WarmRegistry::new()), verification_ledger: verification_ledger::VerificationLedger::new(), self_ref: OnceLock::new(), gossip: OnceLock::new(), @@ -3290,7 +3305,7 @@ mod tests { p2p_content: OnceLock::new(), content_cache: std::sync::Mutex::new(ContentCache::default()), inventory_refresher: OnceLock::new(), - backfilling: std::sync::Mutex::new(std::collections::HashSet::new()), + capsule_acquisition: Arc::new(crate::seams::dig_peer::WarmRegistry::new()), verification_ledger: verification_ledger::VerificationLedger::new(), self_ref: OnceLock::new(), gossip: OnceLock::new(), @@ -3319,6 +3334,17 @@ mod tests { }); } + /// **Proves (#1614):** the §21 backfill leg and the #1576 reshare leg claim against ONE shared + /// single-flight gate, so a single read triggers AT MOST ONE whole-capsule acquisition. This stub + /// asserts the gate is reachable as `Node::capsule_acquisition`; the full dedup pins follow. + #[tokio::test] + async fn capsule_acquisition_gate_is_a_single_shared_registry() { + let (node, _td) = test_node(Some([5u8; 32])); + let key = format!("{}:{}", "ab".repeat(32), "cd".repeat(32)); + // A fresh node has claimed nothing on the shared gate. + assert!(!node.capsule_acquisition.is_warming(&key)); + } + /// **Proves:** capsule backfill (§14.3) is a safe NO-OP on the FFI/consumer path — a node with no /// P2P content engine + no installed self-ref (the browser's in-process node) never spawns a pull /// and never records an in-flight entry, so a resource read there is unchanged. @@ -3333,11 +3359,9 @@ mod tests { node.maybe_backfill_capsule(&store_hex, &root_hex, crate::download::ReadOrigin::Local); // Nothing pulled, nothing left in-flight. assert!(!module_exists(&node.cache_dir, &store_hex, &root_hex)); + let key = format!("{store_hex}:{root_hex}"); assert!( - node.backfilling - .lock() - .unwrap_or_else(|p| p.into_inner()) - .is_empty(), + !node.capsule_acquisition.is_warming(&key), "no in-flight backfill claimed on the consumer path" ); } @@ -3354,11 +3378,9 @@ mod tests { let root_hex = "cd".repeat(32); seed_module(&node, &store_hex, &root_hex, b"already-here"); node.maybe_backfill_capsule(&store_hex, &root_hex, crate::download::ReadOrigin::Local); + let key = format!("{store_hex}:{root_hex}"); assert!( - node.backfilling - .lock() - .unwrap_or_else(|p| p.into_inner()) - .is_empty(), + !node.capsule_acquisition.is_warming(&key), "an already-held capsule claims no in-flight backfill slot" ); } @@ -7534,11 +7556,7 @@ mod tests { let key = format!("{}:{}", store.to_hex(), tip.to_hex()); assert!( - !node - .backfilling - .lock() - .unwrap_or_else(|p| p.into_inner()) - .contains(&key), + !node.capsule_acquisition.is_warming(&key), "a Peer-origin miss must never spawn a background backfill" ); } @@ -7578,14 +7596,112 @@ mod tests { let key = format!("{}:{}", store.to_hex(), tip.to_hex()); assert!( - node.backfilling - .lock() - .unwrap_or_else(|p| p.into_inner()) - .contains(&key), + node.capsule_acquisition.is_warming(&key), "a Local-origin miss (the control) must spawn a background backfill" ); } + /// **Proves (#1614):** the §21 backfill leg and the #1576 reshare leg claim the ONE shared gate, so a + /// read triggers AT MOST ONE whole-capsule pull. Here the reshare leg has already won the race and + /// holds the capsule's single-flight slot; the §21 backfill for the SAME `(store, root)` must then + /// find the slot taken and start NO second pull. + /// **Catches:** the pre-#1614 two-registries defect, where `maybe_backfill` used its own + /// `Node::backfilling` set — blind to the reshare claim — and fired a redundant whole-`.dig` pull. + #[test] + fn backfill_defers_to_an_in_flight_reshare_on_the_shared_gate() { + let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner()); + std::env::remove_var("DIG_NODE_PIN"); + std::env::remove_var("DIG_NODE_ON_MISS"); + std::env::remove_var("DIG_NODE_BACKFILL_ON_MISS"); + let rt = pin_test_rt(); + let (store, tip, rk) = miss_setup(); + let (node, td) = test_node(None); + let node = Arc::new(node); + node.set_self_ref(Arc::downgrade(&node)); + let cid = ContentId::resource(store.0, tip.0, [0xcd; 32]); + attach_p2p( + &node, + vec![dig_download::testkit::mock_provider(5, &cid)], + dig_download::testkit::MockContent::even(10, 1), + MissMode::Redirect, + &td, + ); + + // The reshare leg wins the race first: it claims the capsule on the node's SHARED gate. + let key = format!("{}:{}", store.to_hex(), tip.to_hex()); + let reshare_claim = node + .capsule_acquisition_gate() + .claim(key.clone()) + .expect("the reshare leg claims the fresh gate"); + + // Now drive the identical Local read: the §21 backfill sees the slot already taken. + let resp = rt.block_on(handle_rpc( + &node, + json!({"jsonrpc":"2.0","id":9,"method":"dig.fetchRange","params":{ + "store_id": store.to_hex(), "root": tip.to_hex(), "retrieval_key": rk, + "length": 4096, "offset": 0, + }}), + crate::download::ReadOrigin::Local, + )); + assert_eq!(resp["error"]["code"], json!(CONTENT_REDIRECT), "{resp}"); + + // The only claim on the gate is the reshare leg's: dropping it must free the slot completely, + // proving the backfill did NOT add a second, independent claim. + drop(reshare_claim); + assert!( + !node.capsule_acquisition.is_warming(&key), + "the §21 backfill must NOT hold its own slot once the reshare claim is released — one pull, \ + one gate" + ); + } + + /// **Proves (#1614):** once the §21 backfill leg holds the shared gate, the #1576 reshare leg, + /// claiming the SAME `Arc` (via [`Node::capsule_acquisition_gate`]), is refused — the + /// two legs are wired to ONE registry instance, not two blind ones. + /// **Catches:** the reshare warmer being wired with a fresh `WarmRegistry` (the pre-#1614 shape). + #[test] + fn reshare_defers_to_an_in_flight_backfill_on_the_shared_gate() { + let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner()); + std::env::remove_var("DIG_NODE_PIN"); + std::env::remove_var("DIG_NODE_ON_MISS"); + std::env::remove_var("DIG_NODE_BACKFILL_ON_MISS"); + let rt = pin_test_rt(); + let (store, tip, rk) = miss_setup(); + let (node, td) = test_node(None); + let node = Arc::new(node); + node.set_self_ref(Arc::downgrade(&node)); + let cid = ContentId::resource(store.0, tip.0, [0xcd; 32]); + attach_p2p( + &node, + vec![dig_download::testkit::mock_provider(5, &cid)], + dig_download::testkit::MockContent::even(10, 1), + MissMode::Redirect, + &td, + ); + + // The §21 backfill leg wins the race first (a Local read claims the shared gate). + let resp = rt.block_on(handle_rpc( + &node, + json!({"jsonrpc":"2.0","id":9,"method":"dig.fetchRange","params":{ + "store_id": store.to_hex(), "root": tip.to_hex(), "retrieval_key": rk, + "length": 4096, "offset": 0, + }}), + crate::download::ReadOrigin::Local, + )); + assert_eq!(resp["error"]["code"], json!(CONTENT_REDIRECT), "{resp}"); + + // The reshare leg, claiming the SAME instance, is refused — proof it shares one registry. + let key = format!("{}:{}", store.to_hex(), tip.to_hex()); + assert!( + node.capsule_acquisition.is_warming(&key), + "the §21 backfill must hold the shared slot", + ); + assert!( + node.capsule_acquisition_gate().claim(key).is_none(), + "the reshare leg, claiming the SAME gate, must be refused while the backfill holds it", + ); + } + #[test] fn fetch_through_pulls_from_the_holder_and_serves_the_bytes() { let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner()); diff --git a/crates/dig-node-core/src/peer.rs b/crates/dig-node-core/src/peer.rs index b733e27..bb7f05c 100644 --- a/crates/dig-node-core/src/peer.rs +++ b/crates/dig-node-core/src/peer.rs @@ -2443,6 +2443,9 @@ async fn run_peer_network(node: Arc) -> Result<(), String> { holdings: holdings_flood.clone(), }), node.cache_dir_path(), + // Share the node's ONE acquisition gate (#1614): the reshare warm and the §21 backfill leg + // claim the SAME registry, so a read triggers at most one whole-capsule pull across both. + node.capsule_acquisition_gate(), ); node.set_p2p_content(content); println!( diff --git a/crates/dig-node-core/src/seams/capsule/capsule_store.rs b/crates/dig-node-core/src/seams/capsule/capsule_store.rs index 4bf1865..7305ba0 100644 --- a/crates/dig-node-core/src/seams/capsule/capsule_store.rs +++ b/crates/dig-node-core/src/seams/capsule/capsule_store.rs @@ -103,8 +103,9 @@ pub trait CapsuleStore: Send + Sync { /// delayed. It is a NO-OP when: backfill is disabled; there is no P2P content engine (the /// in-process FFI consumer — it has no upstream/peer network to pull a whole capsule from); the /// capsule is already held locally; or a backfill for this exact capsule is already in flight - /// (deduped via `Node::backfilling`, so a burst of resource reads for the same not-yet-held store - /// triggers ONE whole-`.dig` pull, not one per read). The pull reuses + /// (deduped via the shared `capsule_acquisition` gate — one `WarmRegistry` both this §21 backfill + /// and the P2P reshare leg claim — so a burst of resource reads for the same not-yet-held store, + /// across either acquisition transport, triggers ONE whole-`.dig` pull, not one per read). The pull reuses /// [`Self::gap_fill_generation`] — the authenticated §21 whole-store sync, chain-anchored-root /// pinned + DHT-announced — so a backfilled capsule is verified exactly like every other cached /// generation. @@ -338,16 +339,20 @@ impl CapsuleStore for Node { return; } let key = format!("{store_hex}:{root_hex}"); - // Dedup: claim the in-flight slot; if another read already claimed it, do nothing (a burst of - // resource reads for the same not-yet-held store triggers ONE whole-capsule pull). - { - let mut inflight = self.backfilling.lock().unwrap_or_else(|p| p.into_inner()); - if !inflight.insert(key.clone()) { - return; // a backfill for this capsule is already running - } - } + // Single-flight against the SHARED acquisition gate (#1614): this §21 backfill and the #1576 + // reshare warm are two transports for the SAME capsule, so they claim ONE registry. If the + // other leg (or a prior read on this leg) already claimed this capsule, do nothing — a burst of + // resource reads for the same not-yet-held store triggers exactly ONE whole-capsule pull across + // BOTH legs. The gates ABOVE (origin, config, p2p_content, already-held) all run BEFORE this + // claim, so a gated-out read never consumes a concurrency slot (#1576/#1654). + let Some(claim) = self.capsule_acquisition.clone().claim(key.clone()) else { + return; // an acquisition for this capsule is already in flight on one of the two legs + }; let root = Bytes32(root_bytes); tokio::spawn(async move { + // The claim guard is MOVED into the task so the single-flight slot is held for the whole + // pull and released on completion (or drop, incl. panic), never before the pull spawns. + let _claim = claim; // OPERATOR-VISIBLE at the default level, both ways (#1886): landing a capsule is // the moment this node becomes a holder and the content-replication flywheel turns, // and failing to land one is the moment it silently does not. At `debug!` a broken @@ -363,11 +368,6 @@ impl CapsuleStore for Node { "backfill: whole-capsule pull did not complete (will re-attempt on the next miss)" ), } - // Release the in-flight slot so a later miss can re-attempt if this one failed. - node.backfilling - .lock() - .unwrap_or_else(|p| p.into_inner()) - .remove(&key); }); } diff --git a/crates/dig-node-core/src/seams/dig_peer/module_reshare.rs b/crates/dig-node-core/src/seams/dig_peer/module_reshare.rs index 110c006..bbfc19c 100644 --- a/crates/dig-node-core/src/seams/dig_peer/module_reshare.rs +++ b/crates/dig-node-core/src/seams/dig_peer/module_reshare.rs @@ -105,7 +105,7 @@ impl WarmRegistry { /// /// The claim is released when the returned guard drops — including on panic, so a panicking pull /// cannot permanently block a generation from ever being warmed again. - fn claim(self: &Arc, key: String) -> Option { + pub(crate) fn claim(self: &Arc, key: String) -> Option { let mut guard = self .in_flight .lock() @@ -138,7 +138,10 @@ impl WarmRegistry { } /// Releases a [`WarmRegistry`] claim on drop. -struct WarmClaim { +/// +/// `pub(crate)` so the §21 backfill leg ([`crate::Node::maybe_backfill_capsule`]) can hold a claim on +/// the SAME shared gate as the reshare warm (#1614) — the guard whose drop frees the single-flight slot. +pub(crate) struct WarmClaim { registry: Arc, key: String, } @@ -325,6 +328,13 @@ impl CapsuleWarmer { }) } + /// The single-flight [`WarmRegistry`] this warmer claims against. Exposed so a test can prove the + /// reshare leg and the §21 backfill leg were wired to the SAME shared gate (#1614). + #[cfg(test)] + pub(crate) fn registry(&self) -> &Arc { + &self.registry + } + /// Pull the whole capsule for `(store_hex, root_hex)`, cache it, and announce this node as a /// holder — the full reshare step, awaited. /// @@ -708,6 +718,35 @@ mod tests { ) } + /// **Proves (#1614):** a warmer stores the EXACT `Arc` it was built with — the property + /// [`crate::download::NodeContent::wire_capsule_reshare`] relies on to make the reshare leg claim the + /// node's SHARED gate rather than a fresh one. If `CapsuleWarmer::new` ever cloned into a new registry + /// instead of keeping the passed `Arc`, the two legs would silently double-pull again. + /// **Catches:** the reshare warmer being wired with `Arc::new(WarmRegistry::new())` (the pre-#1614 + /// shape) instead of the node's `capsule_acquisition` gate. + #[test] + fn a_warmer_shares_the_exact_registry_arc_it_was_built_with() { + let dir = temp_dir("shared-registry"); + let shared = Arc::new(WarmRegistry::new()); + let warmer = CapsuleWarmer::new( + Arc::new(NoHolders), + Arc::new(UnusedTransport), + Arc::new(dig_download::FileStateStore::new(dir.join("state"))), + Arc::new(ConfirmingResolver), + WarmPaths { + staging_dir: dir.join("staging"), + cache_dir: dir.join("cache"), + }, + Arc::new(AnnounceSpy::default()), + Arc::clone(&shared), + dig_download::ModuleDownloadConfig::default(), + ); + assert!( + Arc::ptr_eq(warmer.registry(), &shared), + "the warmer must claim against the SAME registry instance it was wired with, not a fresh one" + ); + } + /// **Proves:** a pull that ends in `Err` announces NOTHING and leaves no module at the cache path. /// **Catches:** announcing on finalize-observed / partial staging / a failed pull — which, because /// the announce is driven off cache inventory (#1586), would advertise this node network-wide as an