feat(dig-node): serve getProof/getMetadata/getPublicManifest/getCapsule locally - #179
feat(dig-node): serve getProof/getMetadata/getPublicManifest/getCapsule locally#179MichaelTaylor3d wants to merge 11 commits into
Conversation
16c4e23 to
df07cd9
Compare
6052a87 to
1212c92
Compare
MichaelTaylor3d
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES-REQUIRED (recorded as a comment review — this token shares the PR author identity, so REQUEST_CHANGES is a self-review 422; the blocking findings are the open inline threads below.)
CHANGES-REQUIRED — reviewed at 1212c92e5cecda9d0dbe6ad2405a7e9908db4b09
Independent correctness gate. Own worktree (C:\tmp\worktrees\dn179g, private CARGO_TARGET_DIR); the shared primary checkout was not touched. Full suite green at head: 676 passed / 0 failed (cargo test -p dig-node-core --lib, 253s). All 15 required checks green.
The implementation is good and the PR body is honest — it explicitly disclaims the *.on.dig.net justification, which I independently agree with. Three things block.
Blocking
lib.rs:2299— the anti-fabrication guard has no test. Mutatingif proof.is_empty()toif falseleaves the entire 676-test suite green. The named guardget_proof_errors_rather_than_fabricating_a_proof_it_cannot_producenever reaches it (guard-not-exercised).SPEC.md:887-888— untouched, and now normatively wrong. Still classifiesdig.getCapsuleanddig.getProofas passthrough;dig.getMetadata/dig.getPublicManifestare absent from the catalogue entirely. SPEC.md is the contract an independent reimplementation is built from (§4.2), so it is a contract split, not a doc nit.lib.rs:2467/meta.rs:88— doc asserts a shape the code does not produce. Both say a capsule window carries noinclusion_proof; it always carries"", and the PR's own test asserts exactly that.
Non-blocking (posted inline, resolved by me)
peer.rs:1156— the load-bearing peer-reachability reasoning has no assertion.lib.rs:9357— the anti-rollback test does not discriminate the arm it names.
Confirmed sound — do not re-litigate
dig.getProofis client-verifiable. Probe against a 3-file fixture:proof.leaf == SHA-256(reassembled ciphertext)(eadd3580a2cb…),path.len() == 2(non-trivial path),proof.verify()true,proof.root ==the chain-anchored root. ADigstoreProofVerifierclient can verify what this returns against a chain-anchored root.- Both three-window fixtures survived the rebases (
lib.rs:8603,lib.rs:8701, both2 * WINDOW + 500, middle window asserted). Mutatingcontent_window_envelope's clamp toif start == 0 { (start + WINDOW).min(total) } else { total }failsa_client_can_reassemble_a_multi_window_resource_from_the_envelope_alone. - The two-window fixture at
lib.rs:8569is fine — keep it. Ruling on the judgement call you surfaced: your reasoning holds. The middle-window placement property is discriminated by the reassembly test above (windows_seen == 3), proven by that same mutation, so uniformity buys nothing here. dispatch.rs:88-89reasoning verified in the tree, not taken on trust:peer.rs:1156does terminate inMethod::from_name(m).is_some_and(|m| m.is_peer_reachable()), anddig-rpc-protocol 0.6.0method.rs:218-233excludesGetProof/GetMetadata/GetCapsule/GetModule. No peer-surface widening by any of the five new methods.- Redirects are handled correctly. A redirect is an
error.data.redirectenvelope, so it takesget_proof's passthrough branch as the rustdoc claims — it is not silently flattened to-32004with the provider list discarded. - Two further mutations killed by their own named assertions: pinning
offset = 0inget_capsulefails atlib.rs:9309("a window must be served at the offset requested"); revertingstart == 0 && !chunk_lens.is_null()tostart == 0fails atlib.rs:9302(the chunk_lens-omitted assertion). Every mutation was md5-verified as a real on-disk byte change before its run. - Advertised-vs-implemented now matches for all five new names;
dig.getProofStatusis correctly absent from the catalogue anddig.listCapsulescorrectlypassthrough.
dig-constants check (mandatory, both directions)
Nothing in this diff belongs in dig-constants, and nothing here should be consuming it. The values introduced are JSON-RPC method-name strings and error codes already sourced from dig_rpc_protocol::Method and the crate's existing ROOT_NOT_ANCHORED / download::RESOURCE_UNAVAILABLE constants — no new cross-repo literal, address, port, asset id, or version pin is defined locally. WINDOW is pre-existing and out of scope.
13ef6de to
995b5d7
Compare
…-writable hint The melt gate is the authority for an irreversible, peer-triggered, network-correlated delete. Two cheaper signals were tried for it and both were unsound; this replaces the second with the singleton lineage itself, and the choice is settled by measurement against mainnet rather than by argument. What was wrong -------------- The previous cut concluded "melted" from a NON-EMPTY, all-spent `store_id` hint index. A hint is an unauthenticated CREATE_COIN memo over an arbitrary 32-byte value (#1473), so ANY party can place a record under ANY store's hint for the price of a dust coin. Enumerating all 53 DataLayer launcher coins on mainnet shows why that is fatal: 30 of the 53 LIVE stores have a completely EMPTY store_id hint index — their generations are not hinted to store_id at all. For every one of them a single planted spent coin makes the index non-empty and entirely spent, which the gate could not distinguish from a terminated lineage. Cost to erase a live store network-wide: dust plus fee, no permission, no P2P access, no key material. `run_melt_tick` would have fired it on a timer with no announcement at all. `get_coin_records_by_hint` is also truncatable, and truncation surfaces spent records first — the exact order that manufactures a false melt. What replaces it ---------------- A forward walk of the singleton lineage along real COIN PARENTAGE: 1. Identity + minted — the launcher coin whose `coin_id == store_id` exists and is SPENT. An unspent launcher is Live (not minted yet is the opposite of melted). This fact discriminates nothing by itself; it anchors where the walk starts. 2. Walk forward — follow the single ODD-amount child at each hop. An UNSPENT successor is Live. A spent coin with NO successor is Melted. A coin's `parent_coin_info` is fixed by which coin was actually spent to create it, so placing a coin anywhere in this walk requires spending a generation of the store, which requires the owner's authority. The walk is unwritable by anyone but the owner, and it never consults a hint — the mock panics if either hint query is touched. Fail-closed everywhere else: any transport error INCLUDING mid-walk (an outage must not read as "the lineage ended here"), more than one odd child, an absent launcher, and exceeding the hop ceiling. Zero children at hop 0 is Unknown, not a melt: a minted launcher always created the eve singleton, so an empty first hop means the answer is untrustworthy — which also closes the trap that `coin_records_by_parent_ids` has an empty DEFAULT impl on the trait. Measured against mainnet ------------------------ All 53 DataLayer stores: 51 Live, 1 Melted (the one genuinely terminated store, ending at hop 1), 1 previously mis-capped. Deepest live lineage 599 generations; 29 stores have their tip one hop from the launcher; no ambiguous fork anywhere. MAX_LINEAGE_HOPS is sized from that measurement. The four stores the gate named as live-with-empty-hint-index all classify Live here. Because the walk costs one read per generation and the receive path runs per inbound announcement, verdicts are memoised for a short TTL so a flood of announcements for one held store cannot multiply into repeated walks. A stale verdict can only DELAY a real melt, never cause a delete. Tests: 12 cases drive the real ChainReads trait with a crafted lineage, including the composition the gate flagged as untested and lethal — an empty hint index plus one planted spent coin — asserting Live. All ten inverting mutations of the gate were confirmed to fail their test; the hop-cap test asserts the EXACT read count, because a `<=` bound is also satisfied by a walk that stops far too early. root [workspace.package].version 0.94.0 -> 0.96.0 (minor, new capability). Skips 0.95.0, which PR #179 holds. Co-Authored-By: Claude <noreply@anthropic.com>
07eb9d2 to
afec8b5
Compare
…-writable hint The melt gate is the authority for an irreversible, peer-triggered, network-correlated delete. Two cheaper signals were tried for it and both were unsound; this replaces the second with the singleton lineage itself, and the choice is settled by measurement against mainnet rather than by argument. What was wrong -------------- The previous cut concluded "melted" from a NON-EMPTY, all-spent `store_id` hint index. A hint is an unauthenticated CREATE_COIN memo over an arbitrary 32-byte value (#1473), so ANY party can place a record under ANY store's hint for the price of a dust coin. Enumerating all 53 DataLayer launcher coins on mainnet shows why that is fatal: 30 of the 53 LIVE stores have a completely EMPTY store_id hint index — their generations are not hinted to store_id at all. For every one of them a single planted spent coin makes the index non-empty and entirely spent, which the gate could not distinguish from a terminated lineage. Cost to erase a live store network-wide: dust plus fee, no permission, no P2P access, no key material. `run_melt_tick` would have fired it on a timer with no announcement at all. `get_coin_records_by_hint` is also truncatable, and truncation surfaces spent records first — the exact order that manufactures a false melt. What replaces it ---------------- A forward walk of the singleton lineage along real COIN PARENTAGE: 1. Identity + minted — the launcher coin whose `coin_id == store_id` exists and is SPENT. An unspent launcher is Live (not minted yet is the opposite of melted). This fact discriminates nothing by itself; it anchors where the walk starts. 2. Walk forward — follow the single ODD-amount child at each hop. An UNSPENT successor is Live. A spent coin with NO successor is Melted. A coin's `parent_coin_info` is fixed by which coin was actually spent to create it, so placing a coin anywhere in this walk requires spending a generation of the store, which requires the owner's authority. The walk is unwritable by anyone but the owner, and it never consults a hint — the mock panics if either hint query is touched. Fail-closed everywhere else: any transport error INCLUDING mid-walk (an outage must not read as "the lineage ended here"), more than one odd child, an absent launcher, and exceeding the hop ceiling. Zero children at hop 0 is Unknown, not a melt: a minted launcher always created the eve singleton, so an empty first hop means the answer is untrustworthy — which also closes the trap that `coin_records_by_parent_ids` has an empty DEFAULT impl on the trait. Measured against mainnet ------------------------ All 53 DataLayer stores: 51 Live, 1 Melted (the one genuinely terminated store, ending at hop 1), 1 previously mis-capped. Deepest live lineage 599 generations; 29 stores have their tip one hop from the launcher; no ambiguous fork anywhere. MAX_LINEAGE_HOPS is sized from that measurement. The four stores the gate named as live-with-empty-hint-index all classify Live here. Because the walk costs one read per generation and the receive path runs per inbound announcement, verdicts are memoised for a short TTL so a flood of announcements for one held store cannot multiply into repeated walks. A stale verdict can only DELAY a real melt, never cause a delete. Tests: 12 cases drive the real ChainReads trait with a crafted lineage, including the composition the gate flagged as untested and lethal — an empty hint index plus one planted spent coin — asserting Live. All ten inverting mutations of the gate were confirmed to fail their test; the hop-cap test asserts the EXACT read count, because a `<=` bound is also satisfied by a walk that stops far too early. root [workspace.package].version 0.94.0 -> 0.96.0 (minor, new capability). Skips 0.95.0, which PR #179 holds. Co-Authored-By: Claude <noreply@anthropic.com>
…le locally WIP for the #2071 method sweep — parked while PR #177 (the P0 envelope fix) is re-gated. Co-Authored-By: Claude <noreply@anthropic.com>
getCapsule/getModule joined getProof/getMetadata/getPublicManifest as served=local, so the guard now pins what genuinely REMAINS unserved — listCapsules (needs a chain generation walk) and getProofStatus (polls an execution-proof job this node does not run) — and records why each is honestly unserved rather than merely unwritten. Co-Authored-By: Claude <noreply@anthropic.com>
Rebased onto the #177 re-gate fixes. inclusion_proof is now always emitted (as "" when absent), so the capsule test's absence assertion becomes an empty-string assertion: a capsule window has no per-resource proof and says so explicitly rather than by omission. chunk_lens keeps its omission, and the guard returns with the caller that motivated it. The two are deliberately different: empty-vs-absent distinguishes two states of an APPLICABLE field (inclusion_proof), while chunk_lens is INAPPLICABLE to a whole module and absent says exactly that. Co-Authored-By: Claude <noreply@anthropic.com>
Minor, not patch: getProof/getMetadata/getPublicManifest/getCapsule/getModule are new served capability on the public read tier. Co-Authored-By: Claude <noreply@anthropic.com>
…er-reachable Absence from the Method catalogue also filters it off the permissionless peer surface (is_peer_reachable_method ends in Method::from_name(..).is_some_and(..)), so this arm serves loopback/in-process/gateway only. Promoting the method into dig-rpc-protocol later must decide is_peer_reachable() deliberately rather than inherit it. Co-Authored-By: Claude <noreply@anthropic.com>
Security gate B1/N1/N3/N4. WIP checkpoint — see the follow-up commit for tests. Co-Authored-By: Claude <noreply@anthropic.com>
… dropped program_hash SPEC.md still listed getCapsule/getProof as passthrough and omitted getMetadata/getPublicManifest; the capsule-window docs in lib.rs and meta.rs still claimed no inclusion_proof rides a capsule window, which content_window_envelope contradicts (it is present, empty). Bumped to 0.95.0 — 0.94.0 was taken by #149. Co-Authored-By: Claude <noreply@anthropic.com>
…sifiable The empty-proof guard was a false green: the only test aiming at it made the INNER read fail, which returns at the passthrough branch one level above, so 'if proof.is_empty()' -> 'if false' left all 676 tests passing. The reduction is now a pure function (proof_from_content_answer) driven directly with a SUCCESSFUL read that carries no proof - the one shape where a blank could be dressed as a result. getCapsule gains a bytes-READ guard. Both implementations return byte-identical responses, so no correctness assertion can distinguish a windowed read from a whole-module slurp; the amplification IS the defect. A far-past-EOF request must read ~0, not 128 MiB to discover it has nothing to send. Also pins dig.getPublicManifest as non-peer-reachable, so promoting it into dig-rpc-protocol has to be a deliberate is_peer_reachable() decision. Co-Authored-By: Claude <noreply@anthropic.com>
…laky A single global counter passed in isolation and FAILED in the full suite: tests run in parallel and the read happens on a spawn_blocking thread, so neither a global nor a thread-local can attribute a read to the test that caused it. Per-root attribution can. A flaky guard is worse than none - it gets deleted. Co-Authored-By: Claude <noreply@anthropic.com>
…oc list lint The memoization commit inserted DATA_SECTION_MEMO_CAP between read_public_manifest_blocking's doc comment and its function, so the doc was orphaned onto the const and the function had none. Clippy caught it as doc_lazy_continuation - the const's prose was being parsed as an unindented continuation of the function's Returns list. Fixed structurally rather than by indenting: the doc goes back on its function (updated to note the read is now memoized), and the const gets its own. Co-Authored-By: Claude <noreply@anthropic.com>
…ame from The blob memo was strictly worse than the whole-file read it replaced. FIXED_BLOB_LEN pads every DIGS blob to 128 MiB and ChunkPool is a section INSIDE it, so an entry was ~99% content, not metadata - my doc claimed the opposite. Capped in ENTRIES at 256, that bounds at ~32 GiB, and it is RETAINED for the life of the process where the old cost was transient and needed concurrency. cache.listCached is requires_auth:false, so an attacker enumerates the exact capsule list and pins ~16 of them to exhaust a 2 GiB host. Now caches the two DECODED manifests, which genuinely are kilobytes, so an entry cap is a sound bound. Cold extractions are serialized by a single-flight lock: it coalesces duplicate work (N concurrent requests for one capsule cost one read) and bounds transient memory to one extraction's ~385 MiB rather than N of them. Residency is invisible to every functional assertion - a blob cache returns byte-identical manifests - so a size guard was added. Falsified by re-adding the blob to the entry: it reports 134218536 bytes retained and fails. Co-Authored-By: Claude <noreply@anthropic.com>
9030c04 to
39dea76
Compare
Task
Serve the read methods dig-node classified as "passthrough aliases" — relayed to an upstream, and
therefore
-32601on any node that has none. rpc.dig.net is meant to be an ordinary node (#1997),so it has none, and every client calling these got method-not-found from a node that held the bytes.
Second half of DIG-Network/dig_ecosystem#2071. #177 is merged (
95141b7) and this is now rebaseddirectly onto
main— no longer stacked.Scope note, since this was briefly mis-sequenced: #177 is what restores
*.on.dig.net. This PRdoes not, and never could — the deployed resolver's only RPC call is
dig.getContent(
grep -o 'rpcCall("[^"]*"' sw.js dig-embed.jsreturns exactly that one). This PR matters becauserpc.dig.net is an ordinary node and OTHER clients — the hub, the extension, the CLI — do call these.
Methods now served locally
dig.getProof{ inclusion_proof, root, chunk_lens, program_hash, execution_proof: null, execution_proof_status: "unavailable" }dig.getMetadata{ manifest | null, program_hash, root }— publisher metadata, data-section id 6dig.getPublicManifest{ manifest | null, root }— the enveloped form ofdig.getManifestdig.getCapsule/dig.getModule.digmodule, in the getContent envelopedig.getProofis served genuinely, not stubbedThe constraint on #2071 is explicit: a proof that does not verify is worse than
-32601,because the client would then decrypt unverified bytes with no error raised anywhere.
So
dig.getProofdoes not derive a proof of its own. It runs the ordinarydig.getContentread— same mandatory anchored-root pin (§14.4), same local-first → peer → upstream ladder — and discards
the ciphertext. That makes the proof it returns provably the proof a content read of the same
resource would have verified against; a second derivation could pin a different generation and no
client could tell. The proof itself is computed by the module's own guest wasm inside
serve_blind.When no proof can be produced, the underlying read's error is returned verbatim — never a
proof-shaped result with an empty
inclusion_proof.The test decodes the returned proof and checks it, rather than asserting it is non-empty:
The old guard was rewritten, not deleted
get_proof_is_not_served_as_a_verified_proof_by_the_nodeasserteddig.getProofmust be-32601."Never fabricate a proof" was always the invariant; "never implement the method" was only the
cheapest way to hold it, and it cost every client the ability to re-verify bytes it already held.
The guard now asserts the invariant directly: an unobtainable proof is an error, never a blank
that looks like a proof. No execution attestation is fabricated —
execution_proof: nullwithexecution_proof_status: "unavailable"(#126/#134, SECURITY.md residual #3).Security property found while reviewing this, now recorded in the code
dig.getPublicManifestis dispatched by NAME (it is not yet adig_rpc_protocol::Methodvariant).That absence from the catalogue is what keeps it off the permissionless peer surface —
is_peer_reachable_methodends inMethod::from_name(m).is_some_and(|m| m.is_peer_reachable()), soan unknown name is filtered before the peer surface reaches this dispatch. It therefore serves the
loopback / in-process / gateway surface only.
Promoting the method into
dig-rpc-protocollater MUST make a deliberateis_peer_reachable()decision rather than inherit one, or a gateway-only public read silently widens to the whole peer
network. That reasoning is in
dispatch.rs, not just here.Anti-rollback applies to capsule-scoped reads too
rootmay be omitted or"latest", which every client uses to mean "whatever the chain says" — thenode resolves the tip itself rather than making the caller walk the singleton. An explicitly
requested root MUST equal the anchored tip or the read fails closed with
-32005, with the same#747/#841 bounded-verify tolerance the content read has. Tested across all three capsule-scoped
methods.
Still NOT served, deliberately
dig.listCapsules— needs a chain generation walk this node does not perform.dig.getProofStatus— polls an execution-proof JOB this node does not run; inventing a statuswould be the fabrication above.
Both stay
-32601and the passthrough guard now pins exactly those two, with the reasoning recordedin the test.
Bump
0.93.9 → 0.94.0 (minor — new served capability on the public read tier);
dig-node-core0.41.1 → 0.42.0.
Refs DIG-Network/dig_ecosystem#2071