fix(node): make peer bring-up honest and add a peer-ping ladder diagnostic - #149
Draft
MichaelTaylor3d wants to merge 2 commits into
Draft
fix(node): make peer bring-up honest and add a peer-ping ladder diagnostic#149MichaelTaylor3d wants to merge 2 commits into
MichaelTaylor3d wants to merge 2 commits into
Conversation
Co-Authored-By: Claude <noreply@anthropic.com>
The mTLS peer-RPC listener was bound only after bring_up_dht returned, and bring_up_dht ended by announcing the node's whole inventory into the DHT one content id at a time. Each announce_provider is an iterative Kademlia lookup plus a PUT at the k closest peers, every RPC bounded by the 5s DHT timeout, and they ran against the empty routing table bootstrap leaves behind -- so they ground to the timeout instead of resolving. A node holding 44 capsules announces 68 content ids, which cost 12m40s. For that whole window the node had no listener on 9444 and answered no DHT query, while holding a relay reservation that advertised it as up, and logged nothing at all. It was diagnosed as a crashed bring-up twice. The storage backend was not the cause: the fleet nodes that complete in ~30s have EMPTY caches, so they announce nothing. An EBS-backed node holding 44 capsules would have stalled identically. - Record the inventory content ids on the DhtHandle during bring-up (cheap: a readdir plus one stat per capsule) but move the network announce into a background task started after the pool->routing feed is live, so each PUT has a filling routing table to work against. - Announce with bounded concurrency instead of one at a time, so the cost stops being linear in how much content the node holds -- the tier-0 eager cache (#1934) deliberately grows holdings. - Log the announce start, progress, and completion. A 13-minute silence is indistinguishable from a hang; even unchanged timing, said out loud, is a different operational experience. - Run cache_list_cached on a blocking thread. It is readdir + stat per capsule, and on the S3-backed cache each of those is a round trip that was parking a tokio worker. Refs DIG-Network/dig_ecosystem#1974 Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Author
|
Lane paused on a §1.4 single-writer conflict: another lane now owns Everything is pushed at Overlap to be aware of: this branch relocated the Full resume-ready detail: DIG-Network/dig_ecosystem#1974 and DIG-Network/dig_ecosystem#1985. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP lane for dig_ecosystem #1974 (bring-up visibility / listener ordering) and #1985 (ping-node ladder diagnostic).
Closes DIG-Network/dig_ecosystem#1974
Closes DIG-Network/dig_ecosystem#1985