fix(server): close the connection a device reconnect displaced - #6891
Open
otavio wants to merge 1 commit into
Open
fix(server): close the connection a device reconnect displaced#6891otavio wants to merge 1 commit into
otavio wants to merge 1 commit into
Conversation
A device that reconnects before the server notices its previous socket is gone registers a second connection under the same identifier. Only the last one is ever dialed, so the earlier one is dead weight holding its socket and buffers until its own ping loop times out -- which is up to 35s away, and never arrives at all for a peer that vanished without resetting the connection. The manager already detected this and only logged it. Close the displaced connections on registration, and report a device offline only when the teardown was of its last connection, so evicting a stale connection no longer marks a device that is online offline. `SyncSliceMap` had to become consistent first. `Store` and `Delete` were read-modify-write over a `sync.Map`, so two concurrent registrations for one key could drop one -- leaving a live connection the manager never recorded and so never tears down. Guard it with a mutex and have both report what they observed under the lock: `Store` returns the values it displaced, `Delete` the count that survived. Reading either back afterwards would race with the next caller. `Delete` now also drops the key once its last value is gone, where before an entry stayed behind for every device ever seen. Refs: shellhub-io/team#202
|
Claude finished @otavio's task in 5m 42s —— View job Code Review CompleteReviewed 4 files across code quality, security, testing, language patterns, and architecture — no new issues found. The code looks good as-is. A few things I specifically verified, since they carry the correctness of the fix:
To request another review round, comment --- · |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stale device connections are now torn down when the device reconnects, instead of being
detected, logged and left resident. The connection map they live in became consistent
under concurrent access, which it had to be before eviction could be built on it.
Why
Investigating a managed deployment (58,670 devices, 51,187 online) turned up the
sshservice logging
"Multiple connections stored for the same identifier."with"size":2at175 occurrences/hour. The manager detected the duplicate and only warned about it.
A device that reconnects before the server has noticed its previous socket is gone
registers a second connection under the same identifier. Only the last one is ever dialed,
so the earlier one is dead weight: it holds its socket and buffers until its own ping loop
times out — up to 35s away, and never at all for a peer that vanished without resetting the
connection.
The wider "104k connections for 58k devices" claim in the issue turned out to be a
measurement artifact (a gateway netns counts both legs of every proxied connection), so
this is not the multi-gigabyte win that issue proposed. These are the two real defects that
survived the analysis. Refs: shellhub-io/team#202 — cross-repo, so it will not auto-close.
Changes
SyncSliceMap:StoreandDeletewere read-modify-write over async.Map. Twoconcurrent registrations for one key could drop one, leaving a live connection the
manager never recorded and therefore never tears down. Guarded with a mutex, and both
now report what they observed under the lock —
Storereturns the values it displaced,Deletethe count that survived. Returning them is not a convenience: reading eitherback afterwards would race with the next caller, which is the bug being fixed.
SyncSliceMap.Delete: drops the key once its last value is gone. Previously anentry stayed behind for every device the server had ever seen.
Manager.evict: closes the connections a registration displaced, in bothSet(v1revdial) and
Bind(v2 yamux). It runs in its own goroutine — this is teardown of analready-abandoned connection, and
revdial.Dialer.Closeblocks until its owner observesthe close. Neither belongs on the path registering the live connection.
Offline reporting:
DialerDoneCallbacknow fires only whenDeletereports noconnections remaining. Without this, evicting a stale connection would mark a device that
had just reconnected as offline. It also fixes the pre-existing, rarer flap where a stale
connection dying on its own timeout did the same.
Testing
go test -race ./ssh/...andgolangci-lint run ./ssh/...are clean.Worth a reviewer's attention:
evict(close removed) andfails on both assertions there, so it is not vacuous.
TestConcurrentStoreAndDeleteReportOneWinnerpins the contract the offline decisiondepends on: across 50 concurrent teardowns of one key, exactly one
Deletemay observeit empty. If that ever reports more than one, devices get spurious offline writes.
size > 1warning on theDialpath was deliberately kept. With eviction in place itshould become rare enough to be a real signal rather than the background noise it is now
— it is the cheapest way to tell whether this actually worked in production.