bridges: give each concurrent process its own Machine slot - #43
Open
guygrigsby wants to merge 8 commits into
Open
guygrigsby wants to merge 8 commits into
guygrigsby wants to merge 8 commits into
Conversation
A make install or go build binary reported a commit height (B29), so a running binary could not be matched to the release it came from. The goreleaser build already stamped the tag, so the height scheme only ever showed on source builds, which is every build a dev runs. git describe gives the tag on a release commit and the tag plus distance past it otherwise, for both the Makefile ldflags and the init() fallback plain go build exercises. Keeping B-numbers would have cost nothing to write and stayed unmatchable against the release list forever.
Every existing test exercises internals in-process; nothing executed the shipped binary, so the flag-before-TUI ordering, the PTY handoff to a launched agent, and the return to the picker after the child exits were all untested at the level users hit them. Four tests against the real build: -version, a bad endpoint failing before the TUI takes the terminal, the Pi happy path against a fake Aperture (httptest /v1/models) with a stub pi on PATH recording argv and capturing the generated provider extension while it exists, and the unreachable-endpoint banner. Runs hermetically: temp HOME/XDG, PATH of the stub dir plus system dirs only so host agent binaries cannot leak into the picker. creack/pty is promoted from the module graph; no new third-party code. TERM=dumb because startup asks the terminal its color profile and a PTY never answers — with xterm each run paid the five-second query timeout. Skipped: bridge flows (need a control plane), install flows (network), Windows PTYs.
APT-330: parallel aperture sessions in bridge mode shared one state directory per bridge, so every process registered the same node key and the control plane handed the session to the newest registrant. The older processes looked Running locally while their dials reached nothing, surfacing as hangs and "api error", and the shared directory was being written concurrently on top. A Machine's identity is now numbered by slot. Slot 1 keeps the existing state directory and hostname so authorized devices carry over; slots 2 and up get suffixed siblings. A process claims the lowest free slot with an exclusive non-blocking lock on a lock file under bridges/locks/ when it builds the node and holds it until the Machine closes or is destroyed. Lock files live outside the state directory so removal never deletes an open lock and a claimant's inode cannot be deleted under it. Machines.Destroy logs out every slot the bridge has on disk and fails before any logout when a live process holds one, naming the conflict. Evicting a running session is the silent kill this change removes, not a removal feature. Rejected the shared daemon (a wire protocol, trust boundary and daemon lifecycle for a first iteration) and ephemeral per-process nodes (key expiry forces a browser login per launch). ADR 0006 records the call and the revisit conditions.
ADR 0006 makes every concurrent process its own device, so a fresh slot needed one browser login each before it could join. With TS_AUTHKEY set, tsnet registers the slot non-interactively; a reusable key covers every slot an agent fleet claims. Once a slot has registered, its state directory carries the credentials and the key is not consulted again for it.
APT-330 taught users the hard way that concurrent sessions evict each other; the fix (ADR 0006) and the TS_AUTHKEY escape hatch for unattended sessions were only discoverable by reading the code.
The flags table only names the two vars that mirror flags; TS_AUTHKEY, the Codex install vars and the incidental reads were undocumented.
The suite skipped bridge flows on the assumption they need a control plane. The slot contract turned out to be local: state dirs, held flocks, and menu screens, all observable without a login ever completing. Four tests, one per ADR 0006 decision: concurrent processes claim distinct numbered slots, a dead process's slot is reclaimed before a new one is minted, removal fails naming the conflict while a slot is live and destroys every slot once it is gone, and the 100-slot cap errors instead of probing. The cap test holds the 100 locks itself because an exclusive flock is exactly what a live process presents, which keeps 101 real processes out of the run. Two harness additions this needed: spawn now reaps parked processes via cleanup (a bridge waiting on login never exits on its own), and hermeticEnv drops TS_AUTHKEY and forces BROWSER=true (bridge attempts reach the login-link phase, so a dev box would otherwise authorize fresh slots against a real tailnet or pop browser tabs during make check). Not covered: two sessions actually reaching Aperture through their slots. That is the data plane and does need credentials the test environment does not have.
guygrigsby
force-pushed
the
guygrigsby/APT-330/bridge-concurrency
branch
from
September 22, 2026 20:44
abf4b65 to
b97d3d6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallel aperture sessions in bridge mode evicted each other (APT-330). Every process opened the same state directory for a bridge, so every process registered the same node key, and the control plane handed the session to the newest registrant. The older processes showed Running locally while their dials reached nothing, which surfaced as hangs and "api error". The shared directory was also being written concurrently.
This PR solves the bridge concurrency problem with the naive approach of one tailnet machine per agent.
This PR numbers a bridge's Machine identity by slot. Slot 1 keeps the existing state directory and hostname, so devices users already authorized carry over. Slots 2 and up get suffixed siblings:
bridges/<hex>-Non disk andaperture-cli-<bridgeID>-Nin the admin console. A process claims the lowest free slot when it builds the node and holds it until the Machine closes or is destroyed. The claim is an exclusive non-blocking lock onbridges/locks/<hex>-<slot>.lock,flockon unix andLockFileExon Windows. Locks live outside the state directory so removal never deletes an open lock.Machines.Destroynow logs out every slot the bridge has on disk. It refuses, before any logout, when another live process holds a slot and names the conflict. The removal screens list every device the bridge registered, the way the admin console shows them.TS_AUTHKEY, when set, authorizes a fresh slot without the browser login. Once a slot has registered, its state directory carries the credentials and the key is not consulted again for it. The README gains a section on concurrent sessions and a table of every environment variable the launcher reads. ADR 0006 records the decision and why a shared daemon and ephemeral per-process nodes were rejected.Risks:
TS_AUTHKEYis set. Agent fleets need a reusable key or each slot asks once. The key decides which tailnet the fresh device joins, so a key from the wrong tailnet registers the device there.Destroythat returns at its deadline while the node'sClosehangs keeps slot 1 locked until the close finishes. A reopen in that window takes slot 2, a fresh device and a fresh login, where it used to wait on the stuck close.