diff --git a/docs/superpowers/plans/2026-09-06-shared-hyperd-daemon.md b/docs/superpowers/plans/2026-09-06-shared-hyperd-daemon.md new file mode 100644 index 0000000..821bf40 --- /dev/null +++ b/docs/superpowers/plans/2026-09-06-shared-hyperd-daemon.md @@ -0,0 +1,655 @@ +# Shared `hyperd` Daemon — Phased Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: use +> `superpowers:subagent-driven-development` and execute this plan phase by +> phase. Steps use checkbox (`- [ ]`) syntax for tracking. The main thread owns +> plan revision, commits, final validation, and merge-readiness judgment. +> +> **This plan is not approved.** It accompanies a design *exploration*, and its +> design spec recommends against building most of it in the near term. Phase 1 +> is recommended unconditionally. Phases 2 and 3 are gates. Phases 4 onward +> should not start until those gates pass and a human has answered the decision +> points in the "Decisions a human must make" section below. + +**Goal:** Lift the resident `hyperd` daemon out of `hyperdb-mcp` into a crate +whose job it is, narrowing `hyperdb-mcp`'s accidental public surface before the +`1.0.0` freeze; then, only if measurement and a security floor justify it, +offer cohort-scoped shared-engine acquisition to any `hyperdb-api` consumer. + +**Architecture:** A new `hyperdb-daemon` crate sitting **above** `hyperdb-api`, +mirroring `hyperdb-bootstrap`'s library-plus-`cli`-feature shape. It owns +discovery, the control protocol, the supervisor, and the daemon binary. +`hyperdb-mcp` depends on it and keeps all of its own policy. `hyperdb-api` +gains nothing in the minimum viable slice, because +`Connection::connect(endpoint, …)` already covers connecting to a `hyperd` +someone else owns. + +**Tech stack:** Rust workspace, edition 2024, MSRV 1.88; `hyperdb-api`; +`hyperdb-mcp`; `serde`/`serde_json` for the discovery record; `clap` behind a +`cli` feature; real `hyperd` via `HYPERD_PATH=~/dev/bin/hyperd`; Conventional +Commits; release-please owns all versions. + +**Design specification:** +[`docs/superpowers/specs/2026-09-06-shared-hyperd-daemon-api-design.md`](../specs/2026-09-06-shared-hyperd-daemon-api-design.md) + +**Base:** `upstream/main` @ `2f31b9e` +**Branch (design only):** `docs/shared-hyperd-daemon-design` +**Branch (implementation):** not created; each phase below should get its own. + +--- + +## Phase dependency map + +```mermaid +graph TD + P1["Phase 1 — Extract hyperdb-daemon
PREREQUISITE, deadline 1.0.0"] + P2["Phase 2 — Measure the benefit
GATE"] + P3["Phase 3 — Security floor + #242
GATE"] + P4["Phase 4 — Cohort-scoped acquire()
the feature itself"] + P5["Phase 5 — Lease refcounting
OPTIONAL"] + P6["Phase 6 — Test-harness integration
OPTIONAL"] + P7["Phase 7 — hyperdb-api trait seam
OPTIONAL, decide before 1.0.0"] + + P1 --> P4 + P2 --> P4 + P3 --> P4 + P4 --> P5 + P4 --> P6 + P4 -.-> P7 + style P1 stroke:#50c878,stroke-width:2px,fill:none + style P2 stroke:#e8a33d,stroke-width:2px,fill:none + style P3 stroke:#d64545,stroke-width:2px,fill:none + style P7 stroke:#888,stroke-dasharray:4,fill:none +``` + +| Phase | Status | Blocks | Deadline | +|---|---|---|---| +| 1 — Extract `hyperdb-daemon` | **Prerequisite** | 4 | **`1.0.0`** (removes `pub mod daemon`) | +| 2 — Measure benefit | **Gate** | 4 | none | +| 3 — Security floor and #242 | **Gate** | 4 | none | +| 4 — Cohort-scoped `acquire()` | Feature | 5, 6 | none, land after `1.0.0` | +| 5 — Lease refcounting | Optional | — | none | +| 6 — Test-harness integration | Optional | — | none | +| 7 — `hyperdb-api` trait seam | Optional | — | **`1.0.0`** if chosen at all | + +**Phase 1 is worth executing even if every other phase is rejected.** It +answers open issue #276 and its cost rises at the freeze. + +--- + +## Global constraints + +These apply to every phase and every agent. + +- Read and obey [`AGENTS.md`](../../../AGENTS.md) and the design spec before + editing. Search the whole repository before concluding a surface is absent. +- **Never invent `hyperd` flags or engine parameters.** Obtain `hyperd` via + `make download-hyperd`; start servers through `HyperProcess::new` or the + Makefile targets. Confirm any flag against `hyperd --help` first. Fabricated + parameters have previously made hanging tests look green. +- **A command is green only when its real output and zero exit status were + seen.** No output for roughly 30 s is a hang/failure requiring + investigation, not a pass. These tests start a real `hyperd` subprocess, so a + misconfigured server hangs rather than erroring cleanly. +- Hyper-backed commands use `HYPERD_PATH=~/dev/bin/hyperd`, or go through + `make test` which sets it. +- No narrowing integer `as` casts. Use `TryFrom` with an explicit + error/panic policy. Convert any you touch, even if unrelated to the change. +- Do not edit crate versions, `version.txt`, `.release-please-manifest.json`, + `release-please-config.json`, or the root `CHANGELOG.md`. No `Release-As:` + footers. +- Run `npx markdownlint-cli2` with **no arguments** before every commit that + touches Markdown. Baseline is **0 issues in 68 files**. A nested + `.markdownlint.json` must `extends` the root config. +- Developer and tester agents do not commit. The main thread stages explicit + paths and makes the Conventional Commit after an independent reviewer has no + unresolved Critical or Important finding. +- Role separation is mandatory: **doer ≠ validator ≠ merger.** + +## Execution protocol + +For every behavioural task: + +1. A **tester** agent owns the named test files and proves each planned + assertion fails against the current branch for the intended reason. Capture + the harness's nonzero executed-test count; a Cargo filter reporting zero + tests is a **failed gate** even when Cargo exits zero. Use `-- --exact` when + selecting one fully qualified test. Where a genuinely new interface makes + the first red a compiler error, capture that nonzero compiler failure first, + then add the smallest signature seam and rerun for an executed failing + assertion. +2. An **engineer** agent owns the named production files and makes the + proven-red test green with the smallest conforming change. +3. The engineer runs the focused suite plus: + + ```bash + cargo fmt --all --check + cargo clippy --workspace --all-targets --all-features -- -D warnings + ``` + +4. A fresh read-only **reviewer** receives the spec section, the complete + diff, and the captured red/green/lint output, and reports + Critical/Important/Minor plus a merge verdict. +5. Critical/Important findings return to a fresh engineer, then a fresh + re-review. The main thread independently verifies every claimed fix. + +**Pure-move refactors are an explicit exception to red-before-green.** Phase 1 +is a relocation, so its evidence is that the *existing* daemon test suite +passes unchanged against the new crate, plus a compile-fail assertion that the +old path is gone. Those are characterizations and must be reported as such, +never dressed up as red tests. + +## File map + +| Area | Primary files | Phase | +|---|---|---| +| New crate skeleton | `hyperdb-daemon/Cargo.toml`, `src/lib.rs`, `src/main.rs`, root `Cargo.toml` | 1 | +| Moved daemon modules | `hyperdb-daemon/src/{discovery,health,run,spawn}.rs` | 1 | +| MCP rewiring | `hyperdb-mcp/src/{lib,main,engine,server,watcher,diagnostics}.rs`, `Cargo.toml` | 1 | +| Moved daemon tests | `hyperdb-daemon/tests/`, `hyperdb-mcp/tests/daemon_tests.rs` | 1 | +| Startup benchmark | `hyperdb-api/benches/` or `hyperdb-daemon/benches/`, `docs/BENCHMARK_GUIDE.md` | 2 | +| Transport and permissions | `hyperdb-daemon/src/{discovery,health,run}.rs`, `hyperdb-api/src/process.rs` | 3 | +| Wedged-engine recovery | `hyperdb-daemon/src/run.rs` | 3 | +| Acquisition API | `hyperdb-daemon/src/{acquire,cohort,options}.rs` | 4 | +| Leases | `hyperdb-daemon/src/{health,run}.rs` | 5 | +| Test harness | `hyperdb-api/tests/common/mod.rs`, `hyperdb-api-core/tests/common/mod.rs` | 6 | +| Trait seam | `hyperdb-api/src/{connection,process}.rs` | 7 | + +--- + +## Phase 1 — Extract `hyperdb-daemon` (prerequisite; deadline `1.0.0`) + +**Why now:** answers open issue #276, which asks whether +`hyperdb_mcp::daemon::health` should be public before 1.0. Removing +`pub mod daemon` from `hyperdb-mcp` after `1.0.0` is a major version. Doing it +before is free. This phase stands alone. + +**Scope discipline:** this is a **move**, not a redesign. Do not fix bugs, do +not rename functions, do not change the wire format. Any improvement noticed +along the way becomes an issue, not a diff hunk. A refactor that also changes +behaviour cannot be reviewed as either. + +### 1.1 Create the crate and move the modules + +- [ ] Add `hyperdb-daemon` to the root `Cargo.toml` workspace members. Use + `version.workspace = true` — never hand-edit a crate version. +- [ ] Mirror `hyperdb-bootstrap`'s manifest shape exactly: + `default = ["cli"]`, `cli = ["dep:clap", …]`, so the crate is usable as a + pure library with `default-features = false`. Confirm the real feature + list against + [`hyperdb-bootstrap/Cargo.toml`](../../../hyperdb-bootstrap/Cargo.toml) + rather than copying this line. +- [ ] Depend on `hyperdb-api`. **Verify no cycle** — `hyperdb-api` must not + gain a dependency on `hyperdb-daemon`, now or later. +- [ ] Move `hyperdb-mcp/src/daemon/{discovery,health,run,spawn,mod}.rs` to + `hyperdb-daemon/src/`, preserving inline test modules verbatim. +- [ ] Move the daemon binary entry point out of `hyperdb-mcp/src/main.rs` into + `hyperdb-daemon/src/main.rs`, behind the `cli` feature. +- [ ] Replace MCP-branded identifiers with injected values: the + `PONG_TOKEN = "hyperdb-mcp"` and the `MCP_VERSION`-derived + `DaemonBuildIdentity` become constructor parameters, so `hyperdb-mcp` + supplies its own and the crate has no product branding. **This is the one + permitted behaviour-adjacent change in Phase 1** — the token must remain + byte-identical on the wire for MCP callers, and a test must pin that. +- [ ] Leave `HYPERDB_STATE_DIR`, `HYPERDB_DAEMON_PORT`, and + `HYPERDB_DAEMON_IDLE_TIMEOUT` spelled exactly as they are. They are a + public interface. + +### 1.2 Rewire `hyperdb-mcp` and narrow its surface + +- [ ] Update `engine.rs`, `server.rs`, `watcher.rs`, `diagnostics.rs`, and + `main.rs` to use `hyperdb_daemon::*`. +- [ ] **Delete `pub mod daemon;` from `hyperdb-mcp/src/lib.rs`.** This is the + point of the phase. +- [ ] Decide, and record in the commit message, whether `hyperdb-mcp` + re-exports anything for compatibility. Recommendation: **no re-export.** + Issue #276 established that no downstream library consumer is known, and + a re-export preserves the surface this phase exists to remove. +- [ ] Confirm the MCP keeps every piece of policy: `AttachRegistry` replay, the + `"persistent"` alias and ephemeral scratch, `_table_catalog`, the KV + store, watched directories, the doctor, and its version-takeover UX. + Nothing policy-shaped moves into the new crate. + +### 1.3 Prove the move preserved behaviour + +- [ ] Move `hyperdb-mcp/tests/daemon_tests.rs` to `hyperdb-daemon/tests/`, + keeping every test name. Leave MCP-integration cases behind in + `hyperdb-mcp/tests/`. +- [ ] **Un-`ignore` the eight crash-and-restart tests on Linux and Windows** if + #279's `cfg_attr(target_os = "macos", …)` narrowing did not already reach + them in the new location. Do not silently re-broaden the ignore during the + move; that regression is exactly what #271 was. +- [ ] Add a characterization pinning the wire format across the move: + a `daemon.json` fixture written by `hyperdb-mcp` at `2f31b9e` must + deserialize byte-identically in the new crate, and the new crate's output + must deserialize in the old reader. **Report as a passing + characterization, not as red-before-green.** +- [ ] Add a test asserting `PING` still answers with the MCP's exact token when + the MCP supplies it, so 1.1's de-branding cannot change the wire. +- [ ] Run and capture: + + ```bash + cargo fmt --all --check + cargo clippy --workspace --all-targets --all-features -- -D warnings + HYPERD_PATH=~/dev/bin/hyperd cargo test -p hyperdb-daemon -- --nocapture + HYPERD_PATH=~/dev/bin/hyperd cargo test -p hyperdb-mcp + HYPERD_PATH=~/dev/bin/hyperd cargo test --workspace --exclude hyperdb-api-node --exclude hyperdb-bootstrap + npx markdownlint-cli2 + ``` + + Expected: all green, with the **same** daemon test names passing as before + the move and the previously-ignored eight now executing on Linux. Record + passed/failed/ignored counts before and after; an ignored count that grew is + a failed gate. + +### 1.4 Documentation and changelog + +- [ ] Create `hyperdb-daemon/README.md` (user-facing) and + `hyperdb-daemon/CHANGELOG.md` with a single `## [Unreleased]` section. +- [ ] Add the crate to the architecture diagram and the "Subdirectory guidance" + and layering sections of [`AGENTS.md`](../../../AGENTS.md), and to the + per-crate changelog list in reminder 8 — it becomes the **tenth** + hand-maintained changelog. +- [ ] **Changelog entries needed:** + - `hyperdb-mcp/CHANGELOG.md`, under the existing `## [Unreleased]` → + `### Removed`: the `daemon` module is no longer public. Append to the + existing heading if one is there; a second `### Removed` sibling trips + MD024. + - `hyperdb-daemon/CHANGELOG.md` → `### Added`: initial release. +- [ ] Reference issue #276 in the commit body so the extraction is traceable to + the question it answers. +- [ ] Commit: `refactor(daemon)!: extract the resident hyperd daemon into its own crate` + + Note the `!` position — immediately before the colon. `refactor!(daemon):` + does not match the Conventional Commits header regex release-please uses and + would silently produce no changelog entry. The `!` is warranted because + `hyperdb-mcp` loses a public module. + +**Human decision before this phase commits:** whether to re-export for +compatibility (1.2) and whether the breaking marker is acceptable inside an +`-rc` line. See "Decisions a human must make". + +--- + +## Phase 2 — Measure the benefit (gate) + +**Why it gates:** the design spec's D9 establishes that this repository has +**no** measured `hyperd` spawn-to-usable wall clock, and no `hyperd` memory +figure at all. Every justification for Phase 4 is currently an estimate. +`docs/BENCHMARK_GUIDE.md` and `docs/hyperd-release-benchmarks.md` were both +checked and neither contains one. + +This phase produces numbers, not features. It is cheap. + +### 2.1 Measure cold start + +- [ ] Time `HyperProcess::new(None, None)` followed by one trivial query to + first row. 20 iterations, report median and p95, **release build**. +- [ ] Repeat with `TransportMode::Tcp` and `TransportMode::Ipc` — P3 in the + spec notes UDS performance is unvalidated, and Phase 3 needs this number + anyway. +- [ ] Run on macOS and Linux. Record the exact host, `hyperd` version from + `hyperdb-bootstrap/hyperd-version.toml`, and whether the page cache was + warm. +- [ ] Do **not** assert on a duration in a test. The existing + `phase0_compile_check_spike.rs` prints timings under `--nocapture` and + asserts nothing; follow that pattern. A wall-clock assertion is a + flaky test on shared CI. + +### 2.2 Measure warm acquisition + +- [ ] Time `discover()` plus `Connection::connect` against an already-running + daemon, same iteration count. This is the number Phase 4's value + proposition rests on. +- [ ] Separately time the three degraded discovery paths, because they set + Phase 4's `fallback_deadline`: warm hit (roughly a file read plus one + PING), dead-port timeout (**300 ms** budget today), and a full 16-port + scan (**up to 4.8 s** today). +- [ ] Time cold daemon acquisition — no daemon resident, client must spawn one — + against `SPAWN_TIMEOUT` of 10 s. **If this exceeds private spawn, record + it plainly**; it is the #270 shape and it determines whether + `SharedOrPrivate` can ever be the recommended default. + +### 2.3 Measure memory + +- [ ] Sample RSS of one `hyperd` at idle and after a representative query. +- [ ] Sample total RSS for N = 1, 4, 16 concurrent `hyperd` processes. +- [ ] Note the interaction the spec flags: `memory_limit` defaults to **80 % of + host RAM** and is instance-global, so N processes each believe they may + use 80 % of the machine. + +### 2.4 Record and decide + +- [ ] Add a section to `docs/BENCHMARK_GUIDE.md` with the methodology and + figures. Do **not** add a row to + `docs/hyperd-release-benchmarks.md` — that file takes a row on a `hyperd` + pin bump or a material API change, and conflating a startup measurement + with it would make a future engine delta unattributable. +- [ ] Replace the estimate-labelled figures in the design spec's + "Quantified benefit" section with measured ones, keeping the + measured-versus-estimated labelling convention. +- [ ] **Changelog entries needed:** none. Docs and benches only. +- [ ] Commit: `docs: measure hyperd startup cost and memory footprint` +- [ ] **Gate decision:** does the measured win justify Phase 4? A human + decides. Record the answer and the reasoning in the spec. + +--- + +## Phase 3 — Security floor and wedged-engine recovery (gate) + +**Why it gates:** the design spec's D6 documents a confused-deputy privilege +escalation — a world-readable `daemon.json` publishing the endpoint of a +`--no-password` engine — plus an unauthenticated control port accepting `STOP`. +And P1 is #242: a `hyperd` that stays alive but stops serving is never +recovered, reproduced at over 30 s. + +None of this is acceptable for a standing machine-wide service that a library +enrolled the caller in without asking. + +### 3.1 Restrict the state directory and discovery file + +- [ ] Create `~/.hyperdb` (or `HYPERDB_STATE_DIR`) at `0700` on Unix; set an + equivalent restrictive ACL on Windows. +- [ ] Write `daemon.json` at `0600`, **setting the mode before the content + lands** — write to the temp file with the mode already applied, then + rename. Setting permissions after writing leaves a readable window. +- [ ] Preserve the existing atomicity: temp-file-then-rename with no + pre-delete. #278 removed the pre-delete after establishing that Windows + `rename` uses `MOVEFILE_REPLACE_EXISTING`; **do not reintroduce it.** +- [ ] Red test: a fixture directory created world-readable is corrected, and a + written discovery file has mode `0600`. Assert the real mode via + `PermissionsExt`, not via a proxy. + +### 3.2 Move the shared engine off loopback TCP + +- [ ] **Depends on Phase 2.1's UDS-versus-TCP figures.** `HyperProcess` + currently defaults to TCP "until UDS performance is validated" and the + daemon forces TCP explicitly. This step cannot land on assertion. +- [ ] If UDS holds up: use a Unix domain socket in a `0700` directory with a + `0600` socket, and a named pipe with an explicit DACL on Windows, so peer + identity is enforced by the OS. +- [ ] If TCP must be retained: verify peer UID via + `SO_PEERCRED`/`LOCAL_PEERCRED` on every control connection and reject + mismatches. +- [ ] Red test: a connection from a different UID is refused. **This needs a + real second UID**, so it is likely a documented manual verification with + captured output rather than a CI test. Say so rather than writing a test + that cannot fail. + +### 3.3 Token-gate destructive control commands + +- [ ] Generate a token per daemon; store it in the now-`0600` discovery file so + file permission *is* the authorization. +- [ ] Require it on `STOP` and `REPORT_HYPERD_ERROR`. Leave `PING` and `STATUS` + open — the doctor's 300 ms network-phase budget depends on cheap + unauthenticated probes. +- [ ] Bump the control-protocol version (see 3.5). +- [ ] Red test: `STOP` without a token is refused and the daemon survives; + `STOP` with the token succeeds. Also assert that three unauthenticated + `REPORT_HYPERD_ERROR` messages can no longer trip `RESTART_LIMIT` and + shut the daemon down — that is the denial-of-service path, and it is the + more interesting of the two. + +### 3.4 Answer the engine-credential question, then implement it + +- [ ] **This is open question Q2 and needs a human before any code.** Either + give the shared `hyperd` a generated password stored in the `0600` + discovery file, or document explicitly that endpoint reachability equals + full authority over the engine. +- [ ] If a password: thread it through `HyperProcess::new` parameters and the + client's `Config`. Note that `Connection::connect` currently hardcodes + `Config::new().with_user("tableau_internal_user")` with no password, so + the shared path needs `connect_with_auth` or a builder. +- [ ] If not: the shared mode's documentation must state the boundary in the + first paragraph, not a footnote. + +### 3.5 Version the control protocol + +- [ ] Add an explicit `protocol` integer to `daemon.json` and the `PING` + response. +- [ ] Client rule: support protocol N and N-1. A daemon advertising an + unsupported protocol is treated as **absent**, not as an error, so the + client starts its own daemon rather than failing. +- [ ] Preserve today's forward compatibility — no `deny_unknown_fields`, and + keep #278's lenient-fallback reparse and its `from_fallback` guard that + prevents an old client deleting a live newer daemon's record. +- [ ] Red test: a record with `protocol` two versions ahead is treated as + absent, and the live daemon's file is **not** deleted. + +### 3.6 Recover a wedged-but-live `hyperd` (#242) + +- [ ] Keep `has_exited` polling; add a liveness probe that a `SIGSTOP`-ed or + otherwise wedged engine fails. A trivial query on a dedicated connection + with a bounded timeout is the obvious shape; confirm it does not + false-positive under legitimate heavy load before relying on it. +- [ ] Reuse the existing `RESTART_LIMIT`/`RESTART_WINDOW` limiter. Do not add a + second, independent restart path. +- [ ] Red test: reproduce #242 — `SIGSTOP` the managed child, prove a query + currently blocks past a deadline, then prove the daemon restarts the + engine after the fix. #242 documents the >30 s reproduction; the test must + use a bounded deadline and a killed-and-waited child so a blocked probe + cannot leave a worker behind. +- [ ] Gate the readiness check on #286's two-phase pattern: wait for the killed + endpoint to stop accepting **before** accepting whatever `STATUS` + advertises. A single-phase TCP-connectable gate is the exact bug #286 + fixed, and it made one test pass vacuously in 5 of 20 runs. + +### 3.7 Verify + +- [ ] Run and capture the full gate block from Phase 1.3, plus the daemon + suite on **macOS as well** — P5 in the spec notes macOS CI still skips + all eight restart tests, and a security phase that only runs on Linux is + not a floor. +- [ ] **Changelog entries needed:** `hyperdb-daemon/CHANGELOG.md` under + `### Security` for the permissions, token, and peer-identity work, and + `### Fixed` for #242. +- [ ] Commits, one per subsection: `fix(daemon): restrict state directory + permissions`, `feat(daemon)!: require a token for destructive control + commands`, `fix(daemon): recover a wedged hyperd`. + +--- + +## Phase 4 — Cohort-scoped `acquire()` (the feature) + +**Do not start until Phases 1–3 are complete and the human decisions below are +answered.** Land after `1.0.0`; the design spec's D3 establishes there is no +pre-1.0 reason to rush it. + +### 4.1 Cohorts + +- [ ] Implement `Cohort` — the key that decides which daemon a client joins. + Different cohorts get different daemons, different `hyperd` processes, and + therefore real isolation. +- [ ] Move the state layout to `~/.hyperdb/daemons//daemon.json`, + preserving the single-instance guarantee **per cohort**. The health-port + bind is currently what enforces single-instance; per-cohort scoping needs + that reasoned through again, not assumed. +- [ ] Provide a migration or compatibility path for `hyperdb-mcp`, which has a + resident daemon at the old path in the field. +- [ ] **Open question Q4 blocks the default.** Executable-path-derived cohorts + silently split when a binary moves; a required explicit argument is + honest but less ergonomic. A human decides. +- [ ] Red test: two cohorts yield two daemons and two `hyperd` processes; a + table in one cohort is invisible from the other. + +### 4.2 The acquisition API + +- [ ] Implement `Acquisition::{Private, Shared, SharedOrPrivate}`, `Engine`, + `SharedEngine`, `Options`, `acquire`, and `acquire_async` as specified in + the design spec's D11. +- [ ] `Options` is `#[non_exhaustive]`. `ChartOptions` in `hyperdb-mcp` was + found source-breaking to extend precisely because it was not. +- [ ] Default `Acquisition` is `Private` — existing behaviour, unchanged. +- [ ] Default `idle_timeout` is `Some(_)`, not `None`. Today's `None` is right + for a product that stays warm for its user and wrong for a library: a + `cargo test` run must not leave a resident service behind. +- [ ] **`SharedEngine::drop` must not stop `hyperd`.** It releases the lease + and stops heartbeating. This asymmetry with `HyperProcess::drop` is the + single most important invariant in the crate. +- [ ] Red test, and treat it as the phase's centrepiece: acquire a + `SharedEngine`, drop it, and prove the daemon and its `hyperd` are still + alive and serving. Then acquire a `Private` engine, drop it, and prove + `hyperd` exited — so the test pins **both** halves of the asymmetry and + cannot pass by accident. + +### 4.3 Failure semantics + +- [ ] `Shared` returns a typed error naming the reason: no daemon, spawn + timeout, protocol unsupported, token rejected. +- [ ] `SharedOrPrivate` honours `fallback_deadline`, sized from Phase 2's + measurements so the preferring mode is **provably never slower** than + `Private` by more than that budget. This is the fix for #270's ten-second + stall, not a tuning knob. +- [ ] Fallback logs at `warn` and is readable from `Engine::fallback_reason()`. + The MCP's `debug`-level `Ok(None)` is exactly why #270 was invisible. +- [ ] Libraries never take over a resident daemon. If the incumbent speaks a + compatible protocol, use it even if older; if not, start a separate + cohort daemon. **Never `STOP` an incumbent another process may be using.** + Deliberate takeover stays a CLI-only operator action. +- [ ] Red test: with no daemon reachable, `Shared` errors with the right + variant while `SharedOrPrivate` succeeds, reports `Private`, and stays + inside the deadline. Then: a newer client does **not** stop an older + resident daemon. + +### 4.4 Document the isolation and trust boundary + +- [ ] State the boundary from the design spec's D5 in the crate's README and + the `acquire` rustdoc, in the first paragraph: sharing is supported only + within one OS user, one trust domain, one cohort; `memory_limit` is + instance-global with no per-session equivalent, so a peer can apply + memory pressure; and a peer can take the engine down, at which point + attach state is gone and rebuilding session state is the caller's job. +- [ ] **Q1 must be resolved before this ships.** No test currently proves + whether one session sees another's attached databases on a shared + `hyperd`. Run the experiment: two connections to one `hyperd`, A attaches + with an alias, B queries that alias and enumerates + `pg_catalog.pg_database`. Land it as a permanent characterization test + whichever way it goes, so the documented model has evidence behind it. +- [ ] **Changelog entries needed:** `hyperdb-daemon/CHANGELOG.md` → + `### Added` for the acquisition API; `### Changed` if the state-directory + layout moves under `daemons//`. +- [ ] Commit: `feat(daemon): add cohort-scoped shared engine acquisition` + +--- + +## Phase 5 — Lease-based reference counting (optional) + +Only if Phase 4 ships and heartbeat-only idling proves insufficient in +practice. **Open question Q3.** + +- [ ] Add `LEASE`/`RELEASE` control commands; bump the protocol version. +- [ ] Idle timer runs only while lease count is zero. Heartbeats expire leases + held by processes that died without releasing. +- [ ] Red test: an idle-but-live client past the idle timeout does not have the + engine shut down under it; a killed client's lease expires and the daemon + does eventually idle out. +- [ ] **Changelog:** `hyperdb-daemon/CHANGELOG.md` → `### Added`. +- [ ] Commit: `feat(daemon): track client leases for idle shutdown` + +## Phase 6 — Test-harness integration (optional, highest-value follow-on) + +The design spec identifies test suites as the clearest win: roughly 245 +helper-backed spawn sites against a documented ~121 s `make test`. This is the +phase most likely to repay its cost, and it should be reconsidered as soon as +Phase 2's numbers exist. + +- [ ] Teach `TestConnection`/`TestServer` to accept a shared engine, keeping + `HyperProcess` ownership as the default so nothing regresses. +- [ ] **The risk is test isolation, and it is the whole phase.** Tests + currently get a fresh engine each time; sharing one means leaked state + between tests. Enumerate what leaks — temp tables, attach aliases, + session settings, `memory_limit` pressure — and prove per-test cleanup + before converting a single test. +- [ ] Convert one test file, measure the wall-clock delta, and **stop there + until a human reviews it.** Do not convert the suite on the strength of + one file's improvement. +- [ ] **Changelog:** none — test infrastructure is not public API. +- [ ] Commit: `test: allow the shared engine in test helpers` + +## Phase 7 — `hyperdb-api` trait seam (optional; decide before `1.0.0`) + +**This is the only phase with a `1.0.0` deadline other than Phase 1, and the +design spec recommends against it.** + +`Connection::new(instance: &HyperProcess, …)` takes a concrete type. If a +shared handle should ever be usable there, that parameter must become a trait +bound. Generalising a concrete parameter to `impl Trait` is source-compatible +for ordinary call sites but not for turbofish or function-pointer uses, so the +cheap moment is before the freeze. + +- [ ] **Human decision required — open question Q5.** The recommendation is + **no**: `Connection::connect(endpoint, …)` already covers the shared + case, and adding a trait for symmetry is speculative generality. Adding + it later is a *minor* addition, since a new trait plus a new inherent + method breaks nothing. +- [ ] If yes: add a trait exposing `endpoint()` and `connection_endpoint()`, + implement it for `HyperProcess` and `SharedEngine`, and generalise + `Connection::new`. Consider also making + `Connection::connect_with_endpoint` public and re-exporting + `ConnectionEndpoint` from `hyperdb-api` — it is reachable today only as + the return type of `HyperProcess::connection_endpoint()`. +- [ ] Verify no `hyperdb-api` dependency on `hyperdb-daemon` appears. The trait + lives in `hyperdb-api`; `hyperdb-daemon` implements it. The arrow must + keep pointing one way. +- [ ] **Changelog:** `hyperdb-api/CHANGELOG.md` → `### Added`. +- [ ] Commit: `feat(api): accept any hyperd endpoint provider in Connection::new` + +--- + +## Decisions a human must make + +Ordered by deadline. None should be resolved by an agent. + +| # | Decision | Blocks | Deadline | +|---|---|---|---| +| D-a | Does Phase 1 re-export `daemon` from `hyperdb-mcp` for compatibility? Recommendation: **no**. | 1.2 | `1.0.0` | +| D-b | Is a `refactor(daemon)!:` breaking marker acceptable inside the `-rc` line, given release-please currently computes `1.0.0-rc.3`? | 1.4 | `1.0.0` | +| D-c | Generalise `Connection::new` to a trait? (Q5) Recommendation: **no**. | 7 | `1.0.0` | +| D-d | Does the measured benefit justify Phase 4 at all? | 4 | after Phase 2 | +| D-e | Does the shared `hyperd` get a generated password, or is endpoint reachability documented as full authority? (Q2) | 3.4 | before Phase 4 | +| D-f | What is the default cohort? (Q4) | 4.1 | before Phase 4 | +| D-g | Leases, or heartbeat-only idle? (Q3) | 5 | before Phase 5 | +| D-h | Does `hyperdb-mcp` keep version takeover, or does it become CLI-only? (Q7) | 4.3 | before Phase 4 | + +Two questions need an **experiment**, not a decision, and both are cheap: + +- **Q1 — cross-session attach visibility on one shared `hyperd`.** Blocks 4.4. + Currently unproven in either direction; the design's isolation model assumes + an answer it does not have. +- **Q6 — whether upstream Hyper or the C++/Python/Java APIs already have a + shared-instance concept.** Blocks any public "novel" claim. Needs external + documentation, not a repository grep — the current conclusion rests on + absence of evidence in this tree. + +--- + +## Plan completion gate + +Phase 1 is complete when: + +- `hyperdb-mcp/src/lib.rs` no longer declares `pub mod daemon`; +- every daemon test that passed at `2f31b9e` passes in `hyperdb-daemon` under + its original name, with an ignored count no higher than before; +- the wire-format characterization proves old and new records interoperate in + both directions; +- `hyperdb-daemon` builds with `--no-default-features`; +- strict workspace Clippy, `cargo fmt --check`, the workspace test gate, and + `npx markdownlint-cli2` at **0 issues in 68 files** (plus any Markdown this + phase adds) all have fresh captured zero exits; +- `AGENTS.md`, the architecture diagram, and both affected changelogs are + updated; and +- an independent reviewer confirms the diff is a **move**, with no behavioural + change beyond the de-branding pinned in 1.3. + +Phases 2 and 3 are complete when their measurements and security tests are +captured and recorded, and a human has recorded the Phase 4 go/no-go. + +Phase 4 is complete when, additionally: + +- both halves of the drop asymmetry are proven by test; +- `SharedOrPrivate` is proven never slower than `Private` beyond + `fallback_deadline`; +- a newer client is proven not to stop an older resident daemon; +- Q1 has a permanent characterization test whichever way it resolved; and +- the isolation and trust boundary is documented in the first paragraph of the + crate README and the `acquire` rustdoc. diff --git a/docs/superpowers/specs/2026-09-06-shared-hyperd-daemon-api-design.md b/docs/superpowers/specs/2026-09-06-shared-hyperd-daemon-api-design.md new file mode 100644 index 0000000..961f568 --- /dev/null +++ b/docs/superpowers/specs/2026-09-06-shared-hyperd-daemon-api-design.md @@ -0,0 +1,954 @@ +# Shared `hyperd` Daemon as a First-Class API Capability — Design Exploration + +Explore promoting the resident `hyperd` daemon from an `hyperdb-mcp` internal +mechanism to a capability any `hyperdb-api` consumer can use, so that +short-lived processes stop paying `hyperd` startup cost and stop multiplying +`hyperd` instances. + +**Status:** design exploration — **not approved, not decided, argue with it** +**Date:** 2026-09-06 +**Author:** Stefan Steiner with Cursor (Claude Opus 5) +**Base:** `upstream/main` @ `2f31b9e` +**Branch:** `docs/shared-hyperd-daemon-design` +**Implementation plan:** [`docs/superpowers/plans/2026-09-06-shared-hyperd-daemon.md`](../plans/2026-09-06-shared-hyperd-daemon.md) + +> **This document contains no code and proposes no immediate implementation.** +> It exists to be reviewed and disagreed with. Two of its conclusions are +> negative: the headline framing of the proposal is wrong in a specific and +> useful way (D1), and the most ambitious version of the feature — unrelated +> applications sharing one engine — should not be built at all (D5, D6). A +> narrower subset is worth doing, and one small piece of it has a hard deadline +> at `1.0.0` **whether or not the rest ever happens** (D2). + +--- + +## Problem + +`hyperdb-mcp` runs a resident `hyperd` that multiple MCP sessions share. +Discovery goes through `~/.hyperdb/daemon.json`, liveness through a localhost +health/control port, and the daemon detects `hyperd` crashes and restarts it. +Every *other* consumer of `hyperdb-api` — tests, examples, CLI tools, the +benchmark suite, downstream applications — spawns a private `hyperd` through +`HyperProcess::new`. + +Real engineering went into that daemon, and it plausibly serves more than the +MCP. The proposal is a new mode meaning "use the shared daemon rather than +spawning your own `hyperd`", aimed at workloads that start many short-lived +processes: CLI invocations, test suites, serverless handlers, scripts. + +Three things make this harder than it sounds, and they are what this document +is actually about. + +**The crate graph points the wrong way.** `hyperdb-mcp` → `hyperdb-api` → +`hyperdb-api-core`. The daemon sits at the top of that stack, so +`hyperdb-api` physically cannot consume it where it lives. + +**`hyperdb-api` has no feature flags, by design.** Every capability is always +available, matching the C++/Python/Java APIs +([`AGENTS.md`](../../../AGENTS.md), "Feature Flags"). So the new mode cannot be +gated, and anything added to `hyperdb-api` is paid for by every user of the +crate. + +**We are at `1.0.0-rc.3`.** Public API added before `1.0.0` final is free. +After it, the surface is frozen under semver and additions cost a minor +version while removals cost a major one. + +--- + +## Ground truth + +Measured or read from the tree on 2026-09-06. Every claim below is cited +because several of them contradict the framing of the proposal. + +### What already exists + +- **`hyperdb-api` already connects to a `hyperd` it does not own.** + `Connection::connect(endpoint, database_path, create_mode)` + ([`connection.rs:247`](../../../hyperdb-api/src/connection.rs)), + `Connection::without_database`, `ConnectionBuilder`, + `AsyncConnection::connect` + ([`async_connection.rs:80`](../../../hyperdb-api/src/async_connection.rs)), + `PoolConfig { endpoint, .. }` + ([`pool.rs:232`](../../../hyperdb-api/src/pool.rs)), `SyncPoolConfig`, and + `grpc::GrpcConnection::connect` all take a bare endpoint and hold no + `HyperProcess`. None of them stop `hyperd` on drop. +- **The MCP's daemon mode already uses exactly that path.** + `Engine::try_daemon_mode` calls `Connection::connect(endpoint, …)` and stores + `hyper: None`, so `Engine::drop` cannot stop the shared engine + ([`engine.rs:616`](../../../hyperdb-mcp/src/engine.rs), + [`engine.rs:2290`](../../../hyperdb-mcp/src/engine.rs)). +- **`HyperProcess` is the only thing that spawns or stops `hyperd`.** Its + single public constructor is + `HyperProcess::new(hyper_path: Option<&Path>, parameters: Option<&Parameters>)` + ([`process.rs:246`](../../../hyperdb-api/src/process.rs)), and `Drop` closes + the callback connection and waits up to 5 s + ([`process.rs:1125`](../../../hyperdb-api/src/process.rs)). +- **`HyperProcess::drop()` is load-bearing for the test suite.** + `TestConnection` and `TestServer` both hold a `HyperProcess` field purely so + that scope exit reaps the server + ([`hyperdb-api/tests/common/mod.rs:47`](../../../hyperdb-api/tests/common/mod.rs), + [`hyperdb-api-core/tests/common/mod.rs:45`](../../../hyperdb-api-core/tests/common/mod.rs)). + A non-stopping variant of `HyperProcess` would silently orphan a server per + test. +- **`hyperdb-bootstrap` already sets the precedent for a library-plus-binary + crate**: `default = ["cli"]`, `cli = ["dep:clap", "dep:anyhow", "dep:tracing-subscriber"]` + ([`hyperdb-bootstrap/Cargo.toml:22`](../../../hyperdb-bootstrap/Cargo.toml)), + consumable as a pure library with `default-features = false`. + +### What the daemon actually is + +- **Discovery** is `~/.hyperdb/daemon.json`, written temp-file-then-rename, + carrying `pid`, `hyperd_endpoint`, `health_port`, `started_at`, `version`, + and an optional additive `identity` + ([`discovery.rs:22`](../../../hyperdb-mcp/src/daemon/discovery.rs)). The + state directory is `HYPERDB_STATE_DIR` or `$HOME/.hyperdb`. +- **The control port is line-oriented TCP bound to `127.0.0.1`** + ([`health.rs:116`](../../../hyperdb-mcp/src/daemon/health.rs)) accepting + exactly `PING`, `HEARTBEAT`, `STOP`, `STATUS`, and `REPORT_HYPERD_ERROR` + ([`health.rs:232`](../../../hyperdb-mcp/src/daemon/health.rs)). **There is no + authentication of any kind.** The security model is "bound to loopback". +- **The daemon forces TCP transport** for the shared `hyperd` + (`params.set_transport_mode(TransportMode::Tcp)`, + [`run.rs:206`](../../../hyperdb-mcp/src/daemon/run.rs)), so the engine + endpoint is a loopback TCP port, not a Unix socket. +- **Crash handling** polls `HyperProcess::has_exited` every 5 s and rate-limits + to 3 restarts per 60 s, shutting down if exceeded + ([`run.rs:55`](../../../hyperdb-mcp/src/daemon/run.rs)). +- **Idle timeout defaults to `None`** — the daemon runs forever unless + `--idle-timeout` or `HYPERDB_DAEMON_IDLE_TIMEOUT` is set + ([`run.rs:40`](../../../hyperdb-mcp/src/daemon/run.rs)). The 30-minute + `DEFAULT_IDLE_TIMEOUT_SECS` is a suggestion constant, not the default. +- **There is no reference counting.** Liveness is opt-in `HEARTBEAT`, sent by + the MCP server at most once per 60 s after a successful tool call + ([`server.rs:1702`](../../../hyperdb-mcp/src/server.rs)). Single-instance + enforcement is the health-port bind, not a lease count. +- **Version takeover kills the incumbent**: a client whose semver is strictly + greater sends `STOP`, waits for the PING to fail, and respawns + ([`spawn.rs:146`](../../../hyperdb-mcp/src/daemon/spawn.rs)). +- **`hyperdb_mcp::daemon` is public**, including `daemon::health`, exposing + `DaemonInfo`, `HealthListener`, `DaemonState`, `send_command`, + `report_hyperd_error_to_daemon`, `run_daemon`, `ensure_daemon`, and + `client_should_take_over` ([`lib.rs:41`](../../../hyperdb-mcp/src/lib.rs)). + +### What is MCP policy, not infrastructure + +Attachment replay through `AttachRegistry`, the reserved `"persistent"` alias +and ephemeral scratch database, `_table_catalog` bootstrap, the KV store, +watched-directory pools, the doctor/diagnostics identity comparison, the +`"hyperdb-mcp"` PONG token, and the `MCP_VERSION`-driven takeover rule are all +product decisions. None belongs in a general-purpose library. The genuinely +generic residue is: the discovery file, the port scan, the control-protocol +shape, `HyperProcess` ownership with a restart limiter, and the detached-spawn +skeleton. + +### Authentication and permissions, as they stand + +- **The shared `hyperd` requires no credentials.** `HyperProcess` passes + `--no-password` and `--init-user=tableau_internal_user` + ([`process.rs:488`](../../../hyperdb-api/src/process.rs), + [`process.rs:541`](../../../hyperdb-api/src/process.rs)) and + `Connection::connect` hardcodes `Config::new().with_user("tableau_internal_user")` + with no password ([`connection.rs:221`](../../../hyperdb-api/src/connection.rs)). +- **`daemon.json` is written with default umask permissions.** + `write_discovery_record` does `fs::write` then `fs::rename` with no + `set_permissions` ([`discovery.rs:236`](../../../hyperdb-mcp/src/daemon/discovery.rs)). + A workspace-wide grep finds no `0o600`/`0o700` on any state file or socket + directory. Under the common `umask 022` the file is world-readable. +- **The socket directory is not restricted either.** `create_dir_all` on + `$TMPDIR/hyper-` with no mode + ([`process.rs:377`](../../../hyperdb-api/src/process.rs)). +- **No hyperd threat-model document exists** in `docs/`. + +### Isolation, as far as the tree proves it + +- **Attach is modelled per-connection.** `attach_database`/`detach_database` + are `Connection` methods emitting `ATTACH`/`DETACH DATABASE` + ([`catalog.rs:115`](../../../hyperdb-api/src/catalog.rs), + [`connection.rs:1448`](../../../hyperdb-api/src/connection.rs)), and + `schema_search_path` is set per connection + ([`attach.rs:52`](../../../hyperdb-mcp/src/attach.rs)). +- **But no test proves cross-session alias invisibility on one shared + `hyperd`.** The closest evidence uses two *private* engines + ([`attach_tests.rs:566`](../../../hyperdb-mcp/tests/attach_tests.rs)), which + proves attach state does not survive a new process — a different claim. This + gap is load-bearing and appears again as **Q1**. +- **`memory_limit` is a Hyper-global instance parameter**, default 80 % + ([`hyperdb-api/tests/stress_test/README.md:169`](../../../hyperdb-api/tests/stress_test/README.md)). + Greps for `soft_memory_limit`, `hard_memory_limit`, `admission`, and + per-session limits find nothing. **There is no per-session resource + isolation to configure.** +- **`Persistence::Temporary` is documented as "only available in the current + session"** ([`table_definition.rs:11`](../../../hyperdb-api/src/table_definition.rs)). + +### How production-ready the daemon is + +This matters because the daemon is the asset being promoted, and the answer is +uncomfortable. + +| Merged (UTC) | Commit | Change | +|---|---|---| +| 2026-09-06 18:54 | `06c5da1` | #267 daemon idle-timeout test fix | +| 2026-09-06 20:45 | `56bc0d0` | #279 test-verification gaps | +| 2026-09-06 21:57 | `61fc766` | #280 mutex poison + pool transaction leak | +| 2026-09-06 22:59 | `b54103c` | #286 (carrying #278) restart publish ordering | + +The daemon has existed since 2026-05-25 and has been resident-by-default since +v0.5.0 in early June. Its `daemon/` directory then saw no functional change for +roughly three months, until #243 on 2026-08-28. **Every hardening change listed +above landed inside a single working day**, and every source issue was filed the +same day. Nine distinct concurrency or lifecycle defects were found in that +window, including a non-atomic "atomic" write, a restart publish-ordering +window, mutex poisoning that permanently bricked the MCP after any tool-call +panic, and a transaction leaking back into the connection pool. + +The most consequential finding is not any single bug: **the eight +crash-and-restart tests were `#[ignore]`d unconditionally**, so the daemon +shipped resident-by-default for three months with no automated crash-recovery +coverage on any platform. #279 turned them on and CI went red immediately and +stayed red for two commits until #286. + +Two daemon-shaped risks remain **open**: + +- **#242** — a `hyperd` that stays alive but stops serving is never recovered. + Reproduced by `SIGSTOP`-ing the child: a `SELECT 1` blocked for over 30 s, the + daemon kept seeing the process as alive, and never restarted it. +- **#118** — every MCP tool call serializes behind one engine mutex held across + the whole blocking operation, so one stalled call hangs everything including + `status`. The issue notes this got worse when the daemon became + resident-by-default. + +The verification discipline in the recent PRs is genuinely above average — each +fix was proven red-before-green by reverting the code it guards. But this code +has **zero field exposure**, macOS CI still skips all eight restart tests, and +the Windows suite has two open flakes. + +### Precedent + +No document in this repository states the process model of the C++, Python, or +Java Hyper APIs, and no upstream Tableau term for a shared or attachable +`hyperd` appears anywhere in the tree. What the repo does say is that those +APIs *ship* a `hyperd` binary +([`hyperdb-mcp/README.md:123`](../../../hyperdb-mcp/README.md)) and that +`HyperProcess` defaults match "C++ `HyperProcess` behavior" in the sense of +default *parameters* ([`process.rs:483`](../../../hyperdb-api/src/process.rs)). +Connecting to an independently started server is already documented, but as a +manual step: run `hyperd` yourself and pass the endpoint +([`grpc_query.rs:25`](../../../hyperdb-api/examples/additional_examples/grpc_query.rs)). + +**So a managed shared-daemon mode does appear novel relative to the sibling +APIs** — but that claim rests on absence of evidence in *this* repository, and +should be checked against upstream documentation before being used as a +differentiator in any public material. Recorded as **Q6**. + +--- + +## Decisions + +### D1 — The proposal's framing is wrong: the connect path already exists; only discovery and supervision are missing + +This is the most important conclusion in the document, and it reshapes +everything after it. + +"Expose a new mode meaning use the shared daemon rather than spawning your own +`hyperd`" implies `hyperdb-api` cannot currently talk to a `hyperd` it does not +own. It can, and has been able to all along. `Connection::connect` takes an +endpoint string. `PoolConfig` takes an endpoint string. The MCP's own daemon +mode is built out of those exact calls. + +What is genuinely absent from `hyperdb-api` is narrower: + +1. **Discovery** — turning "the shared daemon, wherever it is" into an endpoint. +2. **Supervision** — starting the daemon if absent, restarting `hyperd` when it + dies, shutting down when idle. + +Item 2 requires a **daemon process**, which requires a binary, argument +parsing, and a logging subscriber. `hyperdb-api` must not grow those, and with +no feature flags it could not gate them if it did. + +The consequence is a much cheaper design than the proposal assumes, and it +inverts the assumed direction: + +```mermaid +graph TD + subgraph assumed["Assumed: daemon must move DOWN"] + A1[hyperdb-mcp] --> A2[hyperdb-api] + A2 --> A3["daemon/*
moved into api or core"] + style A3 stroke:#d64545,stroke-width:2px,fill:none + end + subgraph actual["Actual: daemon belongs ABOVE"] + B1[hyperdb-mcp] --> B2[hyperdb-daemon] + B2 --> B3[hyperdb-api] + B4["other consumers
CLI, tests, scripts"] --> B2 + style B2 stroke:#50c878,stroke-width:2px,fill:none + end +``` + +The daemon is a **supervisor that consumes `HyperProcess`**. It is a higher +layer than `hyperdb-api`, not a lower one. Nothing needs to move down. + +- **Chosen:** treat this as "extract a supervisor crate that sits above + `hyperdb-api`", not "promote daemon code into `hyperdb-api`". +- **Consequence:** in the minimum viable slice, **`hyperdb-api` gains no public + API, no dependency, and nothing entering the 1.0 freeze.** That is the + cheapest available answer and it is also the correct one. + +### D2 — Extract `daemon/*` into a new `hyperdb-daemon` crate, and do it before `1.0.0` regardless of whether the rest ships + +Four placements were considered. + +- **Rejected — move into `hyperdb-api`.** Needs a binary, so it needs `clap` + and a subscriber; with no feature flags every `hyperdb-api` user pays for + them. It also puts a resident-service surface inside the 1.0 semver freeze + and inside the crate whose job is to mirror the sibling Hyper APIs. +- **Rejected — move into `hyperdb-api-core`.** Worse. `hyperdb-api-core` is + positioned as forever-internal and sits *below* the API; supervision is + strictly above wire protocol. This inverts the layering the architecture + documents. +- **Rejected — one crate that `hyperdb-api` depends on.** Makes discovery a + mandatory transitive dependency for every `hyperdb-api` user, most of whom + will never use it, with no flag available to opt out. +- **Chosen — a new `hyperdb-daemon` crate that depends on `hyperdb-api`**, + shipping a client library plus a `hyperdb-daemon` binary behind a + `default = ["cli"]` feature, exactly mirroring `hyperdb-bootstrap`. + `hyperdb-mcp` then depends on `hyperdb-daemon` and deletes its own + `daemon/*`. Dependency direction is `hyperdb-daemon → hyperdb-api`, the same + direction `hyperdb-mcp → hyperdb-api` already goes, so no layering rule + bends. + +**This decision answers open issue #276 directly.** That issue asks whether +`hyperdb_mcp::daemon::health` should be public at all before 1.0, having +observed that `report_hyperd_error_to_daemon` changed signature in a patch +release with no breaking-change entry. The answer this design gives: **no.** +The module is public by accident of module organisation, not by intent — +nothing about running an MCP server requires callers to reach into its daemon +control protocol. The surface belongs to a crate whose stated job is exactly +that, where it can carry its own compatibility promise. + +And it carries a real deadline. `pub mod daemon` in `hyperdb-mcp` is a public +surface today. Removing it after `1.0.0` is a major version. Removing it before +`1.0.0` is free. **So the extraction is worth doing on issue #276's merits +alone, even if every later phase in this document is rejected.** It is also the +cheapest phase, being a move plus a re-export. + +- **Chosen:** extract before `1.0.0`; treat it as a `refactor:`/`fix:` change + that narrows `hyperdb-mcp`'s public surface, independent of the shared-mode + feature. + +### D3 — Everything else lands after `1.0.0` + +Once D1 holds, the pre-1.0 pressure almost entirely evaporates. The shared-mode +API lives in a new crate, which starts at `0.1.0` and can iterate freely +regardless of what the workspace's flagship crate is doing. + +Exactly one candidate `hyperdb-api` change has a pre-1.0 deadline, and it is +small: **whether `Connection::new` should keep taking a concrete +`&HyperProcess`.** + +```rust +// today +pub fn new(instance: &HyperProcess, database_path: impl AsRef, create_mode: CreateMode) -> Result +``` + +If a shared-engine handle should ever be usable where a `HyperProcess` is +usable, that parameter needs to become a trait bound. Generalising a concrete +parameter to `impl Trait` is source-compatible for ordinary call sites but not +for turbofish or function-pointer uses, so it is cheapest to do — or decide +against — before the freeze. + +- **Chosen:** do **not** generalise it now. `Connection::connect(endpoint, …)` + already covers the shared case, and adding a trait purely for symmetry is + speculative generality. Adding the trait later is a *minor* addition, since a + new trait plus a new inherent method breaks nothing. +- **Recorded as the one pre-1.0 API decision a human should confirm** + (**Q5**), because if the answer is "yes, symmetry matters", the window closes + at `1.0.0`. +- **Rejected — ship the shared-mode API before `1.0.0` to get it in free.** + The security floor (D6) is not met, the benefit is unmeasured (D9), and #242 + is open. Rushing an unproven resident-service API into a frozen surface to + save a minor version is a bad trade. + +### D4 — Default to *cohort-scoped* daemons, not one machine-wide daemon + +This is the design's answer to the isolation problem, and it is the difference +between a feature that is safe and one that is not. + +The MCP model is one daemon per user, machine-wide. Generalising *that* to a +library means unrelated applications share an engine, which is where every hard +problem in this document comes from. But re-read where the benefit actually is: +a test suite spawning hundreds of servers, a CLI invoked repeatedly in a loop, +a warm pool of serverless handlers running the same code. **In every one of +those, the processes sharing the daemon are the same application, the same +build, and the same trust domain.** + +So do not multi-tenant. Key the daemon on a **cohort**: a caller-supplied +identifier, defaulting to something derived from the calling application rather +than to a global constant. Different cohorts get different daemons, different +`hyperd` processes, and therefore complete isolation — while still getting the +warm-start sharing that motivated the whole idea. + +```text +~/.hyperdb/ + daemons/ + / + daemon.json 0600 + logs/ +``` + +What this buys: + +- The multi-tenancy question (D5) stops being a blocker and becomes a + documented boundary: sharing happens only within a cohort you named. +- The global `memory_limit` problem stops being cross-application. +- Blast radius is scoped to one application's own processes. +- Version skew mostly disappears, because a cohort is usually one build. + +What it costs: a machine running five cohorts runs five `hyperd` processes, +which is worse than one and better than one-per-process. That is the right +place on the curve, and it is honest about not being free. + +- **Chosen:** cohort-scoped by default; a machine-wide cohort is expressible but + must be opted into explicitly and documented with the D5/D6 caveats. +- **Rejected — one global daemon per user (the MCP model).** Correct for a + single-user developer tool with one product's policy. Not defensible as a + library default; see D5. + +### D5 — Cross-application sharing is not safely offerable, and the reason is not fixable in this repository + +Working through what two unrelated applications on one `hyperd` actually see: + +| Resource | Scope | Leaks across tenants? | +|---|---|---| +| `ATTACH DATABASE` alias | Per connection | Probably not — **unproven**, see **Q1** | +| `CREATE TEMPORARY TABLE` | Per session | No | +| `schema_search_path`, `SET` | Per connection | No | +| Prepared statements | Per connection | No (by protocol usage) | +| `.hyper` file contents | Per file | **Yes** — anyone who attaches it sees it | +| `memory_limit` | **Instance-global** | **Yes, unavoidably** | +| Threads, admission control | Instance-global | **Yes** — no per-session quota exists | +| `hyperd` process liveness | Instance-global | **Yes** — one crash hits every tenant | +| MCP `_table_catalog` | Per persistent file | Yes, if the path is shared | + +The SQL namespace is probably fine. The **resources are not**, and that is the +disqualifying finding: `memory_limit` is a Hyper instance parameter with no +per-session equivalent anywhere in the tree. One tenant's oversized query +applies memory pressure to every other tenant on the instance, and there is no +knob to prevent it because Hyper does not expose one. This is not a gap in the +daemon; it is a property of the engine. + +Blast radius compounds it. When the shared `hyperd` dies, every tenant loses +in-flight transactions *and* every session's attach state. The daemon restarts +`hyperd` — but attachment replay is MCP policy living in `AttachRegistry`, so a +library client would simply find its attachments silently gone against a +freshly restarted engine. Rebuilding session state after a peer-induced restart +is a burden the library would be pushing onto every caller. + +- **Chosen:** define the safe boundary precisely and refuse to cross it. + Sharing is supported **only** when all participants are the same OS user, the + same trust domain, and the same cohort, and when callers accept that a peer + can consume engine memory and can take the engine down. Documented as a + hard contract, not a footnote. +- **Rejected — offer general cross-application sharing with caveats.** The + caveat would have to say "an unrelated application can exhaust your memory + budget and kill your engine, and you must rebuild your session state when it + does". A library feature whose honest documentation reads like that should not + ship. + +### D6 — The security floor is not met today, and control-port authentication alone would be theater + +Threat model. Assets: `.hyper` files readable and writable by the daemon's UID, +and the availability of the shared engine. Actors: other local UIDs on a +multi-user workstation or shared CI runner, and other processes under the same +UID. + +Three live weaknesses, all confirmed above: + +1. **The engine endpoint requires no credentials.** `hyperd` runs with + `--no-password` and a well-known `--init-user`. Anyone who can open the + loopback TCP port connects as `tableau_internal_user` and can then + `ATTACH DATABASE ''`. +2. **`daemon.json` publishes that endpoint world-readably**, because nothing + chmods it and the common umask is `022`. +3. **The control port authenticates nothing.** Any local process can `STOP` the + daemon, or send `REPORT_HYPERD_ERROR` three times in 60 s to trip + `RESTART_LIMIT` and make it shut itself down. + +Chain 1 and 2 and it is a **confused-deputy privilege escalation**: a different +local UID reads the discovery file, connects with no password, and borrows the +daemon owner's filesystem authority over `.hyper` files. For a single-user +developer tool this is a thin risk with a short window. As a general library +capability it becomes a standing, always-on, machine-wide service, which is a +materially different proposition — and CI runners are the worst case, being +multi-tenant with a resident daemon that can outlive the job. + +The important part: **fixing the control port does not fix this.** The control +port is not the interesting attack surface; the `--no-password` engine endpoint +is. Adding a token to `STOP` while leaving the engine open to anyone who can +read a world-readable file would be security theater. + +Minimum floor before any third-party-facing shared mode: + +- **Move the shared engine to Unix domain sockets** in a `0700` directory with a + `0600` socket, and to a named pipe with an explicit DACL on Windows, so peer + identity is enforced by the OS rather than assumed from "loopback". +- **Create `~/.hyperdb` at `0700` and write `daemon.json` at `0600`**, setting + the mode before the content lands, so file permission becomes the + authorization mechanism. +- **Require a capability token** — generated per daemon, stored in the + now-`0600` discovery file — on every state-changing control command. `PING` + and `STATUS` may stay open; `STOP` and `REPORT_HYPERD_ERROR` must not. +- **Decide the engine-credential question explicitly** (**Q2**): either give + the shared `hyperd` a generated password, or document plainly that anyone who + can reach the endpoint has full authority over it. Both are defensible. Not + deciding is not. +- **If TCP must be kept**, verify peer UID via `SO_PEERCRED`/`LOCAL_PEERCRED`. + +Note that the daemon currently *forces* TCP +([`run.rs:206`](../../../hyperdb-mcp/src/daemon/run.rs)) and that the UDS path +in `HyperProcess` is not the runtime default either +([`process.rs:362`](../../../hyperdb-api/src/process.rs), which defaults to +`Tcp` "until UDS performance is validated" despite `TransportMode`'s +`#[default]` being `Ipc`). So the first bullet is not a config change; it needs +that validation done first. + +- **Chosen:** treat the floor as a hard gate. No shared mode is offered to + third parties until UDS/named-pipe transport, restrictive permissions, and + token-gated destructive commands are all in place and the engine-credential + question is answered. + +### D7 — Lease-based lifecycle, and libraries must never take over + +**Who starts it.** Any client, on demand, via detached spawn — the existing +`ensure_daemon` shape. + +**Who stops it.** Nobody explicitly. Idle timeout, and it must be **on by +default** for a library-acquired daemon. Today's `None` default is right for a +product that wants to stay warm for its user and wrong for a library: a +`cargo test` run must not leave a resident service behind. A conservative +default in the minutes, explicitly overridable, is the shape. + +**Reference counting versus idle timeout.** Today there is no refcount, and +pure heartbeat-based idling has a real failure mode: a client that is alive but +idle past the timeout has the engine shut down underneath it. Recommend a +**lease**: clients register on acquire and release on drop, the daemon tracks +lease count, and the idle timer only runs while the count is zero. Heartbeats +remain as the liveness mechanism that expires leases held by processes that died +without releasing. This needs new control commands, hence a control-protocol +version (D8). + +**Crash recovery.** Keep the existing 5-second poll and the 3-per-60-seconds +limiter, but note that **#242 is a prerequisite, not a nice-to-have**: a +`hyperd` that is alive but wedged is currently never recovered, and a >30 s +unrecoverable hang is a much worse experience for a library caller who never +opted into a resident service than for an MCP user who can restart their +client. + +**Version skew and the upgrade case.** The current rule — client with strictly +greater semver sends `STOP` and respawns — is actively dangerous when +generalised. Application A on v1.2 would kill the daemon that application B on +v1.1 is mid-transaction against. For a library: + +- **Libraries never take over.** If the resident daemon speaks a compatible + control protocol, use it as-is even if older. +- If it does not, **start a second daemon in a distinct cohort** rather than + killing the incumbent. +- Deliberate takeover stays available as an explicit operator action in the + CLI, where a human is choosing to do it. + +### D8 — Version the control protocol separately from the crate + +Crate semver is the wrong compatibility axis for a wire protocol, and #276 is +the evidence: `report_hyperd_error_to_daemon` changed signature under a +compatible-looking patch bump, and `send_command_with_timeout` silently +redefined `read_timeout` from per-read to whole-exchange with no signature +change at all. + +Add an explicit `protocol` integer to `daemon.json` and to the `PING` response. +The promise: **a client supports protocol N and N-1**; a daemon advertising an +unsupported protocol is treated as absent rather than as an error, and the +client starts its own cohort daemon. That is a checkable contract, expressible +independently of whatever `hyperdb-daemon`'s crate version happens to be. + +Forward compatibility is already partly right — `daemon.json` has no +`deny_unknown_fields`, so unknown keys are ignored, and #278 fixed the reader +half that was violating that contract. Keep that, and add the explicit integer +so skew is detectable rather than inferred. + +### D9 — Do not build this until the benefit is measured + +The entire premise is that sharing saves startup cost. **This repository does +not contain a measurement of `hyperd` spawn-to-usable wall clock.** See the +quantification section: the honest state is one figure from an adjacent code +path, one CI upper bound, and a set of timeouts. + +That is not a reason to abandon the idea. It is a reason not to design against +a number nobody has. A bounded measurement is cheap — it is a loop around +`HyperProcess::new` plus a trivial query — and it should gate the work, because +the answer changes the design: + +- If cold spawn is ~150 ms, the warm win per process is ~150 ms, and this is + worth doing only for workloads that pay it hundreds of times. +- If cold spawn is seconds on the platforms people actually use, the win is + large and the feature is clearly justified. +- If cold spawn is *cheaper* than the daemon's own cold-acquisition path, the + feature is net-negative for the first process and only ever wins on the + second — which is still fine, but changes what the default should be. + +Issue #270 shows the third case is not hypothetical: a skewed discovery record +made the client wait out the 10-second `SPAWN_TIMEOUT` and then spawn a private +`hyperd` anyway. Cold-path acquisition cost is a first-class design constraint, +which is why D10 puts a deadline on it. + +### D10 — Three explicit intents, no silent fallback, and a hard deadline on the cold path + +Callers have three genuinely different intentions and must be able to say which. + +```rust +/// How to obtain a `hyperd` to talk to. +pub enum Acquisition { + /// Spawn a private `hyperd`. Today's behaviour; remains the default. + Private, + /// Use the shared cohort daemon. Fail if it cannot be reached or started. + Shared, + /// Prefer the shared cohort daemon; fall back to a private `hyperd`. + /// Never slower than `Private` by more than `fallback_deadline`. + SharedOrPrivate, +} +``` + +`SharedOrPrivate` must carry a **deadline**, and this is not a tuning knob but +the fix for the #270 shape. If discovery and spawn have not produced a usable +endpoint within a budget smaller than the measured private-spawn cost, abandon +and spawn private. That makes the preferring mode *provably* not a pessimisation +— the property #270's ten-second stall violated. + +**Fallback must be observable.** The MCP currently logs it at `debug` and +returns `Ok(None)` ([`engine.rs:604`](../../../hyperdb-mcp/src/engine.rs)), +which is exactly why #270 was invisible in normal operation. For a library: + +- `Shared` returns a **typed error** naming why (no daemon, spawn timed out, + protocol too old, token rejected). +- `SharedOrPrivate` succeeds, but the handle reports which mode it got **and** + why the preferred one was declined, and it logs at `warn`, not `debug`. +- The default stays `Private`, so existing behaviour is untouched and sharing + is always something a caller asked for. + +### D11 — The proposed API + +Concrete, additive, and living in `hyperdb-daemon`. Note that the connect calls +are today's unmodified `hyperdb-api` calls. + +```rust +// ---- hyperdb-daemon ---- + +/// A `hyperd` obtained by either route. Knows whether it owns the process. +pub enum Engine { + Private(hyperdb_api::HyperProcess), + Shared(SharedEngine), +} + +impl Engine { + /// libpq endpoint, usable with every `hyperdb-api` connect API. + pub fn endpoint(&self) -> &str; + pub fn connection_endpoint(&self) -> &ConnectionEndpoint; + pub fn acquired(&self) -> Acquired; // Private | Shared + /// Why `Shared` was declined, when `SharedOrPrivate` fell back. + pub fn fallback_reason(&self) -> Option<&FallbackReason>; +} + +/// Drop releases the lease and stops heartbeating. +/// It does NOT stop `hyperd` — that asymmetry with `HyperProcess::drop` is +/// the whole point and is the invariant most worth testing. +pub struct SharedEngine { /* … */ } + +impl SharedEngine { + pub fn endpoint(&self) -> &str; + pub fn daemon_pid(&self) -> u32; + pub fn protocol_version(&self) -> u32; + pub fn cohort(&self) -> &Cohort; +} + +#[non_exhaustive] +#[derive(Clone, Debug)] +pub struct Options { + pub cohort: Cohort, + pub state_dir: Option, + pub idle_timeout: Option, // Some(_) by default — see D7 + pub fallback_deadline: Duration, // see D10 + pub hyperd_path: Option, + pub parameters: Option, // applied only when we start it +} + +pub fn acquire(how: Acquisition, options: &Options) -> Result; +pub async fn acquire_async(how: Acquisition, options: &Options) -> Result; +``` + +`Options` is `#[non_exhaustive]` deliberately: `ChartOptions` in `hyperdb-mcp` +was found to be source-breaking to extend precisely because it was not, and +this crate will grow knobs. + +**Sync, before and after.** The `Connection::new` line changes to +`Connection::connect`; nothing else moves. + +```rust +// today +let hyper = HyperProcess::new(None, None)?; +let conn = Connection::new(&hyper, "db.hyper", CreateMode::CreateIfNotExists)?; + +// shared, or private if the daemon is unavailable +let engine = hyperdb_daemon::acquire( + Acquisition::SharedOrPrivate, + &Options::for_cohort("my-cli"), +)?; +let conn = Connection::connect(engine.endpoint(), "db.hyper", CreateMode::CreateIfNotExists)?; +if engine.acquired() == Acquired::Private { + tracing::warn!(reason = ?engine.fallback_reason(), "shared engine unavailable"); +} +``` + +**Async, before and after.** Note there is no `AsyncConnection` constructor +taking a `HyperProcess` today, so async callers *already* pass an endpoint +string — for them this is purely a change of where the string comes from. + +```rust +// today +let hyper = HyperProcess::new(None, None)?; +let conn = AsyncConnection::connect(hyper.require_endpoint()?, "db.hyper", mode).await?; + +// shared, required +let engine = hyperdb_daemon::acquire_async(Acquisition::Shared, &opts).await?; +let conn = AsyncConnection::connect(engine.endpoint(), "db.hyper", mode).await?; + +// and the pool needs no change at all +let pool = create_pool(PoolConfig::new(engine.endpoint(), "db.hyper"))?; +``` + +The pool example is the clearest illustration of D1: connection pooling over a +shared daemon works today, with no new API, because `PoolConfig.endpoint` is +already a string and the pool has never owned a process. + +--- + +## Quantified benefit + +Labelled honestly. **Two figures are read from the tree; everything else is an +estimate or an explicit gap.** No benchmark was run for this document — another +worker was building concurrently and disk has been tight — so the numbers below +are exactly as strong as their citations and no stronger. + +### `hyperd` startup cost — the primary claimed win, and it is not measured + +| Figure | What it measured | Hardware | Source | Confidence | +|---|---|---|---|---| +| **~156 ms** | First embedded Hyper start in a proc-macro host | **not stated** | [`hyperdb-api-derive/README.md:216`](../../../hyperdb-api-derive/README.md) | **read from tree**, adjacent path | +| **~7 ms** | One `LIMIT 0` dry-run on an already-warm connection | not stated | [`hyperdb-compile-check/src/db.rs:28`](../../../hyperdb-compile-check/src/db.rs) | **read from tree** | +| **"10+ seconds under load"** | CI upper bound, `hyperd` startup alone | CI runners, "especially macOS" | [`daemon_tests.rs:1775`](../../../hyperdb-mcp/tests/daemon_tests.rs) | **read from tree**, upper bound only | +| "20 ms vs 10 s" | An unresolved prior conflict a spike was meant to settle | — | [`phase0_compile_check_spike.rs:10`](../../../hyperdb-api/tests/phase0_compile_check_spike.rs) | **never recorded** | + +`docs/BENCHMARK_GUIDE.md` and `docs/hyperd-release-benchmarks.md` were both +checked first, as instructed. **Neither publishes a spawn-to-usable wall +clock** — they measure throughput, and `hyperd-release-benchmarks.md`'s +"cold-start variance" refers to insert-throughput variance, not process spawn. +The Phase-0 spike prints elapsed times under `--nocapture` but is `#[ignore]`d +and asserts nothing about duration. + +Relevant timeouts bound the space without measuring it: +`HyperProcess::wait_for_callback` polls up to **60 s** +([`process.rs:709`](../../../hyperdb-api/src/process.rs) — note the doc comment +at `:226` says 30 s and is wrong), and the daemon's client-side +`SPAWN_TIMEOUT` is **10 s** ([`spawn.rs:18`](../../../hyperdb-mcp/src/daemon/spawn.rs)). + +**How to measure it, so this gap closes cheaply.** Time `HyperProcess::new` +plus one trivial query to first row, 20 iterations, median and p95, release +build, on macOS and Linux, warm page cache and cold. Then time +`hyperdb_daemon::acquire(Shared, …)` plus the same query against an +already-running daemon. The delta between those two medians *is* the per-process +win, and the second number is the one the design actually needs, because +discovery is not free either: a warm `discover()` is roughly a file read plus +one PING (**~1 ms** per +[`server.rs:1737`](../../../hyperdb-mcp/src/server.rs)), but a dead port costs a +**300 ms** timeout and a full 16-port scan can reach **4.8 s**. + +### Memory footprint of one `hyperd` versus N + +**Not measured anywhere in the repository.** Greps for RSS and footprint +figures find only benchmark-host RAM totals (96 GB, 127.8 GB) and the +qualitative claim that the shared daemon gives "reduced memory overhead" +([`hyperdb-mcp/README.md:36`](../../../hyperdb-mcp/README.md)). + +This is a real gap, and it interacts with D5: `memory_limit` defaults to **80 % +of host RAM** and is instance-global. So N private `hyperd` processes each +believe they may use 80 % of the machine — which is an argument *for* sharing on +memory grounds that nobody has quantified, and simultaneously the mechanism by +which one shared tenant starves another. Measuring idle and post-query RSS for +one process is a `ps`-in-a-loop exercise and should be done in the same pass as +startup. + +### What sharing costs + +Here the repository does have data, and it is more nuanced than "sharing is +free". From `docs/BENCHMARK_GUIDE.md` — Apple M3 Max, 96 GB, `hyperd` +`0.0.26479`, 2026-09-05, medians of 5: + +| Workload | 1 connection | 4 connections | Direction | +|---|---|---|---| +| `AsyncArrowInserter`, 100M rows | **68.90 M/s** | **48.47 M/s** | **30 % worse** | +| `query.full_scan`, async | 24.91 M/s | **73.45 M/s** | ~2× better | +| `query.filtered`, async | 26.90 M/s | 48.31 M/s | better | + +The guide states plainly that **"parallelism no longer helps Arrow inserts"** +and that single-connection `AsyncArrowInserter` outruns the 4-connection +variant, "so spending connections on an Arrow insert buys nothing on this host" +([`BENCHMARK_GUIDE.md:201`](../../../docs/BENCHMARK_GUIDE.md)). It also warns +that the ×4 rows are "order-of-magnitude only", with a ±20–61 % spread, because +four workers contend on a 14-core laptop +([`BENCHMARK_GUIDE.md:190`](../../../docs/BENCHMARK_GUIDE.md)). + +Read for this design: **one `hyperd` serves concurrent readers well and +concurrent bulk writers poorly.** A shared daemon whose tenants are all +ingesting will contend on exactly the workload where extra connections already +measure negative. Note the Windows figures invert this (5.39 → 20.28 M/s for +×4), so the conclusion is host- and engine-version-specific, not universal. + +Beyond throughput, sharing costs: blast radius (one crash hits every tenant, +and attach state does not survive it), the unbounded `memory_limit` interaction, +and #118's serialization if a shared daemon ever grows a single-lock front end. + +### Where this wins, where it is neutral, and where it is worse + +| Workload shape | Verdict | Why | +|---|---|---| +| Test suite, hundreds of sequential server starts | **Clear win** | Pays spawn cost most often; single trust domain; ~245 helper-backed spawn sites exist today against a ~121 s `make test` | +| CLI invoked repeatedly in a loop or script | **Clear win** | Per-invocation spawn dominates a short workload | +| Serverless warm pool, same function | **Likely win** | Many short processes, one trust domain — but only if the daemon survives between invocations, which is platform-dependent | +| Long-running server, one process | **Neutral to negative** | Spawn paid once; adds a discovery dependency, a second failure domain, and a peer that can take the engine down | +| Concurrent bulk ingest from several processes | **Actively worse** | ×4 Arrow insert already measures 30 % *below* single-connection on the benchmark host | +| Unrelated applications, mixed trust | **Do not** | D5 and D6 | + +**The win is narrower than the proposal implies.** It is concentrated in +many-short-processes-same-application, which is real and worth serving, and it +is absent or negative for the single long-lived process that most library users +actually are. + +--- + +## Non-goals + +- **No remote or network daemon.** Loopback and local IPC only. This is not a + `hyperd` broker. +- **No multi-user daemon.** One daemon per `(uid, cohort)`. Never a service + shared across OS users. +- **No cross-trust-domain multi-tenancy.** D5. +- **No attachment replay, catalog, KV store, doctor, or watched directories.** + MCP policy stays in `hyperdb-mcp`. +- **No replacement for `HyperProcess`.** `Private` remains the default and + `HyperProcess::drop` keeps stopping the process, because the test suite + depends on it. +- **No non-stopping variant of `HyperProcess`.** A `HyperProcess` that skips + shutdown on drop would break the dead-man's-switch contract and orphan a + server per test helper. Shared handles are a distinct type. +- **No feature flag on `hyperdb-api`.** Firm repository constraint; the design + satisfies it by adding nothing to `hyperdb-api` at all. +- **No fix for #242 or #118 as part of this work** — but #242 is named a + **prerequisite gate**, not a follow-on. +- **No changes to release automation, crate versions, or the root changelog.** +- **No claim about the C++/Python/Java process model** beyond what this + repository documents. See **Q6**. + +--- + +## Blockers and prerequisites + +- **P1 — #242, a wedged-but-live `hyperd` is never recovered.** Reproduced at + >30 s. Acceptable for a tool a developer can restart; not acceptable for a + library that silently enrolled the caller in a resident service. **Gate.** +- **P2 — the security floor of D6.** UDS/named-pipe transport with enforced + permissions, `0600` discovery file, token-gated destructive commands, and an + answer to the engine-credential question. **Gate for any third-party-facing + mode.** +- **P3 — UDS performance is unvalidated.** `HyperProcess` defaults to TCP + explicitly "until UDS performance is validated" + ([`process.rs:362`](../../../hyperdb-api/src/process.rs)) and the daemon + forces TCP. D6's first bullet cannot land until that validation happens. +- **P4 — the benefit is unmeasured.** D9. **Gate.** +- **P5 — macOS CI skips all eight daemon restart tests**, and the Windows suite + has two open flakes (#288, #268). Promoting the daemon to a library capability + while its crash-recovery tests run on one platform is not defensible. +- **P6 — the daemon has zero field exposure.** Its newest fix merged the same + day as the release PR. This is an argument for elapsed time, not more code. + +--- + +## Open questions + +Each needs a human decision or an experiment. None should be resolved by +argument alone. + +- **Q1 — Does a session on a shared `hyperd` see another session's attached + databases?** The API models attach per-connection and the MCP's own registry + is per-process, but **no test proves cross-session invisibility on one + shared instance**; the closest evidence uses two private engines, which + proves something else. *Needs an experiment*: two connections to one + `hyperd`, A attaches a file with an alias, B queries that alias and also + enumerates `pg_catalog.pg_database`. If B can see it, the isolation model in + D5 is weaker than assumed and cohort scoping (D4) moves from + strongly-recommended to mandatory. +- **Q2 — Should the shared `hyperd` require credentials?** Generated password + in the `0600` discovery file, versus documenting that endpoint reachability + equals full authority. *Needs a decision*, because D6's floor is incomplete + without it and the choice determines whether UDS permissions are the whole + authorization story or merely part of it. +- **Q3 — Lease refcounting, or heartbeat-only idle?** Leases avoid shutting the + engine out from under an idle-but-live client, at the cost of new control + commands and a protocol bump. *Needs a decision* on whether that complexity + is warranted for the first release, or whether a generous idle timeout plus + heartbeats is enough. +- **Q4 — What is the default cohort?** Derived from the executable path? The + crate name? A required explicit argument with no default? A path-derived + default silently splits cohorts when a binary is rebuilt to a different + location; a required argument is more honest but less ergonomic. *Needs a + decision.* +- **Q5 — Should `Connection::new` be generalised from `&HyperProcess` to a + trait before `1.0.0`?** D3 recommends no. This is the **only** identified + `hyperdb-api` change whose cost rises after the freeze, so it needs an + explicit human confirmation rather than a default. *Deadline: `1.0.0`.* +- **Q6 — Do the C++/Python/Java Hyper APIs, or upstream Hyper itself, already + have a shared-instance concept?** Nothing in this repository describes one, + but that is absence of evidence. *Needs external verification* against + upstream documentation before "novel" is claimed publicly, and terminology + should be aligned rather than invented if a concept already exists. +- **Q7 — Should `hyperdb-mcp` keep its takeover behaviour after extraction?** + D7 says libraries must never take over, but the MCP's binary-upgrade UX + currently depends on it. *Needs a decision* on whether takeover becomes a + CLI-only operator action, and what the MCP's upgrade path looks like if so. + +--- + +## Recommendation + +**Do a narrow subset. Do the extraction now; gate everything else.** + +1. **Extract `daemon/*` into `hyperdb-daemon` before `1.0.0`.** Worth doing on + issue #276's merits alone, independent of this feature, and the window + closes at the freeze. Cheapest phase; answers a standing open question. +2. **Measure startup cost and `hyperd` RSS.** Cheap, and it gates whether any + of the rest is justified. +3. **Fix the security floor and #242** before offering anything to third + parties. +4. **Then, if the measurement holds, ship cohort-scoped `acquire()`** as a + `0.1.0` of the new crate, after `1.0.0`, with `Private` as the default and no + silent fallback. +5. **Do not build cross-application sharing.** The instance-global + `memory_limit`, the shared blast radius, and the `--no-password` endpoint + make it indefensible, and none of the three is fixable in this repository. + +The parts that should **stay** in `hyperdb-mcp` are all of the policy: +attachment replay, the persistent/ephemeral database model, the catalog, the KV +store, watched directories, the doctor, and the product's own version-takeover +UX. + +What the user's proposal gets right is that substantial engineering went into +this daemon and it is under-leveraged. What it gets wrong is the direction: the +daemon does not need to descend into `hyperdb-api`, because the connect path is +already there. It needs to be lifted out of `hyperdb-mcp` into a crate of its +own — which is a smaller, cheaper, and more useful change than the one +proposed, and it is the one with an actual deadline.