docs: design exploration for promoting the shared hyperd daemon to a first-class API capability

[StefanSteiner]

What this is

A design exploration — documents only, no code. Two Markdown files under
docs/superpowers/, following the repo's spec-then-plan convention:

• docs/superpowers/specs/2026-09-06-shared-hyperd-daemon-api-design.md — the spec
• docs/superpowers/plans/2026-09-06-shared-hyperd-daemon.md — a phased plan

Nothing is decided. These are written to be argued with, and two of their
conclusions are negative. Please push back.

The question explored: hyperdb-mcp runs a resident hyperd that MCP
sessions share, while every other hyperdb-api consumer spawns its own via
HyperProcess. Should that daemon become a capability any consumer can use,
so short-lived processes stop paying hyperd startup cost?

Recommendation

Do a narrow subset. Do the extraction now; gate everything else. Don't build
the ambitious version at all.

1. Extract hyperdb-mcp/src/daemon/* into a new hyperdb-daemon crate,
   before 1.0.0. Worth doing on issue report_hyperd_error_to_daemon changed its public signature in a patch release with no BREAKING changelog entry #276's merits alone; the window
   closes at the freeze.
2. Measure hyperd startup cost and memory. Neither exists in the repo
   today. Gates everything after it.
3. Fix the security floor and fix(mcp): recover when a live hyperd becomes unresponsive #242 before offering anything third-party.
4. Then, if the numbers hold, ship cohort-scoped acquisition as a 0.1.0
   of the new crate, after 1.0.0.
5. Do not build cross-application sharing.

Two findings that reshape the proposal

The framing is wrong in a useful way. hyperdb-api already connects to a
hyperd it does not own — Connection::connect, AsyncConnection::connect,
ConnectionBuilder, PoolConfig, and GrpcConnection::connect all take a
bare endpoint and hold no HyperProcess, and none stops hyperd on drop. The
MCP's own daemon mode is built out of exactly those calls
(engine.rs:616 with hyper: None). What's actually missing is discovery
and supervision — and supervision needs a binary, so the daemon belongs in a
crate above hyperdb-api, not moved down into it. The arrow points the
opposite way from what the proposal assumes.

Consequence: the minimum viable slice adds nothing to hyperdb-api — no
public API, no dependency, nothing entering the 1.0 freeze. Connection pooling
over a shared daemon works today with no new API, because PoolConfig.endpoint
is already a string.

Cross-application sharing should not be built. memory_limit is a Hyper
instance-global parameter defaulting to 80% of host RAM, and greps find no
per-session equivalent anywhere. One tenant can apply memory pressure to every
other tenant and there is no knob to prevent it — that's a property of the
engine, not a gap in the daemon. Combine it with a shared crash blast radius
(attach state does not survive a restart, and replay is MCP policy) and the
honest documentation would read "an unrelated application can exhaust your
memory budget and kill your engine, and you must rebuild your session state
when it does." Instead the design proposes cohort-scoped daemons: share
within one application and trust domain, which is where all the claimed
benefit actually is.

Bearing on #276

Issue #276 asks whether hyperdb_mcp::daemon::health should be public at all
before 1.0. This design answers it: no. The module is public by accident of
module organisation — nothing about running an MCP server requires callers to
reach into its daemon control protocol. The surface belongs to a crate whose
stated job is exactly that, where it can carry its own compatibility promise.

That gives Phase 1 a real deadline: removing pub mod daemon after 1.0.0 is
a major version; before it, free. Phase 1 is worth executing even if every
other phase here is rejected. The plan also proposes versioning the control
protocol separately from crate semver, which addresses #276's root cause — a
wire protocol whose compatibility axis was crate semver.

Security and trust boundary

Three confirmed weaknesses, and they chain:

1. The shared hyperd runs --no-password with --init-user=tableau_internal_user
   (process.rs:488, :541), and Connection::connect hardcodes that user
   with no password (connection.rs:221).
2. daemon.json is written with default umask permissions — no
   set_permissions anywhere (discovery.rs:236), so under umask 022 it is
   world-readable, publishing the endpoint.
3. The control port authenticates nothing (health.rs:232); any local
   process can STOP the daemon, or trip RESTART_LIMIT with three
   REPORT_HYPERD_ERROR messages.

Chained, 1 + 2 is a confused-deputy privilege escalation: another local UID
reads the file, connects with no password, and borrows the daemon owner's
filesystem authority over .hyper files. Thin risk for a single-user dev tool;
materially different as a standing machine-wide service, with CI runners the
worst case. Critically, fixing the control port alone would be security
theater — the --no-password engine endpoint is the real surface.

Honest assessment of the asset

The daemon has been resident-by-default since v0.5.0 in early June, but its
entire hardening history is a single working day (2026-09-06: #267, #279, #280,
#286-carrying-#278), with nine distinct concurrency and lifecycle defects found
in that window. The load-bearing finding: the eight crash-and-restart tests
were #[ignore]d unconditionally, so it shipped for three months with no
automated crash-recovery coverage on any platform — and when #279 turned them
on, CI went red and stayed red for two commits. The recent verification
discipline is above average, but this code has zero field exposure, macOS
CI still skips all eight restart tests, and #242 (a wedged-but-live hyperd is
never recovered, >30s observed) and #118 remain open.

Open questions needing a decision

Ordered by deadline. Full tradeoffs are in the spec.

Needs	Question	Deadline
Decision	Does the extraction re-export daemon for compatibility? Rec: no	1.0.0
Decision	Is a refactor(daemon)!: breaking marker acceptable inside the -rc line?	1.0.0
Decision	Generalise Connection::new from &HyperProcess to a trait? Rec: no	1.0.0
Decision	Does the measured benefit justify the feature at all?	after measuring
Decision	Generated password for the shared hyperd, or document reachability as full authority?	before the feature
Decision	What is the default cohort — executable path, crate name, or required explicit?	before the feature
Decision	Does hyperdb-mcp keep version takeover, or does it become CLI-only?	before the feature
Experiment	Does one session see another's attached databases on a shared hyperd? Currently unproven either way	before the feature
Experiment	Do the C++/Python/Java APIs or upstream Hyper already have a shared-instance concept?	before claiming novelty

The first experiment matters most: the API models attach per-connection and the
MCP registry is per-process, but no test proves cross-session invisibility on
one shared instance — the closest evidence uses two private engines, which
proves something else.

Quantified benefit — and a gap

Figures are labelled measured vs. estimated. Neither docs/BENCHMARK_GUIDE.md
nor docs/hyperd-release-benchmarks.md contains a hyperd spawn-to-usable
wall clock, and no hyperd memory figure exists anywhere in the repo. The
best available are ~156 ms (first embedded Hyper in a proc-macro host, hardware
unstated) and "10+ seconds under load" as a CI upper bound. So measuring is a
gate, not a follow-up, and the spec says how.

What the repo does have is multi-connection data, and it cuts against the
premise for one workload: AsyncArrowInserter measures 68.90 M/s at 1
connection vs 48.47 M/s at 4, with the guide stating plainly that
"parallelism no longer helps Arrow inserts." A shared daemon whose tenants all
ingest concurrently contends on exactly the workload where extra connections
already measure negative.

The win is narrower than the proposal implies: clear for test suites and
repeatedly-invoked CLIs, neutral-to-negative for the single long-lived process
most library users actually are.

Verification

• npx markdownlint-cli2 (no arguments): 0 issues, 70 files — baseline was
  0 in 68, plus the 2 files this PR adds.
• No Rust source modified. git show --stat is two .md files only.
• No workflow, crate version, manifest, or root changelog touched. No
  Release-As: footer. Release-please PR chore: release main #282 untouched.
• No changelog entry added: docs-only, no public API change. The plan records
  where entries will be needed per phase.

Please do not merge

This is an exploration for review. Refs #276, #270, #242, #118.

[StefanSteiner]

Closing immediately and unpublishing this branch.

This design exploration included a concrete threat model for the shared-daemon trust boundary, which turned out to describe an unpatched weakness in code that ships today rather than a hypothetical property of a future feature. This repository is public, and SECURITY.md directs security reports to https://www.sfdc.co/SubmitVuln rather than to the issue tracker, so that analysis should not have been published here first.

The design work itself is retained offline and will be reopened with the trust-boundary section reduced to a non-actionable pointer once the underlying finding has been reported through the documented channel.