diff --git a/AGENTS.md b/AGENTS.md index cf41f96c..366a084a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,11 +12,11 @@ network**. You describe services in a YAML manifest; stunt serves them on local over real TLS with subdomain routing). Stateful behavior (databases, tokens, webhooks) comes from adapters written in a **sandboxed Starlark** scripting layer. -**Install:** `brew install stuntapi/tap/stunt` · **Run from source:** `go run ./cmd/stunt` · +**Install:** `brew install --cask stuntapi/tap/stunt` · **Run from source:** `go run ./cmd/stunt` · **One-shot demo:** `stunt demo` -> **Adapters are embedded.** All reference adapters ship INSIDE the binary -> (3.4 MB). `stunt catalog search` lists all 91 offline, and +> **Adapters are embedded.** All reference adapters ship INSIDE the binary. +> `stunt catalog search` lists all of them offline, and > `stunt adapter add ` resolves a bundled adapter to an `embedded:` > source that `stunt up` extracts from the binary — no git clone, no network. > Use `git:` / local-path sources for community or custom adapters. diff --git a/README.md b/README.md index bce20b42..2d7e468b 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,14 @@ stunt demo # boots a stateful Stripe-style sim; prints copy-paste curl th # lists it back (stateful!), captures it, and fires a webhook — all locally ``` +**Contents:** [Why](#why) · [Install](#install) · [Quickstart](#quickstart) · +[Manifest](#the-manifest-stuntyaml) · [Rules](#rules--declarative-behavior) · +[Profiles](#profiles--runtime-activatable-behavior) · [Adapters](#adapters) · +[Transports](#transports) · [State & primitives](#state--primitives) · +[Webhooks](#webhooks--events) · [Networking & TLS](#networking--tls) · +[Observability & lifecycle](#observability--lifecycle) · [Determinism](#determinism) · +[Safety](#safety--trust) · [Status](#status--roadmap) · [Reference](#reference--contributing) + --- ## Why @@ -79,14 +87,34 @@ stunt down # stop a backgrounded `stunt up` stunt clean # reset all adapter state to seed fixtures ``` -Inline, declarative service (no adapter needed) — probabilistic faults, templates, conditions: +Point your client at the served address (`YOURAPI_BASE_URL=http://127.0.0.1:8000`) and +run your tests. Every request is logged, state persists across requests and restarts, +and `stunt clean` gives you a fresh world. + +--- + +## The manifest (`stunt.yaml`) + +One file declares everything stunt serves. A service is either **rules-only** +(declarative responses, no code) or **adapter-backed** (a directory of Starlark handlers ++ state — see [Adapters](#adapters)): ```yaml version: 1 -rng_seed: 42 -network: { mode: port, base_port: 8000 } +rng_seed: 42 # fixed seed → identical synthetic data + fault rolls every run +network: + mode: port # one port per service: base_port, +1 each, alphabetical + base_port: 8000 + services: - example: + stripe: + adapter: embedded:stripe-style # bundled IN the binary — nothing to clone, no network + config: + webhook_url: http://127.0.0.1:9999/hooks # where events_emit() delivers webhooks + myapi: + adapter: ./adapters/myapi-style # local dir, or git:github.com/org/repo@ref + max_body_bytes: 8388608 # per-request body cap (default 1 MiB; oversize → 413) + example: # rules-only service — inline declarative behavior rules: - match: { method: GET, path: /hello } when: { chance: 20 } # 20% of replies error @@ -95,20 +123,156 @@ services: respond: { status: 200, body: { template: '{"message":"hi","id":"{{ faker.ID "k" }}"}' } } ``` -Stateful adapter service (the bundled Stripe-style sim): +**Networking modes:** `port` (each service on `127.0.0.1:`; `network.mode` is +required — pick one explicitly) or `subdomain` +(real TLS via a locally-generated CA, SNI-routed `https://stripe.localhost`-style hosts; +see [Networking & TLS](#networking--tls)). + +**Adapter sources:** `embedded:` (ships in the binary) · a local path · +`git:github.com/org/repo@ref`. `stunt adapter add ` wires a source into the +manifest for you. + +--- + +## Rules — declarative behavior + +Rules answer requests without any code. Within a service they evaluate **in declaration +order, first match wins** — a catch-all `match: { path: "/**" }` is the usual 404 +backstop: + +```yaml +rules: + - name: flaky-when-debugging + match: + method: GET + path: /flaky/** # glob over the path + headers: { X-Debug: "1" } # header conditions + when: + chance: 20 # percent probability, 0..100 (rng_seed-driven) + # expr: "request.body.amount > 1000" # OR a boolean expression over request.* + respond: + status: 503 + headers: { Content-Type: application/json, Retry-After: "1" } + body: { inline: { error: simulated } } # OR { file: body.json } + # OR { template: tmpl.json } + latency_ms: 100 # simulated latency + # behavior: timeout # force a hang; drops the connection after + # latency_ms (default 30s) +``` + +**Templates** are Go `text/template` with fake-data helpers — `{{ faker.Email }}`, +`{{ faker.ID "k" }}`, `{{ uuid }}`, `{{ now.Format "2006-01-02T15:04:05Z07:00" }}` — so +even rules-only services return varied, realistic, deterministic payloads. + +Rules-only services cover static and probabilistic mocking; when you need **state** +(create → list → mutate), **auth flows**, or **webhooks**, use an adapter. + +--- + +## Profiles — runtime-activatable behavior + +A **profile** is a named behavior mode you can switch on and off at runtime — the +on-demand version of "what does my client do when the dependency misbehaves?". +Retry/backoff paths, circuit breakers, degraded UX: activate the profile, run the test, +deactivate. No YAML edits, no restart. + +```bash +$ stunt profile activate launch-day +activated preset "launch-day" +(runtime-only — resets on restart; `stunt up --profile` boots with one) + +$ curl -s http://127.0.0.1:8000/v1/charges | head -c 60 +{"error":"rate_limit_error"} # the world changed — same YAML, no restart + +$ stunt profile deactivate +deactivated all profiles +``` + +### The three ways a profile comes to exist + +**1. Rule bundles per service** — declared right in `stunt.yaml`. While active, the +rules run as a **pre-dispatch override layer**: they intercept requests *before* +handlers and base rules, so they reach handler-backed routes that base rules cannot +(matching your real fault-injection needs — chaos that can't touch `/v1/charges` would +be useless). ```yaml -version: 1 -network: { mode: port, base_port: 8000 } services: stripe: - adapter: ./adapters/stripe-style - config: - webhook_url: http://localhost:9090/webhook # events_emit() delivers here + adapter: embedded:stripe-style + profiles: # rule bundles for THIS service + degraded: + description: occasional 429s + slow responses + rules: + - match: { path: /v1/** } + when: { chance: 30 } + respond: { status: 429, body: { inline: { error: rate_limited } } } ``` -Then `curl http://127.0.0.1:8000/v1/charges -H "Authorization: Bearer sk_test_demo" -d '{"amount":1000,"currency":"usd"}'`, -list it back, capture it — state persists across requests and restarts. +**2. Adapter-authored modes** — an adapter ships behavior modes its handlers implement, +so the *provider's own* degraded semantics come pre-modeled. The sqs-style adapter +declares `throttled` in its `adapter.yaml`; its handlers read the `profile_active()` +builtin: + +```yaml +# adapters/sqs-style/adapter.yaml +profiles: + throttled: "alternate ReceiveMessage calls return empty — exercise consumer retry/backoff paths" +``` + +```python +# inside a handler: +if profile_active() == "throttled": + return respond(200, {"Messages": []}) # this receive yields nothing; client retries +``` + +**3. Global presets** — one activation assigns profiles across services, so a whole +scenario flips at once: + +```yaml +profiles: # top level of stunt.yaml + launch-day: # `stunt profile activate launch-day` + description: both dependencies degraded + set: + stripe: degraded # the manifest bundle from (1) + sqs: throttled # authored by the sqs-style adapter in (2) +``` + +Nothing predefined ships — presets like `launch-day` are yours to declare. A name +declared in *both* the manifest and an adapter activates both layers together (manifest +rules + the handler behavior), which is the point: one name, one world. + +### Driving profiles + +```bash +stunt profile list # every activatable profile, active ones marked +stunt profile show launch-day # what it sets, where it's active +stunt profile activate launch-day # preset +stunt profile activate throttled # unique name → auto-targeted to its service +stunt profile activate degraded --service stripe # disambiguate a shared name +stunt profile deactivate # all services +stunt profile deactivate --service sqs +stunt up --profile launch-day # boot default (unknown names fail before serving) +``` + +The dashboard's **profiles** panel does the same with one click, and the read commands +(`stunt profile list`, `stunt requests`, `stunt ps`, …) print `--json` for scripts. + +**Semantics worth knowing:** + +- **Runtime-only by design** — activation is server state, not config; a restart resets + the world (`stunt up --profile` restores a default if you want one). +- **Precedence** — active profile rules run before handler/base-rule dispatch (see (1)). + WebSocket upgrades and GraphQL dispatch earlier still, so profiles don't intercept + those two transports. +- **Determinism** — chance rules inside a profile draw the same per-service stream as + base rules (fixed `rng_seed` → reproducible fault rolls). Adapter modes that keep + counters across calls (like sqs-style's alternating throttling) persist the counter + in service state: restart resets the *activation*, not the counter — pair those with + `stunt reset ` for a fully fresh sequence. Details in the + [determinism contract](#determinism). + +--- ## Adapters @@ -164,66 +328,147 @@ exact signatures, gRPC/WebSocket/GraphQL sections): **[`adapters/README.md`](ada signed webhook delivery (HMAC/ECDSA/Ed25519 schemes), derive-on-read async state machines (RUNNING/FAILED + failure injection), multipart uploads, byte-exact binary round-trips. -## Transports & primitives - -- **Transports**: REST, gRPC (unary **and streaming**), WebSocket, GraphQL (full introspection + DoS limits). -- **Rules engine**: first-match-wins; `match` (method/path globs/headers), `when.chance` (probabilistic), - `when.expr` (boolean over `request.*`), `respond` (status/headers/latency/timeout + `body.inline|file|template`). -- **Primitives**: Collection (SQLite), KV, Blob (FS), Identity (HMAC tokens), Events (webhooks w/ retry), - Clock+scheduler (deterministic), Generator, Validator (JSON-Schema). State persists in `.stunt/state/`. -- **Networking**: optional portless.dev-style TLS proxy on `*.localhost` (HTTP/2, local CA). The privileged - listener forwards to an **unprivileged** engine, so adapter code never runs as root. WSS passthrough verified. -- **Profiles**: runtime-activatable behavior modes, all declarative. Declare rule bundles per service in - `stunt.yaml` (`profiles:`), ship modes in an adapter's `adapter.yaml` (handlers read `profile_active()`), - compose global presets that assign several services at once. `stunt profile activate ` (or the - dashboard panel) flips the world — "launch-day latency", "degraded dependency" — without touching YAML or - restarting; active profile rules run as a pre-dispatch override, so they reach handler-backed routes base - rules can't (WebSocket and GraphQL transports dispatch earlier — profiles don't intercept those). Runtime-only - by design: a restart resets the world; `stunt up --profile ` boots with one. - - ```yaml - services: - stripe: - adapter: embedded:stripe-style - profiles: # rule bundles for THIS service - degraded: - description: occasional 429s - rules: - - match: { path: /v1/** } - when: { chance: 30 } - respond: { status: 429, body: { inline: { error: rate_limited } } } - profiles: # one activation assigns across services - launch-day: - set: - stripe: degraded - sqs: throttled # authored by the sqs-style adapter - ``` - - ``` - $ stunt profile activate launch-day - $ stunt profile activate throttled # unique name → auto-targeted - $ stunt profile deactivate - ``` - Determinism contract: with a fixed `rng_seed`, chance rules are deterministic for serial - traffic from boot (per service); parallel traffic preserves counts, not order; every chance - rule on a service draws one shared stream. - -## Observability dashboard - -Every running server serves its **own localhost dashboard** — a live request inspector (HTTP/gRPC/WebSocket, bodies, copy-as-curl, replay), a **state browser** (the collections/kv/blobs your tests created), **snapshot/restore** for deterministic runs, and an **instance manager**. A matching CLI (`--json`) backs every feature. +--- + +## Transports + +- **REST** — routes with `{param}` captures, per-method handlers, query/body/header + access, streaming request/response bodies, multipart. +- **gRPC** — unary **and streaming** (server/client/bidi) from a real + `FileDescriptorSet` your adapter ships; clients work unmodified against the local address. +- **WebSocket** — connect/message/disconnect handlers; the Discord-style adapter's + Gateway (HELLO→IDENTIFY→READY→dispatch) is the reference implementation. +- **GraphQL** — schema-first with a Starlark resolver layer, full introspection, + query-depth/complexity limits so a pathological query can't wedge your test run. + +Mix transports in one adapter (echo-style serves gRPC + WebSocket from the same manifest). + +--- + +## State & primitives + +Adapters (and the engine) run on a small set of stateful primitives. State lives in +`.stunt/state/` under the manifest, persists across requests **and restarts**, and +resets only when you say so: + +| Primitive | What it's for | +|---|---| +| **Collection** | documents in SQLite — insert/get/list/update/delete | +| **KV** | key-value + atomic counters (`store_kv_incr` — id sequences) | +| **Blob** | binary/large content on the filesystem, byte-exact round-trips | +| **Identity** | HMAC-backed tokens — mint, validate, scopes; expiry paths for 401 tests | +| **Events** | webhook delivery with per-provider signing and retry (see below) | +| **Clock + scheduler** | deterministic time — virtual clocks for billing cycles, Test Clocks | +| **Generator** | synthetic data (the `{{ faker.* }}` templates) | +| **Validator** | JSON-Schema validation of requests/responses | + +Lifecycle commands: + +```bash +stunt clean # reset EVERYTHING to seed fixtures (state, CA, hosts) +stunt reset stripe # reset one service's state on a RUNNING server +stunt snapshot save -o pre-migration.tar.gz # capture the whole world +stunt snapshot load pre-migration.tar.gz # and put it back — deterministic replays +stunt state collections stripe # browse a service's state from the CLI (blobs/kv too) +``` + +--- + +## Webhooks — events + +Adapters emit webhooks like the real provider does: the stripe-style sim POSTs +`charge.created` to your sink when you capture a charge. Configure the destination once: + +```yaml +services: + stripe: + adapter: embedded:stripe-style + config: + webhook_url: http://127.0.0.1:9999/hooks # events_emit() delivers here +``` + +Handlers deliver with `events_emit("charge.created", {...})`; adapters whose providers +sign their webhooks (Stripe, Twilio, Square, GitHub, …) compute the **real signature +scheme** — HMAC-SHA256, ECDSA, Ed25519 — over the exact bytes, and expose the registered +target to handlers via `events_target()` (for providers that MAC the destination URL +into the signature). Delivery retries with exponential backoff, like a real provider +would. + +--- + +## Networking & TLS + +**Port mode** (default): each service on `127.0.0.1:`, starting at `base_port`. +Zero setup; point clients at the port. + +**Subdomain mode**: real **TLS** with per-service subdomains — +`https://stripe.localhost`, `https://sqs.localhost` — via a locally-generated CA, an +SNI-routing reverse proxy, and a managed `/etc/hosts` block: +```yaml +network: + mode: subdomain + tld: localhost + tls: true + sync_hosts: true # manage the /etc/hosts block for *.tld + # spoof_real_hosts: true # redirect REAL hostnames (api.stripe.com) to the local sim ``` + +```bash +stunt trust # install stunt's CA into the system trust store (privileged) +stunt proxy start # start the TLS reverse proxy +stunt hosts sync # manage hosts entries manually if needed +``` + +The privileged listener forwards to an **unprivileged** engine, so adapter code never +runs as root; HTTP/2 and WSS pass through verified. + +--- + +## Observability & lifecycle + +Every running server serves its **own localhost dashboard** — a live request inspector for +HTTP traffic (bodies, headers, copy-as-curl, replay), a **state browser** (the +collections/kv/blobs your tests created), **snapshot/restore** for deterministic runs, +the **profiles** panel, and an **instance manager**. A matching CLI (`--json`) backs +every feature: + +```bash $ stunt up dashboard: http://127.0.0.1:54321 (token: 9f3c…) -$ stunt ui # open it -$ stunt requests --follow # live feed in the terminal -$ stunt ps # list running servers across all manifests +$ stunt ui # open it +$ stunt requests --follow # live request feed in the terminal +$ stunt replay # re-issue a captured request +$ stunt ps # every running stunt server, across manifests +$ stunt doctor # health check: CA, manifest, adapters, ports ``` -Loopback-only, token-authed, DNS-rebinding-guarded; sensitive headers redacted; logging is async (never backpressures requests). Full guide with screenshots: **[`docs/dashboard.md`](docs/dashboard.md)**. +Servers stop gracefully (`stunt stop` / `stunt down` drain in-flight requests), work on +every OS, and can run as a system service (`stunt service install`). Loopback-only, +token-authed, DNS-rebinding-guarded; sensitive headers redacted; logging is async and +never backpressures requests. Full guide with screenshots: +**[`docs/dashboard.md`](docs/dashboard.md)**. ![Request inspector](docs/img/dashboard-hero.png) +--- + +## Determinism + +The whole point of a stunt double: the same run twice must behave the same way. + +- **`rng_seed`** fixes synthetic data (same seed → same fakes, same ids) and fault + rolls (`when.chance`), per service, from a fresh boot. +- **Chance rules** draw one shared per-service stream in evaluation order — traffic to + a chanced path shifts subsequent rolls on that service, so readiness-probe a + chance-free path. Parallel traffic preserves failure *counts*, not per-request order. +- **The clock** is virtual where adapters model time (billing cycles, token expiry, + Test Clocks) — no sleeping in tests. +- **`stunt clean` / `stunt reset`** restore the seed world; **`stunt snapshot`** + captures and restores mid-run state for replay-style tests. + +--- + ## Safety & trust The defining property: **a community adapter is safe to install** — adapter logic is sandboxed @@ -231,6 +476,8 @@ Starlark with no host I/O, bounded by execution-step limits; all file reads an a are path-containment-guarded; `stunt adapter lint` enforces synthetic-data-only. See **[SECURITY.md](SECURITY.md)** for the full threat model. +--- + ## Contributing Contributions are welcome — especially **adapters**. See **[CONTRIBUTING.md](CONTRIBUTING.md)** @@ -239,18 +486,14 @@ lint-adapters). Quick path: `stunt adapter new myapi-style` → edit → `stunt ## Status & roadmap -**v0.3.0 — cross-platform lifecycle + Starlark ergonomics.** `stunt stop`/`down` -now work on **Windows** (previously they errored and left the server holding its -port) and shut servers down **gracefully** on every platform via a dashboard -`POST /api/shutdown` endpoint (draining in-flight requests instead of hard-killing). -In Starlark handlers, `req` supports attribute access (`req.method`, `req.headers`) -alongside dict access, and header lookups are now case-insensitive. - -**v0.2.2 — observability dashboard.** Core plus a full per-server dashboard (live -request inspector, state browser, snapshot/restore, instance manager) are built, -self-tested (`just ci` green), and dogfooded. On the roadmap: a **public catalog** -(today's `stunt catalog` is offline/bundled + git refs), `stunt setup` privileged-path -hardening, and broader adapter coverage. +Every reference adapter ships embedded in the binary, and nearly all are verified by +real test suites — official provider SDKs driven end-to-end in CI plus engine-level +suites — with the per-adapter scorecard published in +**[`CONFORMANCE.md`](CONFORMANCE.md)**. The website ([stuntapi.com](https://stuntapi.com)) +carries the live conformance matrix and adapter catalog. + +On the roadmap: a **public catalog** (today's `stunt catalog` is offline/bundled + git +refs), `stunt setup` privileged-path hardening, and broader adapter coverage. **Not planned for v1**: GraphQL subscriptions, npm adapter distribution. Found a security issue? See **[SECURITY.md](SECURITY.md)** — do not open a public issue. @@ -266,4 +509,7 @@ Found a security issue? See **[SECURITY.md](SECURITY.md)** — do not open a pub - **Operating guide:** `AGENTS.md` (or run `stunt llm` for the in-binary reference) — the full manifest schema, CLI reference, and the complete Starlark handler API. +- **Adapter authoring:** `adapters/README.md` — the `adapter.yaml` schema and the complete + Starlark builtins reference with exact signatures. +- **Dashboard guide:** `docs/dashboard.md`. - **Contributing:** see `CONTRIBUTING.md`. diff --git a/docs/dashboard.md b/docs/dashboard.md index 1b591023..4f5a9c8d 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -43,16 +43,16 @@ running server from there, so you almost never need `--url`/`--token`. ## 1. Request inspector -The **requests** tab is a live feed of every request hitting your sims — REST, -gRPC (unary + streaming), and WebSocket — with method, path, status, transport, and -**sub-microsecond** timing. +The **requests** tab is a live feed of every HTTP request hitting your sims — with +method, path, status, and **sub-microsecond** timing. (gRPC and WebSocket traffic is +served and exercised by your tests, but is not captured in the request log.) ![Request inspector — live feed](img/dashboard-hero.png) ### What's captured Each row records: a monotonic **sequence number** (gap-free ordering), timestamp, -**service**, **transport** (`http`/`grpc`/`ws`), **method**, **path**, **status**, +**service**, **method**, **path**, **status**, **duration** (microseconds), and the request/response **headers** + **bodies**. - **Bodies are captured by default.** This is a localhost dev tool whose killer diff --git a/internal/cli/llm.go b/internal/cli/llm.go index d87b1842..cd6b776c 100644 --- a/internal/cli/llm.go +++ b/internal/cli/llm.go @@ -63,7 +63,7 @@ and an instance manager. ` + "`stunt ui`" + ` opens it; every command below has demo one-shot stateful Stripe-style demo doctor CA + manifest + adapter + port health check clean wipe state, CA, hosts block (keeps manifest + adapters) - catalog search|show discover adapters (--json) # all 95 embedded, works offline + catalog search|show discover adapters (--json) # all embedded, works offline adapter new|add|lint|test|list build/validate adapters (lint MUST pass) hosts sync|clean manage /etc/hosts (subdomain TLS mode) proxy start TLS reverse proxy (subdomain mode)