From e54f066552f09a77f0d77f73fd9d0b57893c8cd2 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 31 Jul 2026 00:06:07 -0700 Subject: [PATCH 01/24] Add design specs for the provider architecture and permission model epic Five specs covering the work proposed in PR #838 (issues #777-#782), written so the next implementation pass has a normative behavioral contract: - Pluggable providers: identity lifecycle contract (mint/recognize/hash/ tombstone), trait minimalism, adapter parity, validation table - Permission model: signal precedence (opt-out over TCF), fail-closed jurisdiction resolution, policy file validation, decision-matrix testing - Migration and rollout: behavior-preservation matrix, ID stability vectors, loud-failure requirements, operator recipes - Client-cycle EC resolve endpoint: threat model and prerequisites; on hold until its open questions get an issue - Integration response-header hook: #782 contract with ordering and collision policy, ships only with a real consumer --- ...26-07-30-client-cycle-ec-resolve-design.md | 120 +++++++++ ...integration-response-header-hook-design.md | 69 +++++ .../2026-07-30-permission-model-design.md | 254 ++++++++++++++++++ .../2026-07-30-pluggable-providers-design.md | 239 ++++++++++++++++ ...07-30-provider-migration-rollout-design.md | 161 +++++++++++ 5 files changed, 843 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md create mode 100644 docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md create mode 100644 docs/superpowers/specs/2026-07-30-permission-model-design.md create mode 100644 docs/superpowers/specs/2026-07-30-pluggable-providers-design.md create mode 100644 docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md new file mode 100644 index 000000000..674de01ff --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -0,0 +1,120 @@ +# Design Spec: Client-Cycle Edge Cookie Providers and the Resolve Endpoint + +**Status:** Draft — **prerequisites unmet; do not implement against this spec +until its open questions (§7) are resolved in a dedicated issue** +**Author:** Engineering +**Issue references:** none yet (this spec exists to force one; #778 does not +cover this feature) +**Related specs:** `2026-07-30-pluggable-providers-design.md` +**Last updated:** 2026-07-30 + +> **Context.** PR #838 shipped, undeclared and unspec'd, a second provider +> _type_: a "client-cycle" EC provider whose identifier is established by a +> browser POST to a new public endpoint (`POST /_ts/api/v1/ec/resolve`), +> plus a demo provider (`client-fixed`) and a JS bundle. Review found the +> endpoint accepted cross-origin identity-setting posts with no origin +> check, minted cookies with no identity-graph row (violating an invariant +> the organic path enforces explicitly), was registered on only one of four +> adapters, and could never round-trip because the core did not recognize +> non-HMAC identifiers. None of that is an argument the feature is a bad +> idea — vendor identity systems with a browser leg (e.g. signed-envelope +> schemes) are a real integration target. It is an argument that the feature +> needs a threat model before an implementation. This spec is that threat +> model and the bar an implementation must clear. + +--- + +## 1. Overview + +A **client-cycle** EC provider establishes the identifier via a browser +round trip: server-injected first-party JS obtains or derives a value in the +page (typically a signed envelope from a vendor identity system), posts it to +a Trusted Server endpoint, and the endpoint — after provider-specific +verification — sets the first-party `ts-ec` cookie. + +This differs from server-side providers in one security-critical way: **the +identifier is attacker-influenceable input**, not server-derived evidence. +Everything in this spec follows from that. + +## 2. Threat model + +| Threat | Vector | Consequence if unmitigated | +| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Cross-site identity fixation** | `text/plain` POST is a CORS-simple request: any page on the web can `fetch(resolveUrl, {method: "POST", credentials: "include", body: payload})` with no preflight | An attacker pins a chosen identity onto a victim's first-party cookie jar — login-CSRF for the ad-identity layer; the victim's activity accretes to an attacker-controlled ID | +| **Replay** | A captured valid payload (from the attacker's own session or a leak) replayed against another browser | Same as fixation, without needing to mint payloads | +| **Phantom identity** | Endpoint sets the cookie without an identity-graph row | Later requests carry an EC that the KV graph has never seen; downstream sync and withdrawal logic operate on an identity that half-exists (the organic generation path explicitly refuses to write a cookie when the graph write fails, for exactly this reason) | +| **Un-tombstoneable identity** | Core does not recognize the provider's identifier shape | Withdrawal cannot expire or tombstone the identity — a compliance failure, not just a bug | +| **Amplification** | The page script cannot observe an HttpOnly cookie, so it cannot know the cookie is already set | A POST on every page view of every session (PR #838's JS gated on reading a cookie its own server marked HttpOnly, making the guard permanently false) | + +## 3. Requirements on the endpoint + +`POST /_ts/api/v1/ec/resolve` (final path TBD) MUST: + +1. **Reject cross-site requests.** Require a same-site assertion: `Origin` + (or `Sec-Fetch-Site: same-origin/same-site`) validated against the + publisher's origin set; requests without a validating header are + rejected. CSRF-token designs are acceptable but not required if + origin-based rejection is enforced. +2. **Verify the payload cryptographically per provider.** The provider's + `resolve_from_client` accepts only payloads that are signed by an + expected party, **audience-bound** to this publisher, and **expiring** + (bounded lifetime, single-use where the scheme allows). A provider whose + payloads are replayable constants fails this bar by construction. +3. **Preserve the identity-graph invariant.** The cookie is set only after + the corresponding graph row is written, mirroring the organic path. Graph + unavailable → no cookie, same as organic generation. +4. **Round-trip through the lifecycle contract.** The identifier set here + must be recognized, hashed, and tombstonable by the selected provider + (providers spec §3). The conformance suite runs against every + client-cycle provider. +5. **Exist on every adapter.** Route registration goes through shared route + wiring; the parity suite asserts the endpoint's presence and behavior on + all four adapters. (PR #838 registered it on Fastly only, so the same + config on the Axum dev server proxied the POST to the publisher origin.) +6. **Be uncacheable and permission-gated.** `Cache-Control: no-store`; + the same `store-on-device` permission gate as organic EC creation runs + before any cookie is set. + +## 4. Requirements on the page script + +- The re-post guard must not depend on reading an HttpOnly cookie. Either + the server injects a "resolved" marker the script _can_ read (a + non-identity companion cookie or an injected page variable), or the + endpoint is cheap-idempotent and rate-limited per session; the design must + state which and test it. +- The JS module ships through the standard integration bundle mechanism, + loaded only when a client-cycle provider is the selected EC provider. +- Any constant shared between Rust and TS (endpoint path, marker name) is + asserted equal by a test, not "kept in sync by hand". + +## 5. Demo providers + +A demonstration provider (fixed identifier, no verification) fails §3.2 by +design and therefore MUST NOT be selectable in a production build: gate it +behind a cargo feature or `#[cfg(test)]` so the settings validator does not +accept its key in release artifacts. PR #838's `client-fixed` was selectable +in any production config, giving every visitor the same identity, with a doc +sentence as the only guardrail. + +## 6. Testing + +- Endpoint: origin-rejection, expired/replayed/foreign-audience payload + rejection, graph-unavailable refusal, permission-gate refusal — each as an + integration test, not only unit tests. +- Browser round trip (JS → POST → Set-Cookie → next request recognized) in + the integration suite; PR #838 shipped the JS with in-process unit tests + only, including one asserting a state (reading the HttpOnly cookie) that + cannot occur in a real browser. +- Parity: all four adapters. + +## 7. Open questions — to be settled in the feature's issue before any code + +1. Which concrete vendor scheme is the first real consumer, and does its + envelope format satisfy §3.2 (audience binding, expiry)? If no concrete + consumer exists, the feature waits — the demo provider is not a + consumer. +2. Does the resolve flow need consent-state echo in its response (so the + page can react), and if so what is the minimal disclosure? +3. Rate limiting / abuse posture at the edge for an unauthenticated POST. +4. Whether the endpoint should be versioned separately from the identify + API family it sits beside. diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md new file mode 100644 index 000000000..59d354547 --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -0,0 +1,69 @@ +# Design Spec: Integration Response-Header Hook + +**Status:** Draft +**Author:** Engineering +**Issue references:** #782 +**Related specs:** `2026-07-30-pluggable-providers-design.md` +**Last updated:** 2026-07-30 + +> **Context.** Issue #782 already specifies this feature well; its done-when +> is the contract. PR #838 shipped the trait and registry wiring with **no +> adapter call site** — `apply_response_headers` had zero production +> callers, so the feature existed only in its own unit test. This short spec +> restates the contract plus the two details the issue left open (ordering +> and collision policy), and adds the rule that prevents a repeat: the hook +> lands with a consumer or not at all. + +--- + +## 1. Overview + +Integrations can today rewrite request-path behavior (proxies, attribute +rewriters, head injectors) but cannot mutate **response** headers. The hook +adds that: an integration registers a response-header mutator via its +`IntegrationRegistration` builder, and every adapter applies all registered +mutators to the outbound response for HTML document responses it processed. + +## 2. Contract + +- `IntegrationRegistration::builder(ID).with_response_mutator(...)` registers + a mutator; `IntegrationRegistry::apply_response_headers(...)` applies all + registered mutators in registration order. +- **Every adapter calls the apply point** on its outbound-response path for + processed documents. The call site lives in shared response-finalization + code where one exists; where adapters finalize independently, each adapter + gains the call and a test proving it. +- Mutators run **after** Trusted Server's own response-header handling + (EC Set-Cookie emission, EC header clearing, privacy headers) so a mutator + cannot be silently clobbered by later core steps — in PR #838's ordering, + provider-supplied headers were inserted before the EC header-clearing pass + and could be stripped by it. + +## 3. Collision policy + +- Mutators may not touch **reserved headers**: `Set-Cookie` for the EC + cookie, the `x-ts-*` namespace, and the consent/privacy headers core + emits. Attempts are dropped and logged at `warn` with the integration id. +- For non-reserved headers, the mutator API distinguishes **append** from + **replace** explicitly; the default is append. Replacing a header the + origin set is a deliberate act, visible in the mutator's code. +- Later registrations see earlier mutations (order = registration order, + which is deterministic). + +## 4. Done-when (from #782, sharpened) + +1. Trait + builder + registry application, each public item documented. +2. **At least one real consumer ships in the same PR** — an existing + integration registering a mutator for a real need (or, failing a real + need, the feature waits; scaffolding with only self-referential tests is + dead code and will be removed). +3. Every adapter applies mutations on its outbound path, with a per-adapter + route test asserting an integration-set header appears in the response. +4. A parity-suite case asserts identical mutation behavior across adapters. +5. Reserved-header and append/replace semantics covered by unit tests. + +## 5. Size + +This is a ~150-line feature plus tests. It has zero coupling to the provider +architecture or the permission model and should land as its own small PR, +first in the epic's sequence. diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md new file mode 100644 index 000000000..f3504a04d --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -0,0 +1,254 @@ +# Design Spec: Jurisdiction Permission Model + +**Status:** Draft +**Author:** Engineering +**Issue references:** #779 +**Related specs:** `2026-07-30-pluggable-providers-design.md`, +`2026-07-30-provider-migration-rollout-design.md` +**Last updated:** 2026-07-30 + +> **Context.** PR #838 proposed a permission model whose review surfaced two +> classes of defect this spec exists to prevent in the next pass: (1) silent +> behavioral inversions of consent-signal precedence — most seriously, a +> present TCF string short-circuiting GPC/GPP/US-Privacy opt-outs — and +> (2) fail-open jurisdiction resolution when geolocation is disabled. The +> precedence table (§4) and the failure-mode matrix (§6) are the two +> documents whose absence allowed those defects to hide in a 67-file diff. +> They are normative: an implementation whose behavior differs from these +> tables is wrong, whatever its tests say. + +--- + +## 1. Overview + +The permission model replaces the hard-wired jurisdiction gate +(`allows_ec_creation` and its companions) with a single resolved +**permission set** per request. Every data decision Trusted Server itself +makes — EC creation, EC withdrawal, EID transmission into the bidstream, +and provider execution (see providers spec §5) — reads that set. + +The set is resolved from three inputs: + +1. **Jurisdiction** — the country/region the request resolves to (§5). +2. **Policy** — a declarative, version-controlled map from jurisdiction to a + baseline acquisition rule per permission (§3). +3. **Signals** — the request's privacy signals: TCF, GPP, GPC, US Privacy + (§4). + +Scope: the model governs decisions Trusted Server makes. Downstream RTB +partners receive the full, unmodified regulatory context and make their own +compliance decisions. + +## 2. Vocabulary: enforced permissions only + +Permissions are named by IAB TCF Europe purpose identifiers, used strictly as +technical identifiers (no CMP or TCF policy is implemented by naming them). + +**Rule: a purpose appears in the model only when it has both a signal mapping +and an enforcement point.** PR #838 shipped 11 purposes of which 9 were +inert — computed into the bitset and consumed by nothing but a startup log — +while the policy file invited operators to set flags (e.g. +`market-research: denied`) that changed nothing. A policy vocabulary that +overstates what is enforced is a compliance hazard, not forward +compatibility. + +The initial vocabulary is therefore exactly: + +| Identifier | TCF purpose | Enforcement points | +| ------------------------- | ----------- | -------------------------------------------------------------------- | +| `store-on-device` | 1 | EC provider execution; EC creation; withdrawal/tombstone eligibility | +| `select-personalised-ads` | 4 | EID transmission into the bidstream (jointly with `store-on-device`) | + +(The identifier strings are the IAB names verbatim, including their original +spelling.) The extension procedure — add the signal mapping, add the +enforcement point, add the policy vocabulary entry, in one change — is +documented in the policy file header. Policy validation **rejects** a rule +that references an identifier outside the current vocabulary, so the file can +never promise more than the code enforces. + +## 3. Policy + +### 3.1 Format + +A YAML map, embedded at build time, of named **groups** (baselines) and +**rules** keying countries (`FR`) and country/state pairs (`US/CA`) to a +group with optional per-permission overrides. Each permission resolves to an +**acquisition rule**: + +- `granted` — set without any signal, +- `requires_signal` — set only when a signal grants it (opt-in), +- `denied` — never set, even when a signal grants it. + +Overrides support all three targets: `+perm` (granted), `-perm` (denied), and +`~perm` (requires_signal). PR #838 supported only `+`/`-`, making the most +common real-world override — "this state requires a signal for personalized +ads" — inexpressible without duplicating a whole group. Groups may use a +`default:` shorthand for unlisted permissions. + +### 3.2 Validation — at build time, not request time + +The embedded file is validated by a `build.rs` step (or an equivalent +always-run CI test that asserts the parse explicitly): a malformed committed +file fails the **build**, never a request. PR #838's file was parsed lazily +behind a `OnceLock` with an `expect`, meaning a bad edit that escaped unit +tests became a 500 on every request. + +Validation rejects: + +- unknown fields anywhere (`deny_unknown_fields` on every deserialized + struct — PR #838's untagged rule enum silently swallowed a misspelled + `permission:` key, dropping the operator's override with no diagnostic); +- rule keys that are not plausible ISO 3166-1 alpha-2 / ISO 3166-2 codes; +- references to permissions outside the enforced vocabulary (§2); +- groups that neither list every permission nor provide `default:`. + +### 3.3 One source of jurisdiction truth + +The codebase currently carries a second, runtime-configurable jurisdiction +list (`consent.gdpr.applies_in`) used by the auction consent gate. Two +independently maintained country tables that both express "where GDPR +applies" will drift (in PR #838, adding `CH` to one had no effect on the +other). Requirement: either the auction gate derives its jurisdiction class +from the same resolved policy, or a CI test asserts that every country in +`applies_in` resolves to an opt-in (`requires_signal`) baseline in the policy +file, and vice versa for the shipped defaults. + +### 3.4 Shipped table coverage + +A CI test asserts every member of the GDPR country list resolves to an opt-in +baseline (this is what catches a `DK:` typoed as `DL:` — a defect that in +PR #838 survived parse, startup, and all tests, silently dropping Denmark to +the operator default). Countries intentionally not listed are governed by +§5's default-country rules; the policy header documents that this is the +fallback, and the shipped example default is the most protective baseline. + +## 4. Signal precedence — normative table + +Signals are classified: + +- **Opt-out signals** (affirmative withdrawal): GPC header; GPP sections + carrying a sale/sharing opt-out; US Privacy opt-out. +- **Consent records**: a decodable TCF string (standalone or embedded in + GPP), which may grant or refuse individual purposes. + +**Precedence, highest first:** + +1. Policy `denied` — never set, regardless of any signal. +2. **Opt-out signal — always revokes**, regardless of any consent record + present. A GPC header revokes `store-on-device` and + `select-personalised-ads` even when an accompanying TCF string consents to + them. _(This is the rule PR #838 inverted: its resolution returned from + inside the TCF branch before ever reaching the opt-out check, so a + consenting CMP string made the browser's GPC signal a no-op — a + CCPA-facing regression. The pre-existing tests pinning this rule — + `ec_blocked_us_state_gpc_overrides_tcf` and companions — are reinstated + against the new API, not deleted.)_ +3. Consent record refusal — a TCF record present and refusing the purpose + revokes it. +4. Consent record grant — a TCF record present and consenting grants it + (subject to 1–2). +5. No signal — the policy baseline decides: `granted` sets it, + `requires_signal` leaves it unset. + +### 4.1 Decision matrix + +For each enforced permission, with baseline _B_ ∈ {granted, +requires_signal, denied}: + +| Opt-out present | TCF present | TCF consents | Result | +| --------------- | ----------- | ------------ | -------------------------------------------------- | +| yes | — | — | **unset** (and withdrawal semantics apply, §4.2) | +| no | yes | no | unset (withdrawal applies only where §4.2 says so) | +| no | yes | yes | set, unless B = denied | +| no | no | — | set iff B = granted | + +### 4.2 Withdrawal vs. absence + +Two distinct outcomes, never conflated: + +- **Withdrawal** (destructive: expire the EC cookie, write revocation + tombstones) requires an **affirmative** signal: an opt-out signal, or a + TCF record refusing `store-on-device` **in a jurisdiction whose baseline is + opt-in** (`requires_signal`). A visitor who has simply not yet made a + choice is never stripped of an existing identity. +- In a jurisdiction whose baseline is `granted`, a TCF refusal prevents + _new_ grants but does not tombstone: tombstones are irreversible + revocation markers, and PR #838 wrote them for visitors in unregulated + jurisdictions whose global CMP emitted a purpose-refusing string — + permanent identity loss under a regime the deployment never opted into. +- Withdrawal checking follows the same precedence as §4: an opt-out signal + triggers withdrawal even when a consenting TCF record is present. + +`ec_storage_withdrawn` (or its successor) gets direct unit coverage for every +row above; in PR #838 the headline "withdrawal expires identity" behavior had +no unit test at all. + +## 5. Jurisdiction resolution + +1. A selected geo provider resolves country and optional region; rules match + `country/region` first, then `country`, case-insensitively. +2. **Provider selected, lookup fails for a request** → the configured + `[geo] default_country` rules apply (per #779). +3. **No geo provider selected** → every request resolves to + `default_country`. This turns jurisdiction into a static constant, which + is only honest when the operator can genuinely assert single-jurisdiction + traffic. Constraint: **startup fails** when no geo provider is selected + _and_ the default country's baseline resolves any permission to `granted`, + unless the operator sets an explicit acknowledgment + (`[geo] assume_single_jurisdiction = true`). Without this, the natural + migration config (`default_country = "US"`, geo unset) silently grants + `store-on-device` and EID transmission to every EU visitor — the + highest-severity finding of the PR #838 review. The startup log always + prints the effective baseline and whether geo is live. +4. `default_country` is required; startup fails without it (per #779). The + shipped example uses the most protective baseline. + +## 6. Failure-mode matrix — normative + +| Condition | Resolution behavior | +| ---------------------------------------------------- | ------------------------------------------------ | +| Geo lookup fails at request time (provider selected) | `default_country` baseline | +| No geo provider configured | `default_country` baseline, gated by §5.3 | +| Country resolved, no matching rule | `default_country` baseline | +| Region resolved, no region rule | Country rule | +| Malformed policy file | Build failure (§3.2) — unreachable at runtime | +| No `default_country` | Startup failure | +| Undecodable TCF/GPP string | Treated as absent; opt-out signals still honored | +| Signals contradict (opt-out + consent) | Opt-out wins (§4) | + +The overall posture is **fail-closed**: every ambiguous state resolves to the +configured baseline or more restrictive, and the one configuration that could +convert "no information" into "granted" (§5.3) requires an explicit operator +assertion to exist. + +## 7. Enforcement points + +Exactly three consumers in this epic, all reading the same resolved set: + +1. **Provider execution** (providers spec §5) — all three provider kinds. +2. **EC lifecycle** — creation requires `store-on-device`; withdrawal per + §4.2. +3. **Bidstream EIDs** — transmission requires `store-on-device` ∧ + `select-personalised-ads`. + +## 8. Testing strategy + +- **The decision matrix is the test plan.** Every row of §4.1 and §4.2 × + each baseline, plus every row of §6, as table-driven tests. The ~24-case + matrix deleted by PR #838 (net −18 tests in the consent module, replaced + by happy-path cases only) is restored in equivalent form against the new + API; signal-precedence conflicts (opt-out + consenting TCF) are mandatory + cases, not optional ones. +- Policy validation tests for every §3.2 rejection. +- Shipped-table coverage test (§3.4) and split-brain consistency test + (§3.3). +- One end-to-end integration scenario per posture: opt-in jurisdiction with + and without consent, opt-out jurisdiction with GPC (including GPC + a + consenting TCF string), and the no-geo/default-country path. + +## 9. Out of scope + +- Additional purposes (extension procedure in §2). +- Runtime-loadable policy (the embedded file is deliberate: policy changes + are code reviews). If runtime policy is wanted later, it is its own spec + with its own validation story. diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md new file mode 100644 index 000000000..04663b58c --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -0,0 +1,239 @@ +# Design Spec: Pluggable Edge Cookie, Device, and Geo Providers + +**Status:** Draft +**Author:** Engineering +**Issue references:** #777, #778, #780, #781 +**Related specs:** `2026-07-30-permission-model-design.md`, +`2026-07-30-provider-migration-rollout-design.md`, +`2026-07-30-client-cycle-ec-resolve-design.md` +**Last updated:** 2026-07-30 + +> **Context.** PR #838 proposed a first implementation of this epic in a single +> change. Review of that PR surfaced design gaps this spec exists to close +> before a second implementation pass: an identity abstraction that owned +> minting but not recognition, per-adapter divergence in provider selection, +> silent misconfiguration modes, and speculative trait surface with no +> production caller. This spec is the authoritative statement of what the +> provider architecture must do; where it contradicts PR #838, this spec wins. + +--- + +## 1. Overview and goals + +Trusted Server makes three per-request data decisions that are currently +hard-wired: whether to create or keep an Edge Cookie (EC) identity, how to +classify the requesting device, and whether to resolve geolocation. Each +becomes a **provider**: a selectable component chosen in operator +configuration, with a deliberately neutral default. + +Goals: + +- A deployment picks an implementation per concern (including none) without a + code change to Trusted Server core. +- Defaults are neutral: with no configuration, no EC is created, device + classification uses only the User-Agent, and no geolocation is performed. A + default deployment makes no third-party or host-specific call. +- A provider **declares** the permissions its data use requires (see the + permission model spec); **core enforces** that declaration. A provider + cannot authorize itself. +- All adapters (Fastly, Axum, Cloudflare, Spin) behave identically for + identical configuration, or fail loudly at startup where a host cannot + satisfy the selected provider. + +Non-goals: + +- No vendor provider ships in this epic beyond the host-platform + implementations named below. +- The client-cycle (browser round-trip) provider type is **out of scope** + here; it has its own spec and must clear that spec's requirements first. + +## 2. Provider taxonomy + +| Concern | Trait | Built-in default | Opt-in host implementation | +| ----------- | -------------------- | --------------------------- | ----------------------------------------------------------------- | +| EC identity | `EdgeCookieProvider` | none (stateless) | `hmac` (in core; HMAC over client IP, preserves today's identity) | +| Device | `DeviceProvider` | `builtin` (User-Agent only) | `fastly` (JA4 / HTTP-2 fingerprints) | +| Geo | `GeoProvider` | none (no location) | `platform` (host geo lookup) | + +Selection keys are strings in operator configuration: + +```toml +[ec] +provider = "hmac" + +[ec.providers.hmac] +passphrase = "example-passphrase" + +[device] +provider = "builtin" + +[geo] +provider = "platform" +``` + +## 3. The identity lifecycle contract + +This is the section PR #838 lacked, and the source of its most structural +defect: the trait abstracted **minting** an identifier but left +**recognition** (`is_valid_ec_id`), **hashing** (`ec_hash`), and **KV key +normalization** hard-coded to the built-in HMAC shape. Any provider whose +identifiers do not match `{64hex}.{6alnum}` minted cookies that the very next +request discarded, and whose identities could never be tombstoned on +withdrawal. + +An `EdgeCookieProvider` owns the **complete lifecycle** of the identifiers it +mints. Every lifecycle operation core performs on an EC value MUST be routed +through the selected provider: + +| Lifecycle operation | Where core uses it today | Contract | +| -------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Mint** | EC generation on first eligible request | Provider returns the identifier (and only core writes the cookie). | +| **Recognize** | Reading `ts-ec` back from the request; deciding `ec_was_present` | Provider validates that a returned cookie value is one of its identifiers. A value the selected provider does not recognize is treated as absent. | +| **Hash / normalize** | KV identity-graph keys, log redaction | Provider (or a provider-supplied codec) maps an identifier to its stable KV key form. | +| **Tombstone** | Withdrawal: expiring the cookie and writing revocation markers | The identifiers eligible for tombstoning are exactly those the provider recognizes — never a shape-gated subset. | + +**Invariant:** for every provider `P` and every identifier `id` minted by `P`, +`P.recognize(id)` is true, `P` produces a stable KV key for `id`, and a +withdrawal request carrying `id` tombstones it. A conformance test suite MUST +assert this round-trip for every shipped provider, and the suite MUST be +written so a future provider crate can run it against its own implementation. + +## 4. Trait surface: minimalism rule + +Every trait method MUST have at least one production (non-test) caller in the +same PR that introduces it. Speculative surface observed in PR #838 that MUST +NOT ship without a caller: + +- `keys_equal` (no production caller; existed to serve a unit test), +- `GeneratedEdgeCookie::response_headers` (empty in all built-ins, plumbed + through three layers), +- `IdentityInput.permissions` / `IdentityInput.consent` (ignored by all + built-ins), +- `DeviceProvider::required_permissions` / `GeoProvider::required_permissions` + **unless** the enforcement point of §5 lands in the same change. + +If a future feature needs one of these, it arrives with that feature. + +The minimal `EdgeCookieProvider` surface implied by §3 is: + +```rust +pub trait EdgeCookieProvider { + /// Stable configuration key ("hmac"). + fn id(&self) -> &'static str; + /// Permissions this provider's data use requires. Enforced by core. + fn required_permissions(&self) -> PermissionSet; + /// Mint an identifier from request evidence. + fn generate(&self, input: &IdentityInput<'_>) -> Result>; + /// Whether `value` is an identifier this provider minted. + fn recognize(&self, value: &str) -> bool; + /// Stable KV key form of a recognized identifier. + fn kv_key(&self, id: &EcId) -> KvKey; +} +``` + +(Names indicative; the shape is normative.) + +## 5. Permission enforcement is core's job — for all three provider kinds + +Before executing **any** provider (EC, device, or geo), core resolves the +request's permission set (see the permission model spec) and refuses to run a +provider whose `required_permissions()` are not all set. PR #838 declared this +method on all three traits but consulted it only for the EC provider; the +device and geo declarations were decorative. That is worse than absent — it +reads as a gate and is not one. The enforcement point MUST be a single shared +code path used by all three provider kinds, with a test per kind proving a +provider declaring an unset permission does not execute. + +## 6. Selection, validation, and failure modes + +All validation happens at **settings construction** — a misconfiguration is a +startup error, never a request-time error and never a silent behavior change. + +| Configuration state | Behavior | +| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `provider` names an unknown key | Startup error listing valid keys. | +| `provider` set, its `[ec.providers.]` block missing | Startup error. | +| `[ec.providers.]` block present, `provider` unset | **Startup error.** (In PR #838 this silently ran stateless — the half-migrated config becomes a production identity outage detected by revenue drop. Rejecting it is the fix.) An operator who genuinely wants stateless deletes the block. | +| `provider` set to an implementation the running adapter cannot satisfy (e.g. a provider requiring host TLS fingerprints on an adapter that has none) | Startup error at adapter wiring time. Adapters declare their host capabilities to the composition root; the root checks the selected provider's needs against them **once**, at startup — not per request. | +| No `provider`, no providers block | Valid: the neutral default for that concern. | + +Unknown fields inside every provider config block are rejected +(`deny_unknown_fields` on all new settings structs — PR #838 applied it to +`Ec` but not to `EcProviders`, `DeviceConfig`, or `GeoConfig`, so a typo like +`providr` was silently ignored). + +## 7. Composition root and adapter parity + +Provider construction happens in exactly one place per concern +(`build_ec_provider`, `build_device_provider`, `build_geo_provider`), called +by **every** adapter. No adapter may wire a concrete implementation directly: +in PR #838 the Cloudflare adapter installed its host geo unconditionally, +so identical configuration produced different jurisdictions on different +adapters — which the permission model then turned into different privacy +outcomes. + +Requirements: + +- Each adapter's runtime-services setup routes through the shared builders. +- Providers are constructed **once** per application instance and stored in + app state; PR #838 rebuilt the provider (cloning the secret into a fresh + `Box`) up to three times per request. +- The cross-adapter parity suite gains cases asserting: (a) the selected + provider is honored on every adapter, (b) the neutral default performs no + host call on every adapter, and (c) a capability-unsatisfiable selection + fails startup on the adapters that cannot satisfy it. + +## 8. Crate layout and CI + +Provider crates live flat under `crates/` following the existing naming +convention: `crates/trusted-server-geo-fastly`, +`crates/trusted-server-device-fastly`. (PR #838 introduced a nested +`crates/geo/fastly` layout that broke the directory–package correspondence +every other member follows.) No placeholder directories: a `crates/…/README.md` +with no crate ships when the first crate does. + +Every new crate is added to the `.cargo/config.toml` aliases +(`check-fastly`, `clippy-fastly`, `test-fastly`, `build-fastly`) in the same +PR that adds the crate, and to the CI gate list in `CLAUDE.md`. PR #838's new +crates compiled only transitively and were never linted with `-D warnings` +nor had a single test. + +## 9. Behavior preservation notes + +Two defaults chosen for neutrality change effective behavior on existing +Fastly deployments; both are called out in the migration spec and must be +prominent in release notes: + +- **Bot gate.** The pre-provider EC bot gate required JA4 _and_ platform + class. With `device.provider = "builtin"` the gate degrades to User-Agent + heuristics. Restoring the stronger gate requires `[device] provider = +"fastly"`; the migration guide lists this as a behavior-preserving step for + Fastly deployments. +- **Geo.** With no geo provider, jurisdiction resolution falls to the + configured default country. The permission model spec (§5) constrains this + combination so it cannot silently grant permissions to mis-attributed + traffic. + +## 10. Testing strategy + +- Provider conformance suite (§3 invariant) run against every shipped + provider. +- Enforcement tests per provider kind (§5). +- Settings validation tests for every row of the §6 table, including the + block-without-selector rejection. +- Parity suite additions of §7. +- Unit tests inside each provider crate; crates with no native-target tests + still get clippy coverage via the alias wiring of §8. + +## 11. Implementation order + +1. Traits + lifecycle contract + conformance suite, `hmac` provider + passing it (behavior-identical to today; see migration spec §3 for the + ID-stability vectors). +2. Settings selection + validation table. +3. Composition root + all four adapters wired through it, parity cases. +4. Device and geo providers with the shared enforcement point of §5. + +Each step is independently reviewable; none depends on the permission model +landing first (the EC gate keeps its current jurisdiction logic until the +permission model PR replaces it). diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md new file mode 100644 index 000000000..c00f8c348 --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -0,0 +1,161 @@ +# Design Spec: Provider and Permission Model — Migration and Rollout + +**Status:** Draft +**Author:** Engineering +**Issue references:** #777–#781 (epic) +**Related specs:** `2026-07-30-pluggable-providers-design.md`, +`2026-07-30-permission-model-design.md` +**Last updated:** 2026-07-30 + +> **Context.** The provider/permission epic is a breaking change to a live +> identity system. PR #838's review showed that the riskiest part of such a +> change is not the new code but the transition: silent misconfiguration +> modes, undeclared behavior changes discovered by deleted tests, and no +> written statement of which pre-change behaviors were guaranteed to +> survive. This spec is that statement. Any implementation PR in the epic +> must reconcile its diff against §2's matrix and list every deliberate +> divergence in its description. + +--- + +## 1. Scope + +Covers the transition of existing deployments from the hard-wired EC / +device / geo behavior to the provider architecture and permission model. +Applies to every implementation PR in the epic, and to the operator-facing +migration guide that ships with the last of them. + +## 2. Behavior-preservation matrix + +For each decision the system makes today, the target behavior after the epic, +and whether that is a preservation or a declared change. **Silent changes are +defects.** PR #838 changed six of these without declaring any; each was +discoverable only because a deleted test had pinned the old behavior. + +| # | Decision (today) | After epic | Status | +| --- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Policy decision: the shipped `US` baseline decides. If the baseline grants `store-on-device` without a signal, that is a **declared change** requiring sign-off in the policy file review, with rationale in the file itself | Declared change (if made) | +| 4 | UK request, no TCF record → no EC | Same, unless the policy file deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | Unknown jurisdiction (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC allowed, never tombstoned | Same (withdrawal only where baseline is opt-in; permission spec §4.2) | Preserved | +| 7 | EID transmission requires storage + personalization consent where regulated | Same via `store-on-device` ∧ `select-personalised-ads` | Preserved | +| 8 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | +| 9 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"` | Declared change with a documented restore step (§5) | + +Rows 3 and 4 are policy decisions, not code decisions: they belong in the +`permissions.yaml` review, made explicitly by maintainers — not implied by an +implementation. + +## 3. Identity stability guarantee + +For a deployment that selects `provider = "hmac"` and carries its passphrase +over verbatim: + +- The minted identifier is **bit-identical** to today's: + `HMAC-SHA256(passphrase, normalized_ip)` in the existing encoding. +- Cookie name, attributes, and max-age are unchanged; existing `ts-ec` + cookies are recognized by the provider. +- KV identity-graph keys (`ec_hash`, normalization) are unchanged; no + existing graph row is orphaned. + +Enforced by **pinned known-answer tests**: fixed passphrase + IP → exact +expected identifier, cookie string, and KV key, committed as vectors so any +divergence fails CI rather than rotating a production identity base. + +## 4. Configuration migration + +Old shape: + +```toml +[ec] +passphrase = "example-passphrase" +``` + +New shape: + +```toml +[ec] +provider = "hmac" + +[ec.providers.hmac] +passphrase = "example-passphrase" +``` + +Requirements: + +1. **Old key fails loud.** `[ec] passphrase` is rejected at startup with a + message naming the new location — not a generic unknown-field error. +2. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no + `provider = "hmac"` selector is a startup error (providers spec §6). In + PR #838 this configuration — the exact state an operator following the + docs reaches if they miss one line — validated green and silently minted + zero ECs. +3. **The example config ships the migrated happy path**, uncommented: + `provider = "hmac"` with its block, `[geo] default_country`, and (for + Fastly) the behavior-preserving `[device] provider = "fastly"` and + `[geo] provider = "platform"` lines present with a comment stating what + removing them changes. PR #838's example shipped the passphrase block + uncommented with the selector commented out — steering operators directly + into the silent-stateless state. +4. Every misconfiguration in the providers spec §6 table fails at + **startup**. Request-time failure for a configuration error is a defect. +5. Config-store payload validation (`ts config push`) applies the same + rules, so a bad config is rejected at push time, before any instance + restarts into it. + +## 5. Behavior-preserving migration recipe (operator-facing) + +The migration guide (a new `docs/guide/` page, linked from the release notes) +gives one copy-pasteable recipe per adapter for "keep exactly today's +behavior": + +```toml +[ec] +provider = "hmac" +[ec.providers.hmac] +passphrase = "" + +[device] +provider = "fastly" # Fastly deployments: preserves the JA4 bot gate + +[geo] +provider = "platform" # preserves per-request jurisdiction detection +default_country = "FR" # used only when the host lookup fails +``` + +and separately documents the neutral configuration and what it does _not_ do. +The guide states explicitly that `default_country` alone does not replace geo +lookup, and why the permissive-default + no-geo combination requires the +explicit acknowledgment flag (permission spec §5.3). + +## 6. Rollout sequence and observability + +1. Implementation PRs land in the epic's order (providers first, permission + model second); each is reviewable against §2 in isolation. +2. Before/after deploy, operators watch **EC issuance rate** and EID + attachment rate; the migration guide names these as the canary metrics, + because the failure mode of a bad migration is a silent drop to zero (or a + silent grant to everyone), not an error rate. +3. Startup logs always print: selected provider per concern, whether geo is + live, the effective default baseline, and the count of granted-without- + signal permissions. One line, greppable, stable format. +4. Rollback is config-only where possible: reverting to the previous + config version restores the previous behavior on the previous binary. The + one irreversible artifact is withdrawal tombstones — which is why §2 row 6 + (no tombstones without affirmative withdrawal in an opt-in jurisdiction) + is non-negotiable. + +## 7. Documentation deliverables + +- Migration guide page (§5), linked from `CHANGELOG.md` and the release + notes. +- `configuration.md` documents **every** valid `provider` value for all + three concerns, and documents environment-variable overrides only if they + actually work in production builds (in PR #838 the documented + `TRUSTED_SERVER__EC__PROVIDER` override existed only under `#[cfg(test)]`). +- The permission model page states the §4 precedence rules of the permission + spec verbatim — operator docs and normative spec must not diverge on + precedence, and prose like "signals are mapped as a grant or a revoke" + without stating which wins is insufficient. From a35f2ca78759a77f89a43fd4b61806b57c2e2209 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 31 Jul 2026 01:00:07 -0700 Subject: [PATCH 02/24] Address self-review findings and move policy into trusted-server.toml Self-review of the five specs (own pass plus an adversarial fresh-eyes pass) surfaced fixes applied here: - Policy location reversed per maintainer decision: the permission policy is a [permissions] section of trusted-server.toml flowing through the config-store pipeline, not a build-time-embedded YAML. Overrides name acquisition rules directly instead of +/- sigils, and a rules.default entry separates resolved-but-unlisted countries from geo default_country. - Removed a circular requirement: geo and device providers are inputs to permission resolution and cannot be gated on its output; the enforcement gate is EC-only and the decorative required_permissions declarations are dropped from those traits. - Fixed the identity-stability guarantee: the EC id has a random per-mint suffix, so known-answer vectors pin the deterministic 64-hex prefix, recognition of existing cookies, and hash-prefix semantics instead of full identifiers. - Split the KV key contract into the verbatim graph-row key and the deliberately-colliding hash prefix that IP-cluster trust depends on. - Declared previously silent behavior changes in the migration matrix: global opt-out honoring (including tombstones) and the fate of non-regulated countries, with a preserving recipe for the latter. - Sequenced the geo neutral-default flip into the permission model PR so no intermediate step zeroes EC issuance under the current fail-closed gate. - Resolved smaller contradictions: withdrawal triggers made exhaustive (including denied-baseline and policy-edit cases), jurisdiction consistency requirement now covers both legacy lists with explicit exceptions, ISO rule-key validation made decidable, response-header reserved surface defined at cookie-name granularity for Set-Cookie, and the host-signals provider's removal made explicit with config rejection. --- ...26-07-30-client-cycle-ec-resolve-design.md | 22 +- ...integration-response-header-hook-design.md | 40 +- .../2026-07-30-permission-model-design.md | 384 ++++++++++++------ .../2026-07-30-pluggable-providers-design.md | 111 +++-- ...07-30-provider-migration-rollout-design.md | 121 ++++-- 5 files changed, 451 insertions(+), 227 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index 674de01ff..1b9eacc12 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -6,7 +6,7 @@ until its open questions (§7) are resolved in a dedicated issue** **Issue references:** none yet (this spec exists to force one; #778 does not cover this feature) **Related specs:** `2026-07-30-pluggable-providers-design.md` -**Last updated:** 2026-07-30 +**Last updated:** 2026-07-31 > **Context.** PR #838 shipped, undeclared and unspec'd, a second provider > _type_: a "client-cycle" EC provider whose identifier is established by a @@ -52,9 +52,10 @@ Everything in this spec follows from that. 1. **Reject cross-site requests.** Require a same-site assertion: `Origin` (or `Sec-Fetch-Site: same-origin/same-site`) validated against the - publisher's origin set; requests without a validating header are - rejected. CSRF-token designs are acceptable but not required if - origin-based rejection is enforced. + publisher's origin allowlist — configuration that does not exist yet and + must be defined by this feature (§7, question 5); requests without a + validating header are rejected. CSRF-token designs are acceptable but not + required if origin-based rejection is enforced. 2. **Verify the payload cryptographically per provider.** The provider's `resolve_from_client` accepts only payloads that are signed by an expected party, **audience-bound** to this publisher, and **expiring** @@ -83,7 +84,11 @@ Everything in this spec follows from that. endpoint is cheap-idempotent and rate-limited per session; the design must state which and test it. - The JS module ships through the standard integration bundle mechanism, - loaded only when a client-cycle provider is the selected EC provider. + loaded only when a client-cycle provider is the selected EC provider. Note + the consequence: bundle content becomes a function of EC configuration, + which interacts with the bundle's content-hash/SRI pinning and caching — + the mechanism today keys off the integration registry, not EC provider + selection (open question, §7). - Any constant shared between Rust and TS (endpoint path, marker name) is asserted equal by a test, not "kept in sync by hand". @@ -118,3 +123,10 @@ sentence as the only guardrail. 3. Rate limiting / abuse posture at the edge for an unauthenticated POST. 4. Whether the endpoint should be versioned separately from the identify API family it sits beside. +5. The shape of the publisher **origin allowlist** that §3.1 validates + against — no such configuration exists today (`publisher.origin_url` is + a single upstream and the cookie domain is a cookie scope, not an origin + set), so it is new settings surface this feature must define. +6. How JS module selection keyed off EC provider configuration coexists + with content-hashed/SRI-pinned bundles (§4) — per-config hashes, cache + keying, and the config-push story for them. diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 59d354547..01aea582d 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -4,7 +4,7 @@ **Author:** Engineering **Issue references:** #782 **Related specs:** `2026-07-30-pluggable-providers-design.md` -**Last updated:** 2026-07-30 +**Last updated:** 2026-07-31 > **Context.** Issue #782 already specifies this feature well; its done-when > is the contract. PR #838 shipped the trait and registry wiring with **no @@ -34,19 +34,28 @@ mutators to the outbound response for HTML document responses it processed. code where one exists; where adapters finalize independently, each adapter gains the call and a test proving it. - Mutators run **after** Trusted Server's own response-header handling - (EC Set-Cookie emission, EC header clearing, privacy headers) so a mutator - cannot be silently clobbered by later core steps — in PR #838's ordering, - provider-supplied headers were inserted before the EC header-clearing pass - and could be stripped by it. + (EC Set-Cookie emission, EC header clearing, privacy headers) so a + mutation cannot be silently stripped by a later core pass. The ordering is + a fresh decision this spec makes — PR #838 never wired the hook, so there + is no existing insertion point to inherit; the implementer places the call + at the end of each adapter's response finalization, and the §4.3 tests pin + it there. ## 3. Collision policy -- Mutators may not touch **reserved headers**: `Set-Cookie` for the EC - cookie, the `x-ts-*` namespace, and the consent/privacy headers core - emits. Attempts are dropped and logged at `warn` with the integration id. +- Mutators may not touch **reserved surface**, which is defined at two + granularities because `Set-Cookie` is multi-valued: (a) reserved header + _names_ — the `x-ts-*` namespace and the consent/privacy headers core + emits; (b) reserved cookie _names_ within `Set-Cookie` — `ts-ec`, + `ts-eids`, and the other `ts-*` cookies core owns. An integration may + append its own `Set-Cookie` values; it may not set or expire a reserved + cookie name. Violations are dropped and logged at `warn` with the + integration id. The reserved-cookie list is a single constant next to the + cookie definitions, not duplicated in the hook. - For non-reserved headers, the mutator API distinguishes **append** from - **replace** explicitly; the default is append. Replacing a header the - origin set is a deliberate act, visible in the mutator's code. + **replace** explicitly; the default is append (for `Set-Cookie`, append is + the only non-reserved operation — replace is not offered). Replacing a + header the origin set is a deliberate act, visible in the mutator's code. - Later registrations see earlier mutations (order = registration order, which is deterministic). @@ -62,8 +71,11 @@ mutators to the outbound response for HTML document responses it processed. 4. A parity-suite case asserts identical mutation behavior across adapters. 5. Reserved-header and append/replace semantics covered by unit tests. -## 5. Size +## 5. Size and sequencing -This is a ~150-line feature plus tests. It has zero coupling to the provider -architecture or the permission model and should land as its own small PR, -first in the epic's sequence. +This is a ~150-line feature plus tests, with zero coupling to the provider +architecture or the permission model. It lands as its own small PR **when +its first real consumer is identified** (§4.2) — at any point in the epic's +sequence, blocking nothing and blocked by nothing. If no consumer +materializes, it does not land; being unblocked is not a reason to ship +scaffolding. diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index f3504a04d..bd5da97c9 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -5,17 +5,19 @@ **Issue references:** #779 **Related specs:** `2026-07-30-pluggable-providers-design.md`, `2026-07-30-provider-migration-rollout-design.md` -**Last updated:** 2026-07-30 +**Last updated:** 2026-07-31 > **Context.** PR #838 proposed a permission model whose review surfaced two > classes of defect this spec exists to prevent in the next pass: (1) silent > behavioral inversions of consent-signal precedence — most seriously, a > present TCF string short-circuiting GPC/GPP/US-Privacy opt-outs — and > (2) fail-open jurisdiction resolution when geolocation is disabled. The -> precedence table (§4) and the failure-mode matrix (§6) are the two +> precedence rules (§4) and the failure-mode matrix (§6) are the two > documents whose absence allowed those defects to hide in a 67-file diff. > They are normative: an implementation whose behavior differs from these -> tables is wrong, whatever its tests say. +> tables is wrong, whatever its tests say. This spec also reverses one +> PR #838 structural decision: policy lives in `trusted-server.toml`, not in +> a build-time-embedded YAML file (§3.1). --- @@ -24,14 +26,14 @@ The permission model replaces the hard-wired jurisdiction gate (`allows_ec_creation` and its companions) with a single resolved **permission set** per request. Every data decision Trusted Server itself -makes — EC creation, EC withdrawal, EID transmission into the bidstream, -and provider execution (see providers spec §5) — reads that set. +makes — EC provider execution, EC creation and withdrawal, and EID +transmission into the bidstream — reads that set (§7). The set is resolved from three inputs: 1. **Jurisdiction** — the country/region the request resolves to (§5). -2. **Policy** — a declarative, version-controlled map from jurisdiction to a - baseline acquisition rule per permission (§3). +2. **Policy** — a declarative map from jurisdiction to a baseline + acquisition rule per permission (§3). 3. **Signals** — the request's privacy signals: TCF, GPP, GPC, US Privacy (§4). @@ -62,72 +64,138 @@ The initial vocabulary is therefore exactly: (The identifier strings are the IAB names verbatim, including their original spelling.) The extension procedure — add the signal mapping, add the enforcement point, add the policy vocabulary entry, in one change — is -documented in the policy file header. Policy validation **rejects** a rule -that references an identifier outside the current vocabulary, so the file can +documented alongside the policy schema. Policy validation **rejects** a rule +that references an identifier outside the current vocabulary, so a policy can never promise more than the code enforces. ## 3. Policy -### 3.1 Format +### 3.1 Location: `[permissions]` in `trusted-server.toml` -A YAML map, embedded at build time, of named **groups** (baselines) and -**rules** keying countries (`FR`) and country/state pairs (`US/CA`) to a -group with optional per-permission overrides. Each permission resolves to an -**acquisition rule**: +Policy is operator-owned runtime configuration, expressed as a +`[permissions]` section of `trusted-server.toml`, flowing through the same +pipeline as every other setting (`ts config push` publishes it as part of the +config blob envelope; instances pick it up like any config change). -- `granted` — set without any signal, -- `requires_signal` — set only when a signal grants it (opt-in), -- `denied` — never set, even when a signal grants it. - -Overrides support all three targets: `+perm` (granted), `-perm` (denied), and -`~perm` (requires_signal). PR #838 supported only `+`/`-`, making the most -common real-world override — "this state requires a signal for personalized -ads" — inexpressible without duplicating a whole group. Groups may use a -`default:` shorthand for unlisted permissions. +This deliberately reverses PR #838, which embedded a `permissions.yaml` at +build time via `include_str!`. That design was rejected because: -### 3.2 Validation — at build time, not request time +- a policy edit — the operation the whole model exists to make easy — + required recompiling and redeploying the binary, cutting against the + runtime config-store pipeline the project has standardized on; +- it introduced a second configuration language and a second validation + path next to the TOML settings machinery that already exists; +- the `include_str!` reached two directory levels above the crate root, + breaking crate packaging; +- validation ran lazily at first use behind a `OnceLock` + `expect`, so a + bad edit that escaped unit tests became a 500 on every request. -The embedded file is validated by a `build.rs` step (or an equivalent -always-run CI test that asserts the parse explicitly): a malformed committed -file fails the **build**, never a request. PR #838's file was parsed lazily -behind a `OnceLock` with an `expect`, meaning a bad edit that escaped unit -tests became a 500 on every request. +Auditability is preserved where it actually lives: the source-controlled +`trusted-server.example.toml` ships the complete recommended policy table +(the reviewable reference artifact), and the operator's own config history — +git for the file, config-store versions for pushes — is the change log. -Validation rejects: +**Compiled-in fallback:** when a config has no `[permissions]` section, a +minimal compiled-in policy applies in which **every permission is +`requires_signal` for every jurisdiction** — the most protective posture. +Absence of policy is always safe; there is no fail-open default. -- unknown fields anywhere (`deny_unknown_fields` on every deserialized - struct — PR #838's untagged rule enum silently swallowed a misspelled - `permission:` key, dropping the operator's override with no diagnostic); -- rule keys that are not plausible ISO 3166-1 alpha-2 / ISO 3166-2 codes; -- references to permissions outside the enforced vocabulary (§2); -- groups that neither list every permission nor provide `default:`. +### 3.2 Format -### 3.3 One source of jurisdiction truth +Named **groups** (baselines) and **rules** mapping a country (`FR`) or +country/state pair (`"US/CA"`) to a group, with optional per-permission +overrides. Each permission resolves to an **acquisition rule**: -The codebase currently carries a second, runtime-configurable jurisdiction -list (`consent.gdpr.applies_in`) used by the auction consent gate. Two -independently maintained country tables that both express "where GDPR -applies" will drift (in PR #838, adding `CH` to one had no effect on the -other). Requirement: either the auction gate derives its jurisdiction class -from the same resolved policy, or a CI test asserts that every country in -`applies_in` resolves to an opt-in (`requires_signal`) baseline in the policy -file, and vice versa for the shipped defaults. +- `granted` — set without any signal, +- `requires_signal` — set only when a signal grants it (opt-in), +- `denied` — never set, even when a signal grants it. -### 3.4 Shipped table coverage +```toml +[permissions.groups.gdpr-eu] +default = "requires_signal" + +[permissions.groups.us-opt-out] +default = "granted" + +[permissions.rules] +FR = "gdpr-eu" +US = "us-opt-out" +# Overrides name explicit acquisition rules — no +/- sigil syntax; TOML +# expresses the target state directly. +"US/CA" = { group = "us-opt-out", overrides = { select-personalised-ads = "requires_signal" } } +# Reserved key: countries that resolve but match no rule. Distinct from +# [geo] default_country, which handles requests that resolve no country at +# all (§5.4). +default = "gdpr-eu" +``` + +A group's `default` covers unlisted permissions; a group may also name +permissions explicitly. Overrides map identifier → acquisition rule, so any +target state (including `requires_signal`) is expressible — PR #838's +`+`/`-` sigil scheme could not express "requires a signal", the most common +real-world override. + +### 3.3 Validation — at config acceptance, not request time + +Policy is validated where every other setting is: at `ts config push` (a bad +policy is rejected before publication) and at settings construction on +startup (a bad stored config produces the startup-error state, never a +per-request failure). -A CI test asserts every member of the GDPR country list resolves to an opt-in -baseline (this is what catches a `DK:` typoed as `DL:` — a defect that in -PR #838 survived parse, startup, and all tests, silently dropping Denmark to -the operator default). Countries intentionally not listed are governed by -§5's default-country rules; the policy header documents that this is the -fallback, and the shipped example default is the most protective baseline. +Validation rejects: -## 4. Signal precedence — normative table +- unknown fields anywhere (`deny_unknown_fields` on every deserialized + struct — PR #838's untagged rule type silently swallowed a misspelled + override key, dropping the operator's override with no diagnostic); +- rule keys whose country part is not in the embedded **assigned** ISO + 3166-1 alpha-2 list (not merely `[A-Z]{2}` — an unassigned code is + almost certainly a typo silently diverting a country to the fallback); + the region part matches `[A-Z0-9]{1,3}`. The `US/CA` slash form is the + house rule-key format corresponding to ISO 3166-2 `US-CA`; +- references to permissions outside the enforced vocabulary (§2); +- references to undefined groups; +- groups that neither list every permission nor provide `default`. + +### 3.4 One source of jurisdiction truth + +Today, `detect_jurisdiction` — driven by the runtime lists +`consent.gdpr.applies_in` and `consent.us_privacy.states` — is the sole +jurisdiction source for **both** the auction consent gate and the EC gate. +The permission model replaces the EC side; if the auction gate keeps reading +the old lists while EC reads policy rules, the two will drift (adding a +country to one has no effect on the other, and an operator has no signal +that they disagree). + +Requirement: the auction gate's jurisdiction class derives from the same +resolved policy (a country is GDPR-class when its rule resolves to an +opt-in baseline for `select-personalised-ads`). Where the legacy lists must +survive an interim period, a CI test asserts consistency between each list +and the policy table, with deliberate divergences recorded as explicit, +commented exceptions in the test — never silent. Both legacy lists are in +scope, not only the GDPR one. + +### 3.5 Shipped-table coverage + +A CI test asserts every member of the GDPR country list resolves to a +GDPR-class baseline in the example policy. This closes a defect class +nothing in PR #838's validation covered: a mistyped country key (`DL:` for +`DK:`) parses cleanly, starts cleanly, and silently drops a member state to +the fallback rule. Countries intentionally unlisted are governed by the +`rules.default` entry (§3.2); the example policy documents that fallback +inline, and ships it as the most protective baseline. + +## 4. Signal precedence — normative Signals are classified: - **Opt-out signals** (affirmative withdrawal): GPC header; GPP sections - carrying a sale/sharing opt-out; US Privacy opt-out. + carrying a sale/sharing opt-out; US Privacy opt-out. Opt-out signals are + honored **globally**, not only in the jurisdictions whose law defines + them — a deliberate, more-protective simplification: scoping a browser's + explicit opt-out to a geolocation guess would honor it for some visitors + and ignore it for others based on IP evidence. (For jurisdictions outside + US states this is a declared behavior change; migration spec §2 records + it.) - **Consent records**: a decodable TCF string (standalone or embedded in GPP), which may grant or refuse individual purposes. @@ -144,7 +212,13 @@ Signals are classified: `ec_blocked_us_state_gpc_overrides_tcf` and companions — are reinstated against the new API, not deleted.)_ 3. Consent record refusal — a TCF record present and refusing the purpose - revokes it. + revokes it. This applies in **every** jurisdiction, including + `granted`-baseline ones: an expressed refusal always beats a policy + default. Note this is a declared, more-protective divergence from the + pre-epic gate, which ignored consent records entirely outside regulated + jurisdictions — the migration spec's matrix (row 6) records it. Refusal + revokes new grants only; whether it also destroys existing identity is + governed strictly by §4.2. 4. Consent record grant — a TCF record present and consenting grants it (subject to 1–2). 5. No signal — the policy baseline decides: `granted` sets it, @@ -155,100 +229,156 @@ Signals are classified: For each enforced permission, with baseline _B_ ∈ {granted, requires_signal, denied}: -| Opt-out present | TCF present | TCF consents | Result | -| --------------- | ----------- | ------------ | -------------------------------------------------- | -| yes | — | — | **unset** (and withdrawal semantics apply, §4.2) | -| no | yes | no | unset (withdrawal applies only where §4.2 says so) | -| no | yes | yes | set, unless B = denied | -| no | no | — | set iff B = granted | +| Opt-out present | TCF present | TCF consents | Result | +| --------------- | ----------- | ------------ | ------------------------------------------------ | +| yes | — | — | **unset** (and withdrawal semantics apply, §4.2) | +| no | yes | no | unset (withdrawal per §4.2, trigger 2) | +| no | yes | yes | set, unless B = denied | +| no | no | — | set iff B = granted | ### 4.2 Withdrawal vs. absence -Two distinct outcomes, never conflated: - -- **Withdrawal** (destructive: expire the EC cookie, write revocation - tombstones) requires an **affirmative** signal: an opt-out signal, or a - TCF record refusing `store-on-device` **in a jurisdiction whose baseline is - opt-in** (`requires_signal`). A visitor who has simply not yet made a - choice is never stripped of an existing identity. -- In a jurisdiction whose baseline is `granted`, a TCF refusal prevents - _new_ grants but does not tombstone: tombstones are irreversible - revocation markers, and PR #838 wrote them for visitors in unregulated - jurisdictions whose global CMP emitted a purpose-refusing string — - permanent identity loss under a regime the deployment never opted into. -- Withdrawal checking follows the same precedence as §4: an opt-out signal - triggers withdrawal even when a consenting TCF record is present. - -`ec_storage_withdrawn` (or its successor) gets direct unit coverage for every -row above; in PR #838 the headline "withdrawal expires identity" behavior had -no unit test at all. +Withdrawal (destructive: expire the EC cookie, write revocation tombstones) +and non-grant (the permission is simply unset) are distinct outcomes, never +conflated. "Baseline" below always means the **resolved acquisition rule for +`store-on-device` in the request's jurisdiction, after overrides** — never a +group label, since a group can mix rules across permissions. + +The triggers, exhaustively — nothing else withdraws: + +1. **An opt-out signal withdraws in every jurisdiction, whatever the + baseline.** (For US states this preserves today's behavior; elsewhere it + is the declared change of §4's global-opt-out rule.) +2. **A TCF record refusing `store-on-device` withdraws iff the baseline is + `requires_signal`.** Where the baseline is `granted`, refusal blocks + _new_ grants but never tombstones: tombstones are irreversible, and + PR #838 wrote them for visitors in unregulated jurisdictions whose + global CMP emitted a purpose-refusing string — permanent identity loss + under a regime the deployment never opted into. +3. **A policy edit is not a user signal.** Tightening a baseline to + `denied` stops new identity but does not itself tombstone identities + minted before the change; cleaning those up is an operational action + (migration spec §6). An affirmative user signal (trigger 1 or 2) still + withdraws them. +4. **Absence of signal never destroys identity.** A visitor who has not yet + made a choice is never stripped of an existing identity. + +Withdrawal checking follows §4 precedence: an opt-out signal triggers +withdrawal even when a consenting TCF record is present. +`ec_storage_withdrawn` (or its successor) gets direct unit coverage for +every trigger above; in PR #838 the headline "withdrawal expires identity" +behavior had no unit test at all. ## 5. Jurisdiction resolution -1. A selected geo provider resolves country and optional region; rules match - `country/region` first, then `country`, case-insensitively. -2. **Provider selected, lookup fails for a request** → the configured - `[geo] default_country` rules apply (per #779). -3. **No geo provider selected** → every request resolves to - `default_country`. This turns jurisdiction into a static constant, which - is only honest when the operator can genuinely assert single-jurisdiction - traffic. Constraint: **startup fails** when no geo provider is selected - _and_ the default country's baseline resolves any permission to `granted`, - unless the operator sets an explicit acknowledgment - (`[geo] assume_single_jurisdiction = true`). Without this, the natural - migration config (`default_country = "US"`, geo unset) silently grants - `store-on-device` and EID transmission to every EU visitor — the - highest-severity finding of the PR #838 review. The startup log always - prints the effective baseline and whether geo is live. -4. `default_country` is required; startup fails without it (per #779). The - shipped example uses the most protective baseline. +### 5.1 Order + +Geo resolution runs **before** permission resolution — jurisdiction is an +input to the permission set, which is why geo providers cannot themselves be +gated on it (providers spec §5). A selected geo provider resolves country +and optional region; rules match `country/region` first, then `country`, +case-insensitively. + +### 5.2 Lookup failure + +Provider selected, lookup resolves nothing for a request → the configured +`[geo] default_country` rules apply (per #779). An adapter whose geo +implementation can never resolve anything must not accept the selection at +all — that is the capability check of providers spec §6, and it prevents a +"selected but always empty" provider from silently converting every request +to §5.3 semantics without §5.3's guard. + +### 5.3 No geo provider selected + +Every request resolves to `default_country` — jurisdiction becomes a static +constant, which is only honest when the operator can genuinely assert +single-jurisdiction traffic. It is not only `granted` baselines that make +this dangerous: with a `requires_signal` baseline, a page-global CMP that +emits a consenting TCF string grants permissions for every mis-attributed +visitor just as effectively. + +Constraint: **startup fails** when an EC provider is selected and no geo +provider is, unless the operator sets an explicit acknowledgment +(`[geo] assume_single_jurisdiction = true`). Stateless deployments (no EC +provider) are exempt. Without this guard, the natural migration config +(`default_country = "US"`, geo unset) silently grants `store-on-device` and +EID transmission to every EU visitor — the highest-severity finding of the +PR #838 review. The startup log always prints the effective baseline and +whether geo is live. + +### 5.4 Defaults, two distinct fallbacks + +`[geo] default_country` is required; startup fails without it (per #779). +It covers requests that resolve **no country at all**. Countries that +resolve but match no rule fall to the policy's `rules.default` entry +(§3.2). The two fallbacks are deliberately separate: "we could not place +this request" and "we placed it somewhere we have no rule for" are +different states, and pre-epic behavior treated them differently (fail +closed vs. non-regulated) — collapsing them is what made PR #838's +migration story unresolvable (migration spec §2, rows 5 and 7). ## 6. Failure-mode matrix — normative -| Condition | Resolution behavior | -| ---------------------------------------------------- | ------------------------------------------------ | -| Geo lookup fails at request time (provider selected) | `default_country` baseline | -| No geo provider configured | `default_country` baseline, gated by §5.3 | -| Country resolved, no matching rule | `default_country` baseline | -| Region resolved, no region rule | Country rule | -| Malformed policy file | Build failure (§3.2) — unreachable at runtime | -| No `default_country` | Startup failure | -| Undecodable TCF/GPP string | Treated as absent; opt-out signals still honored | -| Signals contradict (opt-out + consent) | Opt-out wins (§4) | - -The overall posture is **fail-closed**: every ambiguous state resolves to the -configured baseline or more restrictive, and the one configuration that could -convert "no information" into "granted" (§5.3) requires an explicit operator -assertion to exist. +| Condition | Resolution behavior | +| ---------------------------------------------------- | ------------------------------------------------------------ | +| Geo lookup fails at request time (provider selected) | `default_country` baseline | +| No geo provider configured | `default_country` baseline, guarded by §5.3 | +| Country resolved, no matching rule | Policy `rules.default` | +| Region resolved, no region rule | Country rule | +| No `[permissions]` section | Compiled-in fallback: everything `requires_signal` | +| Malformed policy | Rejected at config push / startup (§3.3) — never per request | +| No `default_country` | Startup failure | +| Undecodable TCF/GPP string | Treated as absent; opt-out signals still honored | +| Signals contradict (opt-out + consent) | Opt-out wins (§4) | + +The overall posture is **fail-closed**: every ambiguous state resolves to +the configured baseline or more restrictive, and the one configuration that +turns "no information" into a static jurisdiction assertion (§5.3) requires +an explicit operator acknowledgment to exist. ## 7. Enforcement points -Exactly three consumers in this epic, all reading the same resolved set: - -1. **Provider execution** (providers spec §5) — all three provider kinds. +Consumers of the resolved set in this epic: + +1. **EC provider execution** (providers spec §5) — the provider's declared + `required_permissions()` must all be set. This gate applies to EC + providers only: geo and device providers execute **before** permission + resolution as its inputs, so gating them on its output would be + circular. Their governance is explicit selection, the capability checks + of providers spec §6, and §2's vocabulary rule — if a future vocabulary + adds a purpose covering geolocation or fingerprinting, gating those + providers will require a two-phase resolution that must be specified + then, not improvised. 2. **EC lifecycle** — creation requires `store-on-device`; withdrawal per §4.2. 3. **Bidstream EIDs** — transmission requires `store-on-device` ∧ `select-personalised-ads`. +The client-cycle resolve endpoint (own spec, currently on hold) would be a +fourth consumer if and when it proceeds. + ## 8. Testing strategy -- **The decision matrix is the test plan.** Every row of §4.1 and §4.2 × - each baseline, plus every row of §6, as table-driven tests. The ~24-case - matrix deleted by PR #838 (net −18 tests in the consent module, replaced - by happy-path cases only) is restored in equivalent form against the new - API; signal-precedence conflicts (opt-out + consenting TCF) are mandatory - cases, not optional ones. -- Policy validation tests for every §3.2 rejection. -- Shipped-table coverage test (§3.4) and split-brain consistency test - (§3.3). +- **The decision matrix is the test plan.** Every row of §4.1 × each + baseline, every trigger of §4.2, and every row of §6, as table-driven + tests. The ~24-case matrix deleted by PR #838 (net −18 tests in the + consent module, replaced by happy-path cases only) is restored in + equivalent form against the new API; signal-precedence conflicts + (opt-out + consenting TCF) are mandatory cases, not optional ones. +- Policy validation tests for every §3.3 rejection, exercised through both + acceptance paths (push-time and startup). +- Shipped-table coverage test (§3.5) and jurisdiction-consistency test + (§3.4) covering both legacy lists. - One end-to-end integration scenario per posture: opt-in jurisdiction with and without consent, opt-out jurisdiction with GPC (including GPC + a - consenting TCF string), and the no-geo/default-country path. + consenting TCF string), the no-geo/default-country path, and the + no-policy compiled fallback. ## 9. Out of scope - Additional purposes (extension procedure in §2). -- Runtime-loadable policy (the embedded file is deliberate: policy changes - are code reviews). If runtime policy is wanted later, it is its own spec - with its own validation story. +- A build-time-embedded policy file (PR #838's approach) — rejected for the + reasons in §3.1, not deferred. +- Per-signal jurisdiction scoping (honoring GPC only where a law defines + it): rejected in favor of the global rule in §4; revisiting it is a + policy-model change requiring its own review. diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index 04663b58c..ea0ac765a 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -6,7 +6,7 @@ **Related specs:** `2026-07-30-permission-model-design.md`, `2026-07-30-provider-migration-rollout-design.md`, `2026-07-30-client-cycle-ec-resolve-design.md` -**Last updated:** 2026-07-30 +**Last updated:** 2026-07-31 > **Context.** PR #838 proposed a first implementation of this epic in a single > change. Review of that PR surfaced design gaps this spec exists to close @@ -33,9 +33,11 @@ Goals: - Defaults are neutral: with no configuration, no EC is created, device classification uses only the User-Agent, and no geolocation is performed. A default deployment makes no third-party or host-specific call. -- A provider **declares** the permissions its data use requires (see the +- An **EC provider declares** the permissions its data use requires (see the permission model spec); **core enforces** that declaration. A provider - cannot authorize itself. + cannot authorize itself. (Geo and device providers are governed + differently — they execute as _inputs_ to permission resolution and cannot + be gated on its output; see §5.) - All adapters (Fastly, Axum, Cloudflare, Spin) behave identically for identical configuration, or fail loudly at startup where a host cannot satisfy the selected provider. @@ -71,6 +73,14 @@ provider = "builtin" provider = "platform" ``` +**Deliberately not carried over from PR #838:** the `host-signals` EC +provider (identity from HMAC over JA4/HTTP-2 TLS fingerprints plus client +IP). Minting _identity_ from TLS fingerprints is a different privacy +proposition from device _classification_ (#780) and was specified by no +issue; if wanted, it returns with its own spec and its own vocabulary +discussion. A config selecting `provider = "host-signals"` is rejected at +startup like any unknown key (migration spec §4). + ## 3. The identity lifecycle contract This is the section PR #838 lacked, and the source of its most structural @@ -85,15 +95,17 @@ An `EdgeCookieProvider` owns the **complete lifecycle** of the identifiers it mints. Every lifecycle operation core performs on an EC value MUST be routed through the selected provider: -| Lifecycle operation | Where core uses it today | Contract | -| -------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Mint** | EC generation on first eligible request | Provider returns the identifier (and only core writes the cookie). | -| **Recognize** | Reading `ts-ec` back from the request; deciding `ec_was_present` | Provider validates that a returned cookie value is one of its identifiers. A value the selected provider does not recognize is treated as absent. | -| **Hash / normalize** | KV identity-graph keys, log redaction | Provider (or a provider-supplied codec) maps an identifier to its stable KV key form. | -| **Tombstone** | Withdrawal: expiring the cookie and writing revocation markers | The identifiers eligible for tombstoning are exactly those the provider recognizes — never a shape-gated subset. | +| Lifecycle operation | Where core uses it today | Contract | +| ------------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Mint** | EC generation on first eligible request | Provider returns the identifier (and only core writes the cookie). | +| **Recognize** | Reading `ts-ec` back from the request; deciding `ec_was_present` | Provider validates that a returned cookie value is one of its identifiers. A value the selected provider does not recognize is treated as absent. | +| **Key for the graph row** | KV identity-graph row reads/writes | The row key is the identifier **verbatim**; the provider guarantees its identifiers are stable and KV-safe. | +| **Hash prefix** | IP-cluster sizing (`cluster_trust_threshold` prefix listing), pull-sync dedupe, log redaction | Provider maps an identifier to its hash prefix. This prefix **deliberately collides** across identifiers minted from the same client evidence — the collision is load-bearing for cluster-trust counting, and a provider that returns a unique-per-identifier value silently breaks it. | +| **Tombstone** | Withdrawal: expiring the cookie and writing revocation markers | The identifiers eligible for tombstoning are exactly those the provider recognizes — never a shape-gated subset. | **Invariant:** for every provider `P` and every identifier `id` minted by `P`, -`P.recognize(id)` is true, `P` produces a stable KV key for `id`, and a +`P.recognize(id)` is true, `P` produces a stable hash prefix for `id` (and +two identifiers minted from the same client evidence share it), and a withdrawal request carrying `id` tombstones it. A conformance test suite MUST assert this round-trip for every shipped provider, and the suite MUST be written so a future provider crate can run it against its own implementation. @@ -110,7 +122,8 @@ NOT ship without a caller: - `IdentityInput.permissions` / `IdentityInput.consent` (ignored by all built-ins), - `DeviceProvider::required_permissions` / `GeoProvider::required_permissions` - **unless** the enforcement point of §5 lands in the same change. + — dropped entirely, not deferred: §5 explains why these two kinds cannot + be permission-gated at all. If a future feature needs one of these, it arrives with that feature. @@ -126,23 +139,35 @@ pub trait EdgeCookieProvider { fn generate(&self, input: &IdentityInput<'_>) -> Result>; /// Whether `value` is an identifier this provider minted. fn recognize(&self, value: &str) -> bool; - /// Stable KV key form of a recognized identifier. - fn kv_key(&self, id: &EcId) -> KvKey; + /// Hash prefix of a recognized identifier (see §3: collides by design + /// across identifiers minted from the same client evidence). + fn hash_prefix(&self, id: &EcId) -> HashPrefix; } ``` -(Names indicative; the shape is normative.) - -## 5. Permission enforcement is core's job — for all three provider kinds - -Before executing **any** provider (EC, device, or geo), core resolves the -request's permission set (see the permission model spec) and refuses to run a -provider whose `required_permissions()` are not all set. PR #838 declared this -method on all three traits but consulted it only for the EC provider; the -device and geo declarations were decorative. That is worse than absent — it -reads as a gate and is not one. The enforcement point MUST be a single shared -code path used by all three provider kinds, with a test per kind proving a -provider declaring an unset permission does not execute. +(Names indicative; the shape is normative. `required_permissions` joins the +trait at step 5 of §11, together with its enforcement point.) + +## 5. Permission enforcement is core's job — for EC providers + +Before executing an **EC provider**, core resolves the request's permission +set (see the permission model spec) and refuses to run a provider whose +`required_permissions()` are not all set, with a test proving a provider +declaring an unset permission does not execute. + +This gate applies to EC providers **only**, and the reason is structural, +not convenience: the permission set is resolved _from_ jurisdiction, which +is resolved _by_ the geo provider — gating geo (or device, which runs in +the same pre-resolution phase) on the resolved set would be circular. +PR #838 declared `required_permissions` on all three traits but consulted +it only for the EC provider; the geo and device declarations were +decorative — worse than absent, because they read as a gate and are not +one. This spec resolves that by **not having** the method on those traits +(§4). Geo and device providers are governed by explicit operator selection, +the capability checks of §6, and the permission model's vocabulary rule: if +a future vocabulary adds a purpose covering geolocation or fingerprinting, +gating those providers will require a two-phase resolution design specified +at that time (permission model spec §7). ## 6. Selection, validation, and failure modes @@ -158,9 +183,9 @@ startup error, never a request-time error and never a silent behavior change. | No `provider`, no providers block | Valid: the neutral default for that concern. | Unknown fields inside every provider config block are rejected -(`deny_unknown_fields` on all new settings structs — PR #838 applied it to -`Ec` but not to `EcProviders`, `DeviceConfig`, or `GeoConfig`, so a typo like -`providr` was silently ignored). +(`deny_unknown_fields` on all new settings structs — the pre-existing `Ec` +struct already has it, but PR #838 shipped `EcProviders`, `DeviceConfig`, +and `GeoConfig` without it, so a typo like `providr` was silently ignored). ## 7. Composition root and adapter parity @@ -210,15 +235,16 @@ prominent in release notes: "fastly"`; the migration guide lists this as a behavior-preserving step for Fastly deployments. - **Geo.** With no geo provider, jurisdiction resolution falls to the - configured default country. The permission model spec (§5) constrains this - combination so it cannot silently grant permissions to mis-attributed - traffic. + configured default country. The permission model spec (§5.3) constrains + this combination so it cannot silently grant permissions to mis-attributed + traffic, and §11 below sequences the default flip so the constraint exists + before the flip does. ## 10. Testing strategy - Provider conformance suite (§3 invariant) run against every shipped provider. -- Enforcement tests per provider kind (§5). +- EC permission-enforcement tests (§5). - Settings validation tests for every row of the §6 table, including the block-without-selector rejection. - Parity suite additions of §7. @@ -232,8 +258,19 @@ prominent in release notes: ID-stability vectors). 2. Settings selection + validation table. 3. Composition root + all four adapters wired through it, parity cases. -4. Device and geo providers with the shared enforcement point of §5. - -Each step is independently reviewable; none depends on the permission model -landing first (the EC gate keeps its current jurisdiction logic until the -permission model PR replaces it). +4. Device and geo provider selection. **The geo neutral default does not + flip in this step**: under the current jurisdiction gate, absent geo + resolves to `Unknown`, which fails closed — flipping the default here + would zero EC issuance for every deployment that had not yet opted into + `[geo] provider = "platform"`. Until step 5, the Fastly adapter's geo + selection defaults to `platform` (today's always-on behavior); the + selector exists, only its default is held back. +5. The permission model PR: flips the geo default to none **in the same + change** that introduces the `default_country` fallback and the §5.3 + acknowledgment guard, and adds the EC permission-enforcement point of + §5; `required_permissions()` appears on the EC trait in this step, not + before (per the §4 minimalism rule). + +Steps 1–4 are independently reviewable, behavior-preserving, and do not +depend on the permission model: the EC gate keeps its current jurisdiction +logic until the permission model PR replaces it. diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index c00f8c348..5e7a327c1 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -5,7 +5,7 @@ **Issue references:** #777–#781 (epic) **Related specs:** `2026-07-30-pluggable-providers-design.md`, `2026-07-30-permission-model-design.md` -**Last updated:** 2026-07-30 +**Last updated:** 2026-07-31 > **Context.** The provider/permission epic is a breaking change to a live > identity system. PR #838's review showed that the riskiest part of such a @@ -32,37 +32,45 @@ and whether that is a preservation or a declared change. **Silent changes are defects.** PR #838 changed six of these without declaring any; each was discoverable only because a deleted test had pinned the old behavior. -| # | Decision (today) | After epic | Status | -| --- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | -| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | -| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | -| 3 | US-state request, no signals at all → no EC (fail-closed) | Policy decision: the shipped `US` baseline decides. If the baseline grants `store-on-device` without a signal, that is a **declared change** requiring sign-off in the policy file review, with rationale in the file itself | Declared change (if made) | -| 4 | UK request, no TCF record → no EC | Same, unless the policy file deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | -| 5 | Unknown jurisdiction (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | -| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC allowed, never tombstoned | Same (withdrawal only where baseline is opt-in; permission spec §4.2) | Preserved | -| 7 | EID transmission requires storage + personalization consent where regulated | Same via `store-on-device` ∧ `select-personalised-ads` | Preserved | -| 8 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | -| 9 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"` | Declared change with a documented restore step (§5) | - -Rows 3 and 4 are policy decisions, not code decisions: they belong in the -`permissions.yaml` review, made explicitly by maintainers — not implied by an -implementation. +| # | Decision (today) | After epic | Status | +| --- | --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Policy decision: the shipped `US` baseline decides. If the baseline grants `store-on-device` without a signal, that is a **declared change** requiring sign-off in the policy review, with rationale in the policy itself | Declared change (if made) | +| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | +| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | +| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | +| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | +| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | + +Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in +the `[permissions]` policy review, made explicitly by maintainers — not +implied by an implementation. ## 3. Identity stability guarantee -For a deployment that selects `provider = "hmac"` and carries its passphrase -over verbatim: - -- The minted identifier is **bit-identical** to today's: - `HMAC-SHA256(passphrase, normalized_ip)` in the existing encoding. -- Cookie name, attributes, and max-age are unchanged; existing `ts-ec` - cookies are recognized by the provider. -- KV identity-graph keys (`ec_hash`, normalization) are unchanged; no - existing graph row is orphaned. - -Enforced by **pinned known-answer tests**: fixed passphrase + IP → exact -expected identifier, cookie string, and KV key, committed as vectors so any -divergence fails CI rather than rotating a production identity base. +Today's EC identifier is `{64-hex}.{6-char}` where the 64-hex part is +deterministic — `HMAC-SHA256(passphrase, normalized_ip)` — and the 6-char +suffix is **random per mint** (an existing test asserts two mints differ). +Full identifiers are therefore not reproducible by design, and no test may +pretend otherwise. What stability means, precisely, for a deployment that +selects `provider = "hmac"` and carries its passphrase over verbatim: + +- **The deterministic prefix is bit-identical.** Pinned known-answer + vectors: fixed passphrase + IP → exact expected 64-hex prefix, committed + so any divergence fails CI rather than rotating the production identity + base. +- **Existing cookies stay recognized.** Fixture `ts-ec` values minted by + the pre-epic code pass the provider's `recognize`, and their graph rows + (keyed by the identifier verbatim) remain reachable — no row is orphaned. +- **The hash prefix keeps its semantics.** `ec_hash` remains the 64-hex + prefix, preserving both its stability and its deliberate collision across + identifiers minted from the same IP — the property IP-cluster trust + counting depends on (providers spec §3). +- **Cookie name, attributes, and max-age are unchanged** (the domain + remains config-derived, as today). ## 4. Configuration migration @@ -87,22 +95,32 @@ Requirements: 1. **Old key fails loud.** `[ec] passphrase` is rejected at startup with a message naming the new location — not a generic unknown-field error. + Implementation note: `Ec` already carries `deny_unknown_fields`, which + would reject the key generically; producing the actionable message means + keeping a deprecated `passphrase` field whose presence triggers the + custom error. 2. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no `provider = "hmac"` selector is a startup error (providers spec §6). In PR #838 this configuration — the exact state an operator following the docs reaches if they miss one line — validated green and silently minted zero ECs. -3. **The example config ships the migrated happy path**, uncommented: +3. **PR #838-era keys fail loud.** `provider = "host-signals"` (shipped by + PR #838, deliberately not carried into this epic — providers spec §2) + and `provider = "client-fixed"` are unknown keys and rejected like any + other, so a config written against the PR #838 example cannot silently + select a provider that no longer exists. +4. **The example config ships the migrated happy path**, uncommented: `provider = "hmac"` with its block, `[geo] default_country`, and (for Fastly) the behavior-preserving `[device] provider = "fastly"` and `[geo] provider = "platform"` lines present with a comment stating what removing them changes. PR #838's example shipped the passphrase block uncommented with the selector commented out — steering operators directly into the silent-stateless state. -4. Every misconfiguration in the providers spec §6 table fails at +5. Every misconfiguration in the providers spec §6 table fails at **startup**. Request-time failure for a configuration error is a defect. -5. Config-store payload validation (`ts config push`) applies the same - rules, so a bad config is rejected at push time, before any instance +6. Config-store payload validation (`ts config push`) applies the same + rules — including `[permissions]` policy validation (permission spec + §3.3) — so a bad config is rejected at push time, before any instance restarts into it. ## 5. Behavior-preserving migration recipe (operator-facing) @@ -122,18 +140,30 @@ provider = "fastly" # Fastly deployments: preserves the JA4 bot gate [geo] provider = "platform" # preserves per-request jurisdiction detection -default_country = "FR" # used only when the host lookup fails +default_country = "FR" # used only when the host lookup fails (fail-closed) + +# Preserves today's treatment of countries outside the regulation lists +# ("non-regulated" → identity allowed). Omit this section to adopt the +# protective default instead: signal required worldwide (§2 row 7). +[permissions.groups.non-regulated] +default = "granted" + +[permissions.rules] +default = "non-regulated" ``` and separately documents the neutral configuration and what it does _not_ do. The guide states explicitly that `default_country` alone does not replace geo -lookup, and why the permissive-default + no-geo combination requires the -explicit acknowledgment flag (permission spec §5.3). +lookup, why the no-geo combination requires the explicit acknowledgment flag +(permission spec §5.3), and that no recipe preserves row 8 of §2 — the +global honoring of opt-out signals is unconditional. ## 6. Rollout sequence and observability -1. Implementation PRs land in the epic's order (providers first, permission - model second); each is reviewable against §2 in isolation. +1. Implementation PRs land in the epic's order (providers spec §11: + providers first with the geo default held at today's behavior, the + permission model PR flipping it together with its guard); each PR is + reviewable against §2 in isolation and states which rows it touches. 2. Before/after deploy, operators watch **EC issuance rate** and EID attachment rate; the migration guide names these as the canary metrics, because the failure mode of a bad migration is a silent drop to zero (or a @@ -143,18 +173,21 @@ explicit acknowledgment flag (permission spec §5.3). signal permissions. One line, greppable, stable format. 4. Rollback is config-only where possible: reverting to the previous config version restores the previous behavior on the previous binary. The - one irreversible artifact is withdrawal tombstones — which is why §2 row 6 - (no tombstones without affirmative withdrawal in an opt-in jurisdiction) - is non-negotiable. + one irreversible artifact is withdrawal tombstones — which is why the + withdrawal triggers (permission spec §4.2) are exhaustive and why §2 + rows 6 and 8 call out tombstoning explicitly. Cleanup of identities + minted before a policy tightening (permission spec §4.2 trigger 3) is an + operational action documented in the guide, not an automatic one. ## 7. Documentation deliverables - Migration guide page (§5), linked from `CHANGELOG.md` and the release notes. - `configuration.md` documents **every** valid `provider` value for all - three concerns, and documents environment-variable overrides only if they - actually work in production builds (in PR #838 the documented - `TRUSTED_SERVER__EC__PROVIDER` override existed only under `#[cfg(test)]`). + three concerns, the full `[permissions]` schema, and environment-variable + overrides only if they actually work in production builds (in PR #838 the + documented `TRUSTED_SERVER__EC__PROVIDER` override existed only under + `#[cfg(test)]`). - The permission model page states the §4 precedence rules of the permission spec verbatim — operator docs and normative spec must not diverge on precedence, and prose like "signals are mapped as a grant or a revoke" From 9886091e5ca843071cfd0ca38f32fbd881a9a33e Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 31 Jul 2026 01:41:14 -0700 Subject: [PATCH 03/24] Address review: close identity/privacy gaps in the provider and permission specs Blocking findings from review of PR #986, all addressed: - Raw EC egress is now a first-class enforcement point with a mandatory egress inventory (user.id, derived request IDs, page bids, proxy/click forwarding, identify, pull/batch sync, graph access): bidstream egress requires both purposes, first-party identity operations require store-on-device, revocation is exempt, and no-provider mode never vacuously allows an existing cookie to egress. - The behavior-preserving recipe carries the complete policy table plus a delta; a partial permissive-default-only policy is called out as the trap it is, and the exact recipe text becomes a CI fixture run through the decision matrix. - The EC permission gate is split: it covers minting and identity use only; parse, canonicalization, and tombstoning always run, with a spy-provider test - a blanket gate would block withdrawal in exactly the state an opt-out produces. - Provider switching gets active-writer/legacy-readers semantics ([ec] legacy_providers) so old identities keep resolving and stay withdrawable; unmatched cookies never egress. - The lifecycle contract now distinguishes the canonical graph key (provider-owned canonicalization, equivalent envelopes collapse) from the cluster prefix, which must be a literal byte prefix of the graph key because cluster sizing is a KV prefix listing; cluster support is an optional capability with an explicit degradation policy. - Device gating rationale corrected: only geo is circular; device is ungated by decision (security classification authorized by operator selection), with the boundary stated - uses beyond security classification need a vocabulary extension and a gate. - Withdrawal triggers made consistent (TCF refusal withdraws under requires_signal or denied), and a withdrawal-durability contract added: tombstones first, cookie expiry only on success, browser-side durable signals as the retry queue, fault-injection tests. - A signal-normalization matrix is now required (dual-TCF conflict modes, expiry, proxy mode, KV fallback, exact GPP fields), and malformed-but-present records fail closed for acquisition instead of degrading to absent. - Auction jurisdiction class is an explicit per-group regime attribute (gdpr / us-privacy / none), never inferred from purpose flags, and a first-class enforcement point. - The no-geo acknowledgment guard now keys on any enabled jurisdiction consumer, not only EC-provider selection. - Policy validation additionally requires rules.default when the section is present, rejects empty sections and case-insensitive duplicate keys, and canonicalizes default_country. - Resolve endpoint: exact Origin-allowlist membership or session-bound CSRF token (Sec-Fetch-Site demoted to defense-in-depth), real replay mitigation (session nonce or one-time consumption), and bounded-input requirements with 413 boundary tests. - Response hook: structured mutation operations that core validates and attributes (no raw header-map access), framing/hop-by-hop headers reserved, and a normative response-eligibility matrix. - Each spec now carries an explicit divergence table against its issue (#778, #779, #782) so there is one acceptance contract. - Non-blocking clarifications folded in: geo lookup-failure residual declared and metered, deterministic entropy required in conformance tests. --- ...26-07-30-client-cycle-ec-resolve-design.md | 46 ++-- ...integration-response-header-hook-design.md | 50 +++- .../2026-07-30-permission-model-design.md | 218 ++++++++++++++---- .../2026-07-30-pluggable-providers-design.md | 157 ++++++++++--- ...07-30-provider-migration-rollout-design.md | 85 ++++--- 5 files changed, 432 insertions(+), 124 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index 1b9eacc12..e0a5529fe 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -50,17 +50,28 @@ Everything in this spec follows from that. `POST /_ts/api/v1/ec/resolve` (final path TBD) MUST: -1. **Reject cross-site requests.** Require a same-site assertion: `Origin` - (or `Sec-Fetch-Site: same-origin/same-site`) validated against the - publisher's origin allowlist — configuration that does not exist yet and - must be defined by this feature (§7, question 5); requests without a - validating header are rejected. CSRF-token designs are acceptable but not - required if origin-based rejection is enforced. -2. **Verify the payload cryptographically per provider.** The provider's - `resolve_from_client` accepts only payloads that are signed by an - expected party, **audience-bound** to this publisher, and **expiring** - (bounded lifetime, single-use where the scheme allows). A provider whose - payloads are replayable constants fails this bar by construction. +1. **Reject cross-site requests with an exact origin check.** The request + is authorized only by **exact membership of the `Origin` header value in + the publisher's origin allowlist** — configuration that does not exist + yet and must be defined by this feature (§7, question 5) — or by a + **session-bound CSRF token**. `Sec-Fetch-Site` is **defense-in-depth + only, never an authorizing alternative**: it carries no origin value to + compare against an allowlist, and `same-site` admits every sibling + subdomain — one compromised or attacker-registered subdomain would be + enough to set identity. Requests with no `Origin` and no valid token are + rejected. +2. **Verify the payload cryptographically per provider — including against + replay.** The provider's `resolve_from_client` accepts only payloads + that are signed by an expected party, **audience-bound** to this + publisher, and **expiring**. Audience binding and expiry alone do not + mitigate replay — a captured token installs in another browser for the + whole validity window — so one of the following is additionally + required: **binding to the requesting browser session** (a server-issued + nonce the payload must embed), or **server-side one-time consumption** + (a replay cache on the payload's unique id). A scheme that can support + neither may only ship if its residual replay window is quantified and + explicitly accepted in the feature's issue — "single-use where the + scheme allows" is not a mitigation. 3. **Preserve the identity-graph invariant.** The cookie is set only after the corresponding graph row is written, mirroring the organic path. Graph unavailable → no cookie, same as organic generation. @@ -75,6 +86,13 @@ Everything in this spec follows from that. 6. **Be uncacheable and permission-gated.** `Cache-Control: no-store`; the same `store-on-device` permission gate as organic EC creation runs before any cookie is set. +7. **Bound every input.** A maximum request-body size (order of the 64 KiB + limit PR #838 at least had), enforced by a **bounded read independent of + `Content-Length`** — a missing, false, or chunked length must not bypass + it; a `Content-Type` allowlist; and a length/character-set constraint on + the resulting identifier that keeps it cookie-safe and within the KV + limits of the providers spec §3. Tests exercise the exact 413 boundary + and the missing/false/chunked-length cases. ## 4. Requirements on the page script @@ -103,8 +121,10 @@ sentence as the only guardrail. ## 6. Testing -- Endpoint: origin-rejection, expired/replayed/foreign-audience payload - rejection, graph-unavailable refusal, permission-gate refusal — each as an +- Endpoint: origin-rejection (including `Sec-Fetch-Site`-only requests, + which must fail), expired/replayed/foreign-audience payload rejection, + graph-unavailable refusal, permission-gate refusal, and the §3.7 body / + content-type / identifier limits at their exact boundaries — each as an integration test, not only unit tests. - Browser round trip (JS → POST → Set-Cookie → next request recognized) in the integration suite; PR #838 shipped the JS with in-process unit tests diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 01aea582d..b91846222 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -29,6 +29,15 @@ mutators to the outbound response for HTML document responses it processed. - `IntegrationRegistration::builder(ID).with_response_mutator(...)` registers a mutator; `IntegrationRegistry::apply_response_headers(...)` applies all registered mutators in registration order. +- **The mutator API is structured operations, not header-map access.** A + mutator returns (or is handed a recorder for) typed operations — + `append(name, value)`, `replace(name, value)`, + `append_set_cookie(cookie)` — which **core validates and applies**, + attributing each to its integration id. PR #838's shape handed the + integration an unrestricted `&mut HeaderMap`, which makes §3's collision + policy unenforceable by construction: core cannot validate or attribute + writes it never sees. An API that cannot express a violation beats one + that promises to catch it. - **Every adapter calls the apply point** on its outbound-response path for processed documents. The call site lives in shared response-finalization code where one exists; where adapters finalize independently, each adapter @@ -45,13 +54,16 @@ mutators to the outbound response for HTML document responses it processed. - Mutators may not touch **reserved surface**, which is defined at two granularities because `Set-Cookie` is multi-valued: (a) reserved header - _names_ — the `x-ts-*` namespace and the consent/privacy headers core - emits; (b) reserved cookie _names_ within `Set-Cookie` — `ts-ec`, + _names_ — HTTP framing and hop-by-hop headers (`Content-Length`, + `Transfer-Encoding`, `Connection`, `Trailer`, `Upgrade`, `TE`, + `Keep-Alive`), the `x-ts-*` namespace, and the consent/privacy headers + core emits; (b) reserved cookie _names_ within `Set-Cookie` — `ts-ec`, `ts-eids`, and the other `ts-*` cookies core owns. An integration may append its own `Set-Cookie` values; it may not set or expire a reserved - cookie name. Violations are dropped and logged at `warn` with the - integration id. The reserved-cookie list is a single constant next to the - cookie definitions, not duplicated in the hook. + cookie name. Violations are rejected at the operation layer (§2) and + logged at `warn` with the integration id. The reserved lists are single + constants next to the definitions they protect, not duplicated in the + hook. - For non-reserved headers, the mutator API distinguishes **append** from **replace** explicitly; the default is append (for `Set-Cookie`, append is the only non-reserved operation — replace is not offered). Replacing a @@ -59,6 +71,24 @@ mutators to the outbound response for HTML document responses it processed. - Later registrations see earlier mutations (order = registration order, which is deterministic). +## 3a. Response eligibility — normative + +Which responses the hook runs on, enumerated so two implementations cannot +diverge silently: + +| Response | Hook runs? | +| --------------------------------------------- | ------------------------------------------------------------ | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Yes, applied at serve time (mutations are not cached) | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` | No | + +This deliberately narrows #782's general "outbound response" phrasing to +processed documents (§6). + ## 4. Done-when (from #782, sharpened) 1. Trait + builder + registry application, each public item documented. @@ -79,3 +109,13 @@ its first real consumer is identified** (§4.2) — at any point in the epic's sequence, blocking nothing and blocked by nothing. If no consumer materializes, it does not land; being unblocked is not a reason to ship scaffolding. + +## 6. Divergences from issue #782 + +This spec supersedes #782 on the following points; the issue is updated to +reference this spec when the PR merges: + +| #782 says | This spec says | Why | +| -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | +| Mutations apply to the outbound response generally | Eligibility is the explicit §3a matrix, centered on processed documents | Pass-through/error/304 mutation has different semantics per adapter; enumerating beats implying | +| Ship the trait + registry + adapter application | Additionally: structured operations API (§2), reserved surface (§3), and a real consumer in the same PR (§4.2) | PR #838 shipped the trait with zero call sites; an unrestricted `&mut HeaderMap` cannot enforce any collision policy | diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index bd5da97c9..a66037a7e 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -112,9 +112,15 @@ overrides. Each permission resolves to an **acquisition rule**: ```toml [permissions.groups.gdpr-eu] +regime = "gdpr" default = "requires_signal" [permissions.groups.us-opt-out] +regime = "us-privacy" +default = "granted" + +[permissions.groups.non-regulated] +regime = "none" default = "granted" [permissions.rules] @@ -123,10 +129,10 @@ US = "us-opt-out" # Overrides name explicit acquisition rules — no +/- sigil syntax; TOML # expresses the target state directly. "US/CA" = { group = "us-opt-out", overrides = { select-personalised-ads = "requires_signal" } } -# Reserved key: countries that resolve but match no rule. Distinct from -# [geo] default_country, which handles requests that resolve no country at -# all (§5.4). -default = "gdpr-eu" +# Reserved key: countries that resolve but match no rule. Required whenever +# the [permissions] section is present. Distinct from [geo] default_country, +# which handles requests that resolve no country at all (§5.4). +default = "non-regulated" ``` A group's `default` covers unlisted permissions; a group may also name @@ -135,6 +141,14 @@ target state (including `requires_signal`) is expressible — PR #838's `+`/`-` sigil scheme could not express "requires a signal", the most common real-world override. +Each group carries a required **`regime`** class (`gdpr`, `us-privacy`, or +`none`). This is the explicit legal-classification channel: consumers that +need a jurisdiction _class_ — above all server-side auction dispatch — read +`regime`, never infer a class from purpose flags. Inference is lossy +(Purpose 1 and Purpose 4 may legitimately carry different rules, and a +non-GDPR operator may choose an opt-in Purpose 4) and would smuggle legal +meaning back into identifiers this spec declares purely technical (§2). + ### 3.3 Validation — at config acceptance, not request time Policy is validated where every other setting is: at `ts config push` (a bad @@ -153,8 +167,17 @@ Validation rejects: the region part matches `[A-Z0-9]{1,3}`. The `US/CA` slash form is the house rule-key format corresponding to ISO 3166-2 `US-CA`; - references to permissions outside the enforced vocabulary (§2); -- references to undefined groups; -- groups that neither list every permission nor provide `default`. +- references to undefined groups, and groups missing the `regime` class; +- groups that neither list every permission nor provide `default`; +- a present `[permissions]` section without a `rules.default` entry (§5.4 + depends on it existing — its absence must be a validation error, not a + runtime surprise); +- an empty `[permissions]` section (ambiguous intent: an operator who wants + the compiled-in fallback omits the section entirely); +- duplicate rule keys under case-insensitive comparison (`FR` and `fr`); +- a `[geo] default_country` that is not an assigned ISO code; it is + canonicalized to uppercase, and startup logs which rule (or + `rules.default`) it resolves to. ### 3.4 One source of jurisdiction truth @@ -167,12 +190,13 @@ country to one has no effect on the other, and an operator has no signal that they disagree). Requirement: the auction gate's jurisdiction class derives from the same -resolved policy (a country is GDPR-class when its rule resolves to an -opt-in baseline for `select-personalised-ads`). Where the legacy lists must +resolved policy, reading the rule's explicit **`regime`** class (§3.2) — a +country is GDPR-class when its rule resolves to a `regime = "gdpr"` group. +The class is never inferred from purpose flags. Where the legacy lists must survive an interim period, a CI test asserts consistency between each list -and the policy table, with deliberate divergences recorded as explicit, -commented exceptions in the test — never silent. Both legacy lists are in -scope, not only the GDPR one. +and the policy's regime classes, with deliberate divergences recorded as +explicit, commented exceptions in the test — never silent. Both legacy +lists are in scope, not only the GDPR one. ### 3.5 Shipped-table coverage @@ -250,16 +274,21 @@ The triggers, exhaustively — nothing else withdraws: baseline.** (For US states this preserves today's behavior; elsewhere it is the declared change of §4's global-opt-out rule.) 2. **A TCF record refusing `store-on-device` withdraws iff the baseline is - `requires_signal`.** Where the baseline is `granted`, refusal blocks - _new_ grants but never tombstones: tombstones are irreversible, and - PR #838 wrote them for visitors in unregulated jurisdictions whose - global CMP emitted a purpose-refusing string — permanent identity loss - under a regime the deployment never opted into. + `requires_signal` or `denied`.** Where the baseline is `granted`, + refusal blocks _new_ grants but never tombstones: tombstones are + irreversible, and PR #838 wrote them for visitors in unregulated + jurisdictions whose global CMP emitted a purpose-refusing string — + permanent identity loss under a regime the deployment never opted into. + (The `denied` arm exists so trigger 3 is coherent: after a policy + tightens to `denied`, an affirmative refusal must still be able to + withdraw a pre-existing identity.) 3. **A policy edit is not a user signal.** Tightening a baseline to `denied` stops new identity but does not itself tombstone identities minted before the change; cleaning those up is an operational action - (migration spec §6). An affirmative user signal (trigger 1 or 2) still - withdraws them. + (migration spec §6). An affirmative user signal (trigger 1, or trigger 2 + under the now-`denied` baseline) still withdraws them — with a test + pinning exactly this sequence: existing EC → policy tightens to + `denied` → refusal arrives → tombstone. 4. **Absence of signal never destroys identity.** A visitor who has not yet made a choice is never stripped of an existing identity. @@ -269,6 +298,54 @@ withdrawal even when a consenting TCF record is present. every trigger above; in PR #838 the headline "withdrawal expires identity" behavior had no unit test at all. +### 4.3 Withdrawal durability + +Withdrawal is two writes — the KV tombstones and the cookie expiry — and +the contract for partial failure is explicit (PR #838 expired the cookie +first and logged-and-swallowed tombstone-write failures, which can leave a +live graph identity with no browser handle pointing at it): + +- **Order: tombstones first, cookie expiry second.** The cookie is expired + only after the tombstone writes succeed. +- **On tombstone-write failure, the cookie is left in place** and the + failure is logged at `error` with a metric. This is deliberately + self-healing: every withdrawal trigger is durable client-side (GPC is a + browser setting, the TCF record lives in the CMP's storage), so the next + request re-presents the signal and retries the whole withdrawal. No + quarantine queue is needed; the browser is the retry queue. +- Identify, batch-sync, and pull-sync treat a tombstone as authoritative + revocation (as today); a row whose withdrawal is pending retry is simply + still live until the retry lands, and never partially withdrawn. +- Fault-injection tests cover: tombstone write fails → cookie untouched, + error logged; subsequent request with the same signal → withdrawal + completes. + +### 4.4 Signal normalization + +§4's precedence operates on normalized inputs: one effective consent +record and one effective opt-out state per request. The normalization +layer is where today's real-world mess lives, and PR #838 collapsed it +silently. The implementation ships a **normalization matrix** — a +table-driven spec-and-test artifact — covering at minimum: + +- **Dual consent records**: standalone TCF cookie vs. GPP-embedded TCF, + including per-purpose disagreement, resolved per the existing configured + conflict modes (restrictive / permissive / newest). Each mode is either + preserved or explicitly retired in the migration matrix — not dropped. +- **Record expiry** and the persisted-KV consent fallback: when a stored + record substitutes for an absent live one, and how staleness is bounded. +- **Proxy/mirror mode** (CMP consent mirrored server-side): where the + mirrored state enters precedence. +- **Exact GPP fields**: which section fields constitute a sale/sharing/ + targeted-advertising opt-out, enumerated per supported section — "GPP + opt-out" is not a single bit. +- **Malformed-but-present records fail closed for acquisition**: a consent + record that is present but undecodable blocks grants (it does not + degrade to "absent", which under a `granted` baseline would turn garbage + into a grant — the fail-open path in both #838 and the first draft of + this spec). A malformed record never triggers withdrawal — destruction + requires an affirmative, decodable signal (§4.2). + ## 5. Jurisdiction resolution ### 5.1 Order @@ -288,6 +365,13 @@ all — that is the capability check of providers spec §6, and it prevents a "selected but always empty" provider from silently converting every request to §5.3 semantics without §5.3's guard. +Declared residual: when the default country's baseline is permissive, a +per-request lookup failure is a per-request grant to traffic of unknown +origin — this path is not fail-closed, and the spec does not pretend it is. +The lookup-failure rate is exported as a metric and logged, so an elevated +rate (a degraded geo backend silently converting traffic to the default) is +observable rather than invisible. + ### 5.3 No geo provider selected Every request resolves to `default_country` — jurisdiction becomes a static @@ -297,14 +381,20 @@ this dangerous: with a `requires_signal` baseline, a page-global CMP that emits a consenting TCF string grants permissions for every mis-attributed visitor just as effectively. -Constraint: **startup fails** when an EC provider is selected and no geo -provider is, unless the operator sets an explicit acknowledgment -(`[geo] assume_single_jurisdiction = true`). Stateless deployments (no EC -provider) are exempt. Without this guard, the natural migration config -(`default_country = "US"`, geo unset) silently grants `store-on-device` and -EID transmission to every EU visitor — the highest-severity finding of the -PR #838 review. The startup log always prints the effective baseline and -whether geo is live. +Constraint: **startup fails** when no geo provider is selected and any +**jurisdiction consumer** is enabled, unless the operator sets an explicit +acknowledgment (`[geo] assume_single_jurisdiction = true`). Jurisdiction +consumers are enumerated, not implied: an EC provider is selected, +server-side auction dispatch is gated on `regime` (§7), or any raw-EC / +EID egress path is active. An EC-provider-only exemption would be too +narrow — a stateless deployment still dispatches auctions off the policy's +regime class, and no geo + a permissive static jurisdiction misclassifies +EU traffic for that decision just as it would for identity. Only a +deployment with **no** jurisdiction-sensitive behavior is exempt. Without +this guard, the natural migration config (`default_country = "US"`, geo +unset) silently grants `store-on-device` and EID transmission to every EU +visitor — the highest-severity finding of the PR #838 review. The startup +log always prints the effective baseline and whether geo is live. ### 5.4 Defaults, two distinct fallbacks @@ -319,17 +409,17 @@ migration story unresolvable (migration spec §2, rows 5 and 7). ## 6. Failure-mode matrix — normative -| Condition | Resolution behavior | -| ---------------------------------------------------- | ------------------------------------------------------------ | -| Geo lookup fails at request time (provider selected) | `default_country` baseline | -| No geo provider configured | `default_country` baseline, guarded by §5.3 | -| Country resolved, no matching rule | Policy `rules.default` | -| Region resolved, no region rule | Country rule | -| No `[permissions]` section | Compiled-in fallback: everything `requires_signal` | -| Malformed policy | Rejected at config push / startup (§3.3) — never per request | -| No `default_country` | Startup failure | -| Undecodable TCF/GPP string | Treated as absent; opt-out signals still honored | -| Signals contradict (opt-out + consent) | Opt-out wins (§4) | +| Condition | Resolution behavior | +| ---------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| Geo lookup fails at request time (provider selected) | `default_country` baseline | +| No geo provider configured | `default_country` baseline, guarded by §5.3 | +| Country resolved, no matching rule | Policy `rules.default` | +| Region resolved, no region rule | Country rule | +| No `[permissions]` section | Compiled-in fallback: everything `requires_signal` | +| Malformed policy | Rejected at config push / startup (§3.3) — never per request | +| No `default_country` | Startup failure | +| Undecodable TCF/GPP record (present but malformed) | Blocks grants (fail-closed acquisition, §4.4); never withdraws; opt-out signals still honored | +| Signals contradict (opt-out + consent) | Opt-out wins (§4) | The overall posture is **fail-closed**: every ambiguous state resolves to the configured baseline or more restrictive, and the one configuration that @@ -350,12 +440,41 @@ Consumers of the resolved set in this epic: providers will require a two-phase resolution that must be specified then, not improvised. 2. **EC lifecycle** — creation requires `store-on-device`; withdrawal per - §4.2. -3. **Bidstream EIDs** — transmission requires `store-on-device` ∧ - `select-personalised-ads`. + §4.2. Recognition, canonicalization, and revocation of an existing + identifier are **never** permission-gated — they must run precisely when + permissions are withdrawn (providers spec §5). +3. **Every raw-EC egress**, not only EIDs. The raw EC identifier leaves the + process today as OpenRTB `user.id`, inside derived auction request IDs, + on the page-bids path, on proxied/click/Testlight forwarding, through the + identify endpoint, through pull/batch sync, and via identity-graph + reads/writes. Gating EIDs alone (PR #838's shape) leaves the raw EC + reaching bidders when Purpose 1 is granted and Purpose 4 refused; worse, + #838's `ec_allowed` was **vacuously true with no provider configured** + (`is_none_or`), so an existing canonical cookie still escaped in + stateless mode. The contract: + - The implementation maintains an **egress inventory**: every code path + where a raw EC (or a value derived from it) leaves the process is + enumerated in one table, each mapped to its required permissions, with + a test per row. + - **Bidstream egress** (`user.id`, EC-derived request IDs, page bids, + EIDs) requires `store-on-device` ∧ `select-personalised-ads` — the raw + EC is identity in the bidstream and is gated exactly as EIDs are. + - **First-party identity operations** (identify, pull/batch sync, + graph reads/writes other than revocation) require `store-on-device`. + - **Revocation paths are exempt** — tombstoning must work when + permissions are unset. + - With **no EC provider configured**, identity use fails closed: a + cookie value present on the request never egresses anywhere; it is + never vacuously allowed. +4. **Bidstream EIDs** — transmission requires `store-on-device` ∧ + `select-personalised-ads` (subsumed by point 3's inventory; listed + separately because it is the one gate PR #838 had). +5. **Server-side auction dispatch** — gated on the explicit policy `regime` + class (§3.4), a first-class enforcement point, not inferred from purpose + flags. The client-cycle resolve endpoint (own spec, currently on hold) would be a -fourth consumer if and when it proceeds. +further consumer if and when it proceeds. ## 8. Testing strategy @@ -365,6 +484,11 @@ fourth consumer if and when it proceeds. consent module, replaced by happy-path cases only) is restored in equivalent form against the new API; signal-precedence conflicts (opt-out + consenting TCF) are mandatory cases, not optional ones. +- The §4.4 normalization matrix as table-driven tests, including every + configured conflict mode and the malformed-record rows. +- The §7 raw-EC egress inventory: one test per inventoried egress proving + the gate, plus a denylist-style check that no ungated egress exists. +- §4.3 fault-injection cases. - Policy validation tests for every §3.3 rejection, exercised through both acceptance paths (push-time and startup). - Shipped-table coverage test (§3.5) and jurisdiction-consistency test @@ -382,3 +506,15 @@ fourth consumer if and when it proceeds. - Per-signal jurisdiction scoping (honoring GPC only where a law defines it): rejected in favor of the global rule in §4; revisiting it is a policy-model change requiring its own review. + +## 10. Divergences from issue #779 + +This spec supersedes #779 on the following points; the issue is updated to +reference this spec when the PR merges, so there is one acceptance contract, +not two: + +| #779 says | This spec says | Why | +| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- | +| Unmatched countries fall to `default_country` | Unmatched-but-resolved countries fall to the policy's `rules.default`; `default_country` covers only unresolved requests | The two states had different pre-epic behavior; collapsing them made migration unresolvable (§5.4) | +| The full TCF purpose vocabulary is modeled | Only enforced purposes appear (§2) | Nine inert purposes in a policy file are a compliance hazard, not forward compatibility | +| Policy is an embedded file | Policy is `[permissions]` in `trusted-server.toml` (§3.1) | Runtime config-store pipeline; validation at push time | diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index ea0ac765a..88035ea91 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -36,8 +36,7 @@ Goals: - An **EC provider declares** the permissions its data use requires (see the permission model spec); **core enforces** that declaration. A provider cannot authorize itself. (Geo and device providers are governed - differently — they execute as _inputs_ to permission resolution and cannot - be gated on its output; see §5.) + differently, for two different reasons spelled out in §5.) - All adapters (Fastly, Axum, Cloudflare, Spin) behave identically for identical configuration, or fail loudly at startup where a host cannot satisfy the selected provider. @@ -95,20 +94,26 @@ An `EdgeCookieProvider` owns the **complete lifecycle** of the identifiers it mints. Every lifecycle operation core performs on an EC value MUST be routed through the selected provider: -| Lifecycle operation | Where core uses it today | Contract | -| ------------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Mint** | EC generation on first eligible request | Provider returns the identifier (and only core writes the cookie). | -| **Recognize** | Reading `ts-ec` back from the request; deciding `ec_was_present` | Provider validates that a returned cookie value is one of its identifiers. A value the selected provider does not recognize is treated as absent. | -| **Key for the graph row** | KV identity-graph row reads/writes | The row key is the identifier **verbatim**; the provider guarantees its identifiers are stable and KV-safe. | -| **Hash prefix** | IP-cluster sizing (`cluster_trust_threshold` prefix listing), pull-sync dedupe, log redaction | Provider maps an identifier to its hash prefix. This prefix **deliberately collides** across identifiers minted from the same client evidence — the collision is load-bearing for cluster-trust counting, and a provider that returns a unique-per-identifier value silently breaks it. | -| **Tombstone** | Withdrawal: expiring the cookie and writing revocation markers | The identifiers eligible for tombstoning are exactly those the provider recognizes — never a shape-gated subset. | +| Lifecycle operation | Where core uses it today | Contract | +| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Mint** | EC generation on first eligible request | Provider returns the identifier (and only core writes the cookie). | +| **Parse / canonicalize** | Reading `ts-ec` back from the request; deciding `ec_was_present`; batch-sync ingestion | Provider parses a cookie value into its **canonical** identifier, or rejects it. Canonicalization is provider-owned: case variants and equivalent envelopes of the same identity (per #778) parse to the same canonical identifier. A value the selected provider does not recognize is treated as absent (but see §6.1 legacy readers). | +| **Canonical graph key** | KV identity-graph row reads/writes | The provider maps a canonical identifier to its graph key: stable, KV-safe (within KV length and character-set limits), collision-free across the provider's identifier space, and namespaced so two providers' key spaces cannot collide. Two equivalent envelopes of one identity map to one key — verbatim cookie bytes as the key would fork graph rows on canonicalization differences and discard today's batch-sync canonicalization. | +| **Cluster prefix** (optional capability) | IP-cluster sizing (`cluster_trust_threshold`, implemented as a **KV prefix listing**), pull-sync dedupe, log redaction | A provider declaring cluster support returns a prefix that is a **literal byte prefix of the canonical graph key** — the cluster count lists keys by prefix, so an independently derived hash that is not an actual key prefix silently reports the wrong cluster size. The prefix deliberately collides across identifiers minted from the same client evidence. A provider without the capability declares so, and cluster-dependent gating follows a configured degradation policy (treat cluster size as unknown, with the KV-write decision that implies made explicit in config) instead of counting garbage. | +| **Tombstone** | Withdrawal: expiring the cookie and writing revocation markers | The identifiers eligible for tombstoning are exactly those the provider parses — never a shape-gated subset. | **Invariant:** for every provider `P` and every identifier `id` minted by `P`, -`P.recognize(id)` is true, `P` produces a stable hash prefix for `id` (and -two identifiers minted from the same client evidence share it), and a -withdrawal request carrying `id` tombstones it. A conformance test suite MUST -assert this round-trip for every shipped provider, and the suite MUST be -written so a future provider crate can run it against its own implementation. +`P.parse` round-trips `id` (including its case variants and equivalent +envelopes, which all canonicalize to the same identifier and graph key); +where `P` declares cluster support, `cluster_prefix(id)` is a literal prefix +of `graph_key(id)` and is shared by identifiers minted from the same client +evidence; and a withdrawal request carrying `id` tombstones it. A conformance +test suite MUST assert this round-trip for every shipped provider — including +case-variant, equivalent-envelope, cross-provider key-namespace, and KV +length/charset cases — and the suite MUST be written so a future provider +crate can run it against its own implementation. Conformance tests inject +deterministic entropy; probabilistic assertions ("two random suffixes +differ") are not accepted. ## 4. Trait surface: minimalism rule @@ -116,7 +121,10 @@ Every trait method MUST have at least one production (non-test) caller in the same PR that introduces it. Speculative surface observed in PR #838 that MUST NOT ship without a caller: -- `keys_equal` (no production caller; existed to serve a unit test), +- `keys_equal` (no production caller; existed to serve a unit test — its + legitimate purpose, #778's equivalent-envelope comparison, is satisfied + structurally by §3's canonicalizing `parse` instead: equivalents + canonicalize to the same identifier, so no comparison method is needed), - `GeneratedEdgeCookie::response_headers` (empty in all built-ins, plumbed through three layers), - `IdentityInput.permissions` / `IdentityInput.consent` (ignored by all @@ -133,15 +141,21 @@ The minimal `EdgeCookieProvider` surface implied by §3 is: pub trait EdgeCookieProvider { /// Stable configuration key ("hmac"). fn id(&self) -> &'static str; - /// Permissions this provider's data use requires. Enforced by core. + /// Permissions this provider's data use requires. Enforced by core for + /// minting and identity use — never for parse/tombstone (§5). fn required_permissions(&self) -> PermissionSet; /// Mint an identifier from request evidence. fn generate(&self, input: &IdentityInput<'_>) -> Result>; - /// Whether `value` is an identifier this provider minted. - fn recognize(&self, value: &str) -> bool; - /// Hash prefix of a recognized identifier (see §3: collides by design - /// across identifiers minted from the same client evidence). - fn hash_prefix(&self, id: &EcId) -> HashPrefix; + /// Parse and canonicalize a cookie value into this provider's + /// identifier; None when unrecognized. Equivalent envelopes and case + /// variants canonicalize to the same identifier. + fn parse(&self, value: &str) -> Option; + /// Canonical KV graph key for a parsed identifier. + fn graph_key(&self, id: &EcId) -> GraphKey; + /// Cluster capability: a literal byte prefix of `graph_key(id)`, shared + /// across identifiers minted from the same client evidence. None when + /// the provider does not support IP-cluster semantics (§3). + fn cluster_prefix(&self, id: &EcId) -> Option; } ``` @@ -150,24 +164,45 @@ trait at step 5 of §11, together with its enforcement point.) ## 5. Permission enforcement is core's job — for EC providers -Before executing an **EC provider**, core resolves the request's permission -set (see the permission model spec) and refuses to run a provider whose -`required_permissions()` are not all set, with a test proving a provider -declaring an unset permission does not execute. +The gate is on **minting and identity use, never on the lifecycle +operations that withdrawal depends on**. Before minting through an EC +provider or using an identity (raw-EC egress, permission model spec §7), +core resolves the request's permission set and refuses when the provider's +`required_permissions()` are not all set. **Parse, canonicalization, graph +lookup for revocation, and tombstoning always run**, permissions or not — a +blanket execution gate would refuse to run the provider in exactly the +state an opt-out produces, making the withdrawal it demands impossible. A +spy-provider test pins the split: with `store-on-device` unset, `generate` +is never called while a withdrawal request still parses the cookie and +writes tombstones. + +The gate applies to EC providers **only**. Geo and device are ungated for +two _different_ reasons, stated separately because only one of them is +structural: + +- **Geo: circularity.** The permission set is resolved _from_ jurisdiction, + which is resolved _by_ the geo provider. Gating geo on the resolved set + is unsatisfiable. +- **Device: a decision, not a circularity.** Device classification is not + an input to permission resolution (the inputs are jurisdiction, policy, + and signals), so ordering geo → resolution → device → EC and gating + device is perfectly implementable. This spec deliberately does not: + the shipped device providers process technical request metadata (UA, + JA4/HTTP-2 fingerprints) for **security classification** — the bot gate + protecting KV-backed identity writes — which must run precisely for + traffic that has granted nothing. The authorization for that processing + is the operator's explicit `[device] provider` selection, and this spec + records that as the decision, with its privacy implication stated: a + device provider whose data use goes beyond security classification (for + example feeding fingerprints into targeting or identity) is **not + authorized by selection alone** and requires a vocabulary extension plus + a gate before it may ship. -This gate applies to EC providers **only**, and the reason is structural, -not convenience: the permission set is resolved _from_ jurisdiction, which -is resolved _by_ the geo provider — gating geo (or device, which runs in -the same pre-resolution phase) on the resolved set would be circular. PR #838 declared `required_permissions` on all three traits but consulted it only for the EC provider; the geo and device declarations were decorative — worse than absent, because they read as a gate and are not one. This spec resolves that by **not having** the method on those traits -(§4). Geo and device providers are governed by explicit operator selection, -the capability checks of §6, and the permission model's vocabulary rule: if -a future vocabulary adds a purpose covering geolocation or fingerprinting, -gating those providers will require a two-phase resolution design specified -at that time (permission model spec §7). +(§4), with the two rationales above in place of the pretense. ## 6. Selection, validation, and failure modes @@ -187,6 +222,37 @@ Unknown fields inside every provider config block are rejected struct already has it, but PR #838 shipped `EcProviders`, `DeviceConfig`, and `GeoConfig` without it, so a typo like `providr` was silently ignored). +### 6.1 Provider switching: active writer, legacy readers + +Switching `[ec] provider` must not strand the identities the previous +provider minted: with only the selected provider recognizing cookies, an +`hmac` → vendor switch turns every existing cookie into "absent", orphans +its graph row, and — worst — makes a later opt-out unable to tombstone it. +The contract: + +- `[ec] provider` names the **active writer**: the only provider that + mints. +- `[ec] legacy_providers = ["hmac"]` (optional list) names **legacy + readers**: providers consulted, in order, for parse, graph lookup, and + tombstoning when the active writer does not recognize a value. Legacy + readers never mint. Each listed key must have its `[ec.providers.]` + block, validated like the active one (§6 table). +- A cookie recognized by a legacy reader is a live identity for + read/withdrawal purposes; whether it is transparently re-minted under the + active writer is a per-deployment choice + (`[ec] rewrite_legacy = true|false`), and re-minting is subject to the + full minting gate of §5. +- Retiring a legacy reader is the explicit end of those identities: + the migration guide documents the cleanup procedure (migration spec §6). +- Tests: switch active provider → request with old cookie → identity still + resolves and a withdrawal tombstones it; old cookie with no matching + legacy reader → treated as absent and **never egresses**. + +Cluster degradation config (referenced from §3): when the active writer +lacks the cluster capability, `[ec] cluster_fallback = "allow" | "deny"` +decides whether KV-backed writes gated on cluster trust proceed; there is +no implicit default — the operator chooses. + ## 7. Composition root and adapter parity Provider construction happens in exactly one place per concern @@ -242,11 +308,14 @@ prominent in release notes: ## 10. Testing strategy -- Provider conformance suite (§3 invariant) run against every shipped - provider. -- EC permission-enforcement tests (§5). +- Provider conformance suite (§3 invariant, deterministic entropy) run + against every shipped provider. +- EC minting-gate tests (§5), including the spy-provider case: permission + unset → `generate` never called, withdrawal still tombstones. +- Legacy-reader tests (§6.1): provider switch → old cookie resolves and + withdraws; unmatched old cookie never egresses. - Settings validation tests for every row of the §6 table, including the - block-without-selector rejection. + block-without-selector rejection and the `legacy_providers` rules. - Parity suite additions of §7. - Unit tests inside each provider crate; crates with no native-target tests still get clippy coverage via the alias wiring of §8. @@ -274,3 +343,15 @@ prominent in release notes: Steps 1–4 are independently reviewable, behavior-preserving, and do not depend on the permission model: the EC gate keeps its current jurisdiction logic until the permission model PR replaces it. + +## 12. Divergences from issue #778 + +This spec supersedes #778 on the following points; the issue is updated to +reference this spec when the PR merges, so implementation has one +acceptance contract: + +| #778 says | This spec says | Why | +| ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | +| Identifier comparison is a provider operation (`keys_equal`) | Comparison is structural: `parse` canonicalizes, so equivalent envelopes become the same identifier and graph key (§3) | Satisfies the same requirement with no comparison method to leave uncalled | +| A provider can return response headers | Dropped (§4) | Empty in every built-in in PR #838, plumbed through three layers with no consumer; returns with the first feature that needs it | +| One built-in provider (HMAC) preserving today's behavior | Same, plus explicit legacy-reader semantics for later switches (§6.1) | Switching was unspecified in #778 and stranded identities in the #838 shape | diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 5e7a327c1..88b2d99a3 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -32,18 +32,19 @@ and whether that is a preservation or a declared change. **Silent changes are defects.** PR #838 changed six of these without declaring any; each was discoverable only because a deleted test had pinned the old behavior. -| # | Decision (today) | After epic | Status | -| --- | --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | -| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | -| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | -| 3 | US-state request, no signals at all → no EC (fail-closed) | Policy decision: the shipped `US` baseline decides. If the baseline grants `store-on-device` without a signal, that is a **declared change** requiring sign-off in the policy review, with rationale in the policy itself | Declared change (if made) | -| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | -| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | -| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | -| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | -| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | -| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | -| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | +| # | Decision (today) | After epic | Status | +| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Policy decision: the shipped `US` baseline decides. If the baseline grants `store-on-device` without a signal, that is a **declared change** requiring sign-off in the policy review, with rationale in the policy itself | Declared change (if made) | +| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | +| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | +| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | +| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | +| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | +| 11 | Raw EC egress (OpenRTB `user.id`, derived request IDs, page bids, proxy/click/Testlight forwarding, identify, pull/batch sync) is gated by the jurisdiction gate today | Gated by the egress inventory (permission spec §7): bidstream egress requires both purposes, first-party identity operations require `store-on-device`, revocation exempt — at least as strict as today for every inventoried path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -109,16 +110,22 @@ Requirements: and `provider = "client-fixed"` are unknown keys and rejected like any other, so a config written against the PR #838 example cannot silently select a provider that no longer exists. -4. **The example config ships the migrated happy path**, uncommented: +4. **Provider switches go through legacy readers.** Changing + `[ec] provider` on a deployment with live identities requires listing + the outgoing provider in `[ec] legacy_providers` (providers spec §6.1) + so existing cookies keep resolving and stay withdrawable; the guide + documents the switch sequence and the retirement/cleanup step that ends + it. +5. **The example config ships the migrated happy path**, uncommented: `provider = "hmac"` with its block, `[geo] default_country`, and (for Fastly) the behavior-preserving `[device] provider = "fastly"` and `[geo] provider = "platform"` lines present with a comment stating what removing them changes. PR #838's example shipped the passphrase block uncommented with the selector commented out — steering operators directly into the silent-stateless state. -5. Every misconfiguration in the providers spec §6 table fails at +6. Every misconfiguration in the providers spec §6 table fails at **startup**. Request-time failure for a configuration error is a defect. -6. Config-store payload validation (`ts config push`) applies the same +7. Config-store payload validation (`ts config push`) applies the same rules — including `[permissions]` policy validation (permission spec §3.3) — so a bad config is rejected at push time, before any instance restarts into it. @@ -129,6 +136,10 @@ The migration guide (a new `docs/guide/` page, linked from the release notes) gives one copy-pasteable recipe per adapter for "keep exactly today's behavior": +The recipe is the **complete recommended policy table from +`trusted-server.example.toml`** — the full GDPR/UK/US jurisdiction rules, +copied, not referenced by omission — plus this delta: + ```toml [ec] provider = "hmac" @@ -140,22 +151,38 @@ provider = "fastly" # Fastly deployments: preserves the JA4 bot gate [geo] provider = "platform" # preserves per-request jurisdiction detection -default_country = "FR" # used only when the host lookup fails (fail-closed) +default_country = "FR" # used only when the host lookup fails (fail-closed + # because FR resolves to the gdpr-eu rule below) + +# ... the full example policy table goes here: gdpr-eu / gdpr-uk / +# us-opt-out groups and their country rules, verbatim ... -# Preserves today's treatment of countries outside the regulation lists -# ("non-regulated" → identity allowed). Omit this section to adopt the -# protective default instead: signal required worldwide (§2 row 7). +# Delta vs. the protective example: preserve today's treatment of countries +# outside the regulation lists ("non-regulated" → identity allowed). Keep +# the example's `default = "gdpr-eu"` instead to require a signal worldwide +# (§2 row 7). [permissions.groups.non-regulated] +regime = "none" default = "granted" [permissions.rules] default = "non-regulated" ``` -and separately documents the neutral configuration and what it does _not_ do. -The guide states explicitly that `default_country` alone does not replace geo -lookup, why the no-geo combination requires the explicit acknowledgment flag -(permission spec §5.3), and that no recipe preserves row 8 of §2 — the +A partial policy is a trap the first draft of this spec fell into: a +`[permissions]` section containing **only** the permissive +default — with no GDPR/US rules — sends _every_ jurisdiction, France +included, to the permissive fallback, because `default_country` selects a +rule like any other country and finds none. The recipe therefore always +carries the full table, and CI pins it: **the exact documented recipe text +is a fixture**, loaded and run through the complete §4.1/§4.2 decision +matrix of the permission spec, asserting per-jurisdiction outcomes match +the pre-epic gate for every preservation row of §2. + +The guide separately documents the neutral configuration and what it does +_not_ do, states explicitly that `default_country` alone does not replace +geo lookup, why the no-geo combination requires the explicit acknowledgment +flag (permission spec §5.3), and that no recipe preserves row 8 of §2 — the global honoring of opt-out signals is unconditional. ## 6. Rollout sequence and observability @@ -174,10 +201,14 @@ global honoring of opt-out signals is unconditional. 4. Rollback is config-only where possible: reverting to the previous config version restores the previous behavior on the previous binary. The one irreversible artifact is withdrawal tombstones — which is why the - withdrawal triggers (permission spec §4.2) are exhaustive and why §2 - rows 6 and 8 call out tombstoning explicitly. Cleanup of identities - minted before a policy tightening (permission spec §4.2 trigger 3) is an - operational action documented in the guide, not an automatic one. + withdrawal triggers (permission spec §4.2) are exhaustive, why partial + withdrawal failure has an explicit tombstones-first, browser-retries + contract (permission spec §4.3), and why §2 rows 6 and 8 call out + tombstoning explicitly. Two operational procedures are documented in the + guide, not automated: cleanup of identities minted before a policy + tightening (permission spec §4.2 trigger 3), and retirement of a legacy + reader after a provider switch (providers spec §6.1), which is the + deliberate end of the identities only that reader can resolve. ## 7. Documentation deliverables From 5c8c2e8930391ac0c834fb69c545bdf4de595a1d Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 31 Jul 2026 07:36:31 -0700 Subject: [PATCH 04/24] Address second review: close permission-algebra, egress, durability, and rollout gaps Blocking findings from the second review of PR #986: - Signal taxonomy gains a grant-signal class (TCF consent, explicit GPP non-opt-out, US Privacy present-and-not-opted-out including N/A) so a requires_signal US rule reproduces today's no-signal-blocks / explicit-non-opt-out-grants behavior, which the two-class model could not express; migration matrix gains rows 3a-3c and the example US group changes to requires_signal. - Auction dispatch gets a normative regime matrix (gdpr / us-privacy / none across consent, opt-out, malformed, expired, absent states), the compiled fallback gains regime = gdpr, and blocked dispatch means no outbound request at all. - The normalization matrix now states outcomes instead of subjects: restrictive/permissive synthesize per purpose, newest selects whole records, expired records are absent entirely, valid-beats-malformed within a family, KV fallback is live-wins with TTL-bounded staleness and an exempt consent-state lookup, mirror mode loses to request records, and GPP fields are enumerated per section. - The egress inventory is a concrete path -> permission table: proxy / click / Testlight forwarding assigned (both purposes, declared as new hardening in split row 11a/11b since those paths are ungated today), identify and pull/batch sync classified as partner exchange (both purposes), and S2S sync authorized by stored provider/version-tagged provenance re-validated against current policy. - Withdrawal drops the false atomicity claim: revocation families are idempotent independent writes, readers fail closed on any present member, and fault-injection covers the Nth-write failure. - Equivalence is provider-declared via fixtures (hmac: hex prefix case-insensitive, suffix case-preserved) instead of a universal case rule; the legacy HMAC grammar is formally reserved as the hmac namespace so verbatim row keys and provider namespacing coexist; global cookie-safe identifier bounds added; non-cluster providers get defined dedupe and redaction. - Legacy readers get full semantics: first-match parse with namespace-overlap validation, recognizing provider's permissions govern, provider/version-tagged provenance, transactional linking rewrite with dual revocation, and provider = "none" as an explicit stateless state that keeps revoke-only legacy readers. - Graph store required at startup when any provider can mint or read; a minted identity is not active until its row commits; a runtime failure matrix covers provider, graph, cluster, rewrite, and geo/device runtime failures. - Rollout gains a dual-read release (N+1 accepts both config shapes, N+2 rejects loudly) since no config is accepted by both current main and a rejecting binary; the preserving recipe becomes one committed valid TOML fixture (the prose delta reopened [permissions.rules], which is invalid TOML); metrics extended with retirement thresholds. - Resolve endpoint defines same-identity no-op / different-identity rejection and an atomic single-key reservation tying replay consumption to graph persistence. - Hook ordering becomes core -> integrations -> inviolable cache/privacy invariant pass (an appended cookie plus replaced public Cache-Control can no longer produce a shared-cacheable cookie response); generic ops reject Set-Cookie; per-integration operation limits and erroring-mutator semantics defined; every eligibility row tested. - Stale device-circularity wording removed from the permission spec; region-form default_country (US/CA) restored; the source-agnostic permission-source requirement of #777/#779 explicitly deferred in the divergence table; fail-closed and most-protective labels qualified with their stated exceptions. --- ...26-07-30-client-cycle-ec-resolve-design.md | 19 ++ ...integration-response-header-hook-design.md | 38 ++- .../2026-07-30-permission-model-design.md | 255 +++++++++++------- .../2026-07-30-pluggable-providers-design.md | 134 +++++++-- ...07-30-provider-migration-rollout-design.md | 123 +++++---- 5 files changed, 383 insertions(+), 186 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index e0a5529fe..88b1fa66b 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -93,6 +93,25 @@ Everything in this spec follows from that. the resulting identifier that keeps it cookie-safe and within the KV limits of the providers spec §3. Tests exercise the exact 413 boundary and the missing/false/chunked-length cases. +8. **Define behavior against an existing identity — no silent + replacement.** When the request already carries a recognized EC: + resolving to the **same** identity is an idempotent no-op (cookie + refreshed, same response); resolving to a **different** identity is + **rejected** — replacing a live identity via an unauthenticated POST is + identity takeover, and any legitimate re-identification flow (account + link, vendor migration) is an explicit linking design this spec does + not authorize (own open question, §7). +9. **Make replay consumption and graph persistence one idempotent + sequence.** Neither naive order works: consume-the-nonce-first makes a + subsequent graph failure unretryable (the token is spent, the identity + never existed); graph-first lets the losers of a replay race leave + residual rows. Required shape: consumption is an **atomic single-key + reservation** keyed by the payload's unique id, recording outcome; the + graph write happens under that reservation and is retried under the + same key; duplicate or racing requests observe the reservation and + receive the original outcome — no second identity, no spent-but-unused + token, no orphan row. Tests cover crash-between-steps and two + concurrent requests with the same payload. ## 4. Requirements on the page script diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index b91846222..1c6251288 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -42,13 +42,20 @@ mutators to the outbound response for HTML document responses it processed. processed documents. The call site lives in shared response-finalization code where one exists; where adapters finalize independently, each adapter gains the call and a test proving it. -- Mutators run **after** Trusted Server's own response-header handling - (EC Set-Cookie emission, EC header clearing, privacy headers) so a - mutation cannot be silently stripped by a later core pass. The ordering is - a fresh decision this spec makes — PR #838 never wired the hook, so there - is no existing insertion point to inherit; the implementer places the call - at the end of each adapter's response finalization, and the §4.3 tests pin - it there. +- **Ordering is three stages, and the last one is inviolable:** core + response-header handling (EC Set-Cookie emission, EC header clearing, + privacy headers) → integration operations → **final cache/privacy + invariant enforcement**, which no integration operation can override. + Running the hook dead-last would be wrong: current `main` deliberately + runs cookie-cache protection _after_ arbitrary header changes, stripping + surrogate caching and forcing private/no-store on any response that sets + a cookie — a hook applied after that recheck could combine an appended + `Set-Cookie` with a replaced public `Cache-Control` into a + **shared-cacheable cookie response**. The invariant pass therefore runs + after all mutations, unconditionally. Middle-stage placement also keeps + the earlier property: an integration mutation is not silently stripped + by ordinary core handling — only by the invariant pass, which logs the + downgrade it applies. ## 3. Collision policy @@ -70,6 +77,14 @@ mutators to the outbound response for HTML document responses it processed. header the origin set is a deliberate act, visible in the mutator's code. - Later registrations see earlier mutations (order = registration order, which is deterministic). +- **Operation-layer hygiene:** generic `append`/`replace` reject the + `Set-Cookie` header name outright — cookies go only through + `append_set_cookie`, so its validation cannot be bypassed by spelling + the header name in a generic op. Per-integration limits bound total + operations, added header count, and added header bytes; exceeding a + limit rejects the excess operations (logged, attributed), never the + response. A mutator that panics or errors is skipped in full — its + operations are all-or-nothing — and the response proceeds without it. ## 3a. Response eligibility — normative @@ -99,7 +114,14 @@ processed documents (§6). 3. Every adapter applies mutations on its outbound path, with a per-adapter route test asserting an integration-set header appears in the response. 4. A parity-suite case asserts identical mutation behavior across adapters. -5. Reserved-header and append/replace semantics covered by unit tests. +5. Reserved-surface, append/replace, operation-limit, and erroring-mutator + semantics covered by unit tests. +6. **Every row of the §3a eligibility matrix has a test** — streaming, + cache-hit, pass-through, redirect, error, and 304 each proven to run or + not run the hook — not merely one positive header test per adapter. +7. The cache/privacy invariant test: an integration appends a cookie and + replaces `Cache-Control` with a public/surrogate-cacheable value → the + final response is private/no-store with surrogate caching stripped. ## 5. Size and sequencing diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index a66037a7e..820426ab5 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -37,6 +37,12 @@ The set is resolved from three inputs: 3. **Signals** — the request's privacy signals: TCF, GPP, GPC, US Privacy (§4). +These are the initial sources. #777/#779 also envision publisher +interaction and external services as permission sources; that +source-interface is **explicitly deferred**, not silently dropped — §10 +records the divergence, and adding a source later means adding a grant- or +opt-out-class input to §4's taxonomy, not a new resolution algorithm. + Scope: the model governs decisions Trusted Server makes. Downstream RTB partners receive the full, unmodified regulatory context and make their own compliance decisions. @@ -56,10 +62,10 @@ compatibility. The initial vocabulary is therefore exactly: -| Identifier | TCF purpose | Enforcement points | -| ------------------------- | ----------- | -------------------------------------------------------------------- | -| `store-on-device` | 1 | EC provider execution; EC creation; withdrawal/tombstone eligibility | -| `select-personalised-ads` | 4 | EID transmission into the bidstream (jointly with `store-on-device`) | +| Identifier | TCF purpose | Enforcement points | +| ------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `store-on-device` | 1 | EC provider execution; EC creation; withdrawal/tombstone eligibility | +| `select-personalised-ads` | 4 | All bidstream and partner identity egress — raw EC in `user.id`, derived request IDs, page bids, EIDs, identify, pull/batch sync (jointly with `store-on-device`; the full path table is §7) | (The identifier strings are the IAB names verbatim, including their original spelling.) The extension procedure — add the signal mapping, add the @@ -97,8 +103,12 @@ git for the file, config-store versions for pushes — is the change log. **Compiled-in fallback:** when a config has no `[permissions]` section, a minimal compiled-in policy applies in which **every permission is -`requires_signal` for every jurisdiction** — the most protective posture. -Absence of policy is always safe; there is no fail-open default. +`requires_signal` for every jurisdiction**, with **`regime = "gdpr"`** so +auction dispatch (§7) is defined and maximally protective too. (This is +the most protective posture that still admits consent — `denied` would be +stricter but would make a signal-carrying deployment inoperable by +default; the distinction is stated, not glossed.) Absence of policy is +always safe; there is no fail-open default. ### 3.2 Format @@ -115,9 +125,13 @@ overrides. Each permission resolves to an **acquisition rule**: regime = "gdpr" default = "requires_signal" +# Opt-out regime, expressed as requires_signal: explicit non-opt-out +# values are grant-class signals (§4), so signal-carrying traffic is +# granted while no-signal traffic stays blocked — matching today's US +# behavior, which `granted` cannot express. [permissions.groups.us-opt-out] regime = "us-privacy" -default = "granted" +default = "requires_signal" [permissions.groups.non-regulated] regime = "none" @@ -175,7 +189,10 @@ Validation rejects: - an empty `[permissions]` section (ambiguous intent: an operator who wants the compiled-in fallback omits the section entirely); - duplicate rule keys under case-insensitive comparison (`FR` and `fr`); -- a `[geo] default_country` that is not an assigned ISO code; it is +- a `[geo] default_country` whose country part is not an assigned ISO + code; it accepts either a country (`FR`) or a country/region key + (`US/CA`) — PR #838 supported region defaults, and a no-geo, + single-state deployment must be able to select its state rule. It is canonicalized to uppercase, and startup logs which rule (or `rules.default`) it resolves to. @@ -210,7 +227,9 @@ inline, and ships it as the most protective baseline. ## 4. Signal precedence — normative -Signals are classified: +Signals are classified into three classes — a two-class model (TCF grant / +opt-out) cannot reproduce today's US behavior, where no-signal traffic is +blocked but an **explicit non-opt-out** value grants: - **Opt-out signals** (affirmative withdrawal): GPC header; GPP sections carrying a sale/sharing opt-out; US Privacy opt-out. Opt-out signals are @@ -220,8 +239,17 @@ Signals are classified: and ignore it for others based on IP evidence. (For jurisdictions outside US states this is a declared behavior change; migration spec §2 records it.) -- **Consent records**: a decodable TCF string (standalone or embedded in - GPP), which may grant or refuse individual purposes. +- **Grant signals** (affirmative permission): a decodable TCF record + consenting to the purpose; an **explicit GPP non-opt-out value** (e.g. + `sale_opt_out = false`); a **US Privacy string present and not opting + out** — including the "not applicable" flag, which today's tests pin as + allowing. Any grant signal satisfies a `requires_signal` baseline; this + is what lets a `requires_signal` US rule preserve today's "no signal → + block, explicit non-opt-out → allow" behavior, which neither `granted` + nor a TCF-only grant class could express. +- **Refusals**: a decodable TCF record refusing the purpose. A refusal is + neither a grant nor an opt-out — it blocks acquisition (precedence 3) + and withdraws only per §4.2. **Precedence, highest first:** @@ -243,8 +271,11 @@ Signals are classified: jurisdictions — the migration spec's matrix (row 6) records it. Refusal revokes new grants only; whether it also destroys existing identity is governed strictly by §4.2. -4. Consent record grant — a TCF record present and consenting grants it - (subject to 1–2). +4. Grant signal — any grant-class signal (TCF consent, explicit GPP + non-opt-out, present-and-not-opted-out US Privacy) grants the permission + (subject to 1–3: a coexisting TCF refusal beats a non-TCF grant signal, + matching today's US-state ordering where a present TCF record decides + before GPP/USP values are consulted). 5. No signal — the policy baseline decides: `granted` sets it, `requires_signal` leaves it unset. @@ -253,12 +284,12 @@ Signals are classified: For each enforced permission, with baseline _B_ ∈ {granted, requires_signal, denied}: -| Opt-out present | TCF present | TCF consents | Result | -| --------------- | ----------- | ------------ | ------------------------------------------------ | -| yes | — | — | **unset** (and withdrawal semantics apply, §4.2) | -| no | yes | no | unset (withdrawal per §4.2, trigger 2) | -| no | yes | yes | set, unless B = denied | -| no | no | — | set iff B = granted | +| Opt-out present | TCF refusal present | Grant signal present | Result | +| --------------- | ------------------- | -------------------- | ------------------------------------------------ | +| yes | — | — | **unset** (and withdrawal semantics apply, §4.2) | +| no | yes | — | unset (withdrawal per §4.2, trigger 2) | +| no | no | yes | set, unless B = denied | +| no | no | no | set iff B = granted | ### 4.2 Withdrawal vs. absence @@ -313,38 +344,45 @@ live graph identity with no browser handle pointing at it): browser setting, the TCF record lives in the CMP's storage), so the next request re-presents the signal and retries the whole withdrawal. No quarantine queue is needed; the browser is the retry queue. -- Identify, batch-sync, and pull-sync treat a tombstone as authoritative - revocation (as today); a row whose withdrawal is pending retry is simply - still live until the retry lands, and never partially withdrawn. -- Fault-injection tests cover: tombstone write fails → cookie untouched, - error logged; subsequent request with the same signal → withdrawal +- **Partial progress is explicit, not atomic.** The tombstone writes are a + **revocation family**: an enumerable, ordered set of independent KV + writes (cookie hash, active-EC hashes), each idempotent, with no + multi-key atomicity assumed — the current storage offers none, and the + spec does not pretend otherwise. A retry resumes the family from the + start (idempotent writes make re-writing completed members harmless). + Withdrawal is **complete** only when every family member is committed; + the cookie expires only then. +- **Reads fail closed on partial families**: every consumer (identify, + batch-sync, pull-sync, egress gates) treats an identity as revoked when + **any** member of its revocation family is present — a partially + withdrawn identity is unusable immediately, even before the family completes. +- Fault-injection tests cover failure at the **Nth** family write (not + only total failure): first write lands, second fails → cookie untouched, + identity already treated as revoked by readers, error logged; subsequent + request with the same signal → family completes and the cookie expires. -### 4.4 Signal normalization +### 4.4 Signal normalization — normative matrix §4's precedence operates on normalized inputs: one effective consent record and one effective opt-out state per request. The normalization layer is where today's real-world mess lives, and PR #838 collapsed it -silently. The implementation ships a **normalization matrix** — a -table-driven spec-and-test artifact — covering at minimum: - -- **Dual consent records**: standalone TCF cookie vs. GPP-embedded TCF, - including per-purpose disagreement, resolved per the existing configured - conflict modes (restrictive / permissive / newest). Each mode is either - preserved or explicitly retired in the migration matrix — not dropped. -- **Record expiry** and the persisted-KV consent fallback: when a stored - record substitutes for an absent live one, and how staleness is bounded. -- **Proxy/mirror mode** (CMP consent mirrored server-side): where the - mirrored state enters precedence. -- **Exact GPP fields**: which section fields constitute a sale/sharing/ - targeted-advertising opt-out, enumerated per supported section — "GPP - opt-out" is not a single bit. -- **Malformed-but-present records fail closed for acquisition**: a consent - record that is present but undecodable blocks grants (it does not - degrade to "absent", which under a `granted` baseline would turn garbage - into a grant — the fail-open path in both #838 and the first draft of - this spec). A malformed record never triggers withdrawal — destruction - requires an affirmative, decodable signal (§4.2). +silently. These are the outcomes — decided here, not delegated to the +implementation; each row marked **changed** also appears in the migration +matrix: + +| Input state | Effective record / outcome | Status | +| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | +| Standalone TCF and GPP-embedded TCF disagree per purpose, mode `restrictive` | Per-purpose **synthesis**: a purpose is consented only when **both** records consent (AND) | Preserved (mode semantics pinned against current tests) | +| Same, mode `permissive` | Per-purpose synthesis: consented when **either** record consents (OR) | Preserved (same pinning) | +| Same, mode `newest` | **Whole-record selection** by `Created` timestamp; tie → the GPP-embedded record | Preserved (same pinning) | +| Expired consent record | Treated as **absent entirely** — grants nothing, refuses nothing, withdraws nothing; the baseline applies. Under a `granted` baseline that means the grant stands: an expired refusal is not current evidence and must not revoke indefinitely | Preserved | +| One valid record + a second malformed record of the same family | The **valid record governs**; the malformed one is ignored with a `warn` log. Fail-closed-on-malformed (below) applies only when no valid record of that family exists | Decided here | +| Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | +| Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | Preserved, circularity resolved | +| Proxy/mirror mode (CMP state mirrored server-side) | Mirror-sourced record enters as a live record; when both the mirror and the request carry records, the **request's record wins** (closer to the user) | Decided here | +| GPP opt-out fields | Enumerated per supported section: the sale, sharing, and targeted-advertising opt-out fields each independently constitute an opt-out signal when set; **all present-and-false** constitutes a grant signal (§4); absent/N-A fields contribute nothing. The exact field list per section ID is an appendix of the implementation PR, reviewed against the GPP spec | Decided here | +| Malformed-but-present record, no valid record of that family | **Blocks grants** (fail-closed acquisition — it does not degrade to "absent", which under a `granted` baseline would turn garbage into a grant, the fail-open path in both #838 and the first draft of this spec). Never triggers withdrawal — destruction requires an affirmative, decodable signal (§4.2) | Changed (declared) | ## 5. Jurisdiction resolution @@ -415,63 +453,88 @@ migration story unresolvable (migration spec §2, rows 5 and 7). | No geo provider configured | `default_country` baseline, guarded by §5.3 | | Country resolved, no matching rule | Policy `rules.default` | | Region resolved, no region rule | Country rule | -| No `[permissions]` section | Compiled-in fallback: everything `requires_signal` | +| No `[permissions]` section | Compiled-in fallback: everything `requires_signal`, `regime = "gdpr"` | +| S2S sync request (no user signals) | Authorized by stored provenance re-validated against current policy (§7) | | Malformed policy | Rejected at config push / startup (§3.3) — never per request | | No `default_country` | Startup failure | | Undecodable TCF/GPP record (present but malformed) | Blocks grants (fail-closed acquisition, §4.4); never withdraws; opt-out signals still honored | | Signals contradict (opt-out + consent) | Opt-out wins (§4) | -The overall posture is **fail-closed**: every ambiguous state resolves to -the configured baseline or more restrictive, and the one configuration that -turns "no information" into a static jurisdiction assertion (§5.3) requires -an explicit operator acknowledgment to exist. +The intended posture is fail-closed, with its two exceptions stated rather +than glossed: geo lookup failure resolves to the configured default (§5.2's +declared, metered residual — permissive defaults make this path fail-open), +and the §5.3 static-jurisdiction configuration exists only behind an +explicit operator acknowledgment. Every other ambiguous state resolves to +the configured baseline or more restrictive. ## 7. Enforcement points Consumers of the resolved set in this epic: 1. **EC provider execution** (providers spec §5) — the provider's declared - `required_permissions()` must all be set. This gate applies to EC - providers only: geo and device providers execute **before** permission - resolution as its inputs, so gating them on its output would be - circular. Their governance is explicit selection, the capability checks - of providers spec §6, and §2's vocabulary rule — if a future vocabulary - adds a purpose covering geolocation or fingerprinting, gating those - providers will require a two-phase resolution that must be specified - then, not improvised. + `required_permissions()` must all be set for minting and identity use. + This gate applies to EC providers only. **Geo** is ungated because + gating it is circular — jurisdiction is an input to permission + resolution. **Device** is ungated by a different, deliberate decision + (it is _not_ a resolution input): its security-classification role must + run for traffic that has granted nothing, and operator selection is the + recorded authorization — providers spec §5 states the decision and its + boundary. If a future vocabulary adds a purpose covering geolocation or + fingerprinting, gating those providers will require a two-phase + resolution specified then, not improvised. 2. **EC lifecycle** — creation requires `store-on-device`; withdrawal per §4.2. Recognition, canonicalization, and revocation of an existing identifier are **never** permission-gated — they must run precisely when permissions are withdrawn (providers spec §5). -3. **Every raw-EC egress**, not only EIDs. The raw EC identifier leaves the - process today as OpenRTB `user.id`, inside derived auction request IDs, - on the page-bids path, on proxied/click/Testlight forwarding, through the - identify endpoint, through pull/batch sync, and via identity-graph - reads/writes. Gating EIDs alone (PR #838's shape) leaves the raw EC - reaching bidders when Purpose 1 is granted and Purpose 4 refused; worse, - #838's `ec_allowed` was **vacuously true with no provider configured** - (`is_none_or`), so an existing canonical cookie still escaped in - stateless mode. The contract: - - The implementation maintains an **egress inventory**: every code path - where a raw EC (or a value derived from it) leaves the process is - enumerated in one table, each mapped to its required permissions, with - a test per row. - - **Bidstream egress** (`user.id`, EC-derived request IDs, page bids, - EIDs) requires `store-on-device` ∧ `select-personalised-ads` — the raw - EC is identity in the bidstream and is gated exactly as EIDs are. - - **First-party identity operations** (identify, pull/batch sync, - graph reads/writes other than revocation) require `store-on-device`. - - **Revocation paths are exempt** — tombstoning must work when - permissions are unset. - - With **no EC provider configured**, identity use fails closed: a - cookie value present on the request never egresses anywhere; it is - never vacuously allowed. -4. **Bidstream EIDs** — transmission requires `store-on-device` ∧ - `select-personalised-ads` (subsumed by point 3's inventory; listed - separately because it is the one gate PR #838 had). -5. **Server-side auction dispatch** — gated on the explicit policy `regime` - class (§3.4), a first-class enforcement point, not inferred from purpose - flags. +3. **Every raw-EC egress and identity operation** — the concrete + inventory, normative per path (one test per row; a denylist check + proves no ungated egress exists): + + | Path | Required permissions | Notes | + | ---------------------------------------------------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | + | OpenRTB `user.id` | `store-on-device` ∧ `select-personalised-ads` | Raw EC is identity in the bidstream — gated exactly as EIDs. PR #838 gated only EIDs, leaving `user.id` reachable with Purpose 4 refused | + | EC-derived auction request IDs | both purposes | Derived values are identity | + | Page-bids path | both purposes | | + | Bidstream EIDs | both purposes | The one gate PR #838 had | + | Proxy / click / Testlight forwarding of the EC cookie or headers | both purposes | **New hardening, declared change** — these paths extract the raw cookie/header without today's jurisdiction gate (migration spec §2 row 11b) | + | Identify endpoint (partner-facing) | both purposes | Partner identity exchange, not a first-party lookup — decided here | + | Pull sync / batch sync (partner identity exchange) | both purposes | Authority source for S2S requests: stored provenance, below | + | Request-scoped graph reads/writes (non-revocation) | `store-on-device` | | + | Revocation paths (tombstones, withdrawal reads) | **exempt** | Must work when permissions are unset | + | Stored consent-state lookup (§4.4) | **exempt**, narrowly scoped | Determining `store-on-device` cannot require `store-on-device` | + + With **no EC provider configured**, identity use fails closed: a cookie + value present on the request never egresses anywhere — never vacuously + allowed (#838's `ec_allowed` was `is_none_or`, vacuously true with no + provider). + + **S2S authority (batch/pull sync).** A server-to-server request carries + no user signals, geo, or `EcContext` to resolve permissions from. Its + authority is the identity's **stored provenance**: a record written at + mint/update time carrying the resolved jurisdiction, regime, grant + basis, and provider/version (providers spec §6.1). A sync request + re-validates that provenance against the **current** policy revision: + if the stored jurisdiction now resolves to `denied` for the required + permission, the row is not updated and is flagged for the operational + cleanup of §4.2 trigger 3. Sync never mints authority of its own. + +4. **Server-side auction dispatch** — gated on the policy `regime` class, + normatively: + + | Regime | Dispatch rule | Preserves | + | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- | + | `gdpr` | Dispatch only with a decodable, unexpired TCF record consenting to Purpose 1. Malformed, expired, or absent record → **no bid request leaves** (no-bid response). | Today's GDPR/unknown arm | + | `us-privacy` | Dispatch proceeds in every signal state, including opt-out — the opt-out strips identity (rows above) but the contextual auction runs. | Today's US-state arm | + | `none` | Dispatch proceeds. | Today's non-regulated arm | + + The **compiled-in fallback policy has `regime = "gdpr"`** (§3.1) — the + no-policy posture must be the most protective for dispatch too, and a + regime-less fallback would leave dispatch undefined. When dispatch is + blocked, nothing leaves for that request: no PBS/APS call, no UA/IP/geo + forwarding to bidders. When dispatch proceeds, what the request may + carry is governed row-by-row by the egress inventory; the full + regulatory context (consent strings) is always forwarded so downstream + partners make their own decisions (§1). The client-cycle resolve endpoint (own spec, currently on hold) would be a further consumer if and when it proceeds. @@ -488,6 +551,13 @@ further consumer if and when it proceeds. configured conflict mode and the malformed-record rows. - The §7 raw-EC egress inventory: one test per inventoried egress proving the gate, plus a denylist-style check that no ungated egress exists. +- The §7 auction-dispatch matrix: every regime × signal state + (consent, opt-out, malformed, expired, absent), including the + no-policy fallback regime, asserting both the dispatch decision and + that a blocked dispatch emits no outbound request. +- The §7 S2S authority path: sync against stored provenance, including + the policy-tightened-to-denied case (no update, flagged for cleanup) + and the exempt consent-state lookup. - §4.3 fault-injection cases. - Policy validation tests for every §3.3 rejection, exercised through both acceptance paths (push-time and startup). @@ -513,8 +583,9 @@ This spec supersedes #779 on the following points; the issue is updated to reference this spec when the PR merges, so there is one acceptance contract, not two: -| #779 says | This spec says | Why | -| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- | -| Unmatched countries fall to `default_country` | Unmatched-but-resolved countries fall to the policy's `rules.default`; `default_country` covers only unresolved requests | The two states had different pre-epic behavior; collapsing them made migration unresolvable (§5.4) | -| The full TCF purpose vocabulary is modeled | Only enforced purposes appear (§2) | Nine inert purposes in a policy file are a compliance hazard, not forward compatibility | -| Policy is an embedded file | Policy is `[permissions]` in `trusted-server.toml` (§3.1) | Runtime config-store pipeline; validation at push time | +| #779 says | This spec says | Why | +| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | +| Unmatched countries fall to `default_country` | Unmatched-but-resolved countries fall to the policy's `rules.default`; `default_country` covers only unresolved requests | The two states had different pre-epic behavior; collapsing them made migration unresolvable (§5.4) | +| The full TCF purpose vocabulary is modeled | Only enforced purposes appear (§2) | Nine inert purposes in a policy file are a compliance hazard, not forward compatibility | +| Policy is an embedded file | Policy is `[permissions]` in `trusted-server.toml` (§3.1) | Runtime config-store pipeline; validation at push time | +| Permission sources are open-ended (#777: publisher interaction, external services may grant) | Sources are jurisdiction, policy, and the §4 signal taxonomy; a pluggable source interface is **deferred** (§1) | Shipping an interface with no second source repeats the inert-surface mistake; the extension path (a new §4 signal class) is defined instead | diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index 88035ea91..c4dc9cc37 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -94,26 +94,49 @@ An `EdgeCookieProvider` owns the **complete lifecycle** of the identifiers it mints. Every lifecycle operation core performs on an EC value MUST be routed through the selected provider: -| Lifecycle operation | Where core uses it today | Contract | -| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Mint** | EC generation on first eligible request | Provider returns the identifier (and only core writes the cookie). | -| **Parse / canonicalize** | Reading `ts-ec` back from the request; deciding `ec_was_present`; batch-sync ingestion | Provider parses a cookie value into its **canonical** identifier, or rejects it. Canonicalization is provider-owned: case variants and equivalent envelopes of the same identity (per #778) parse to the same canonical identifier. A value the selected provider does not recognize is treated as absent (but see §6.1 legacy readers). | -| **Canonical graph key** | KV identity-graph row reads/writes | The provider maps a canonical identifier to its graph key: stable, KV-safe (within KV length and character-set limits), collision-free across the provider's identifier space, and namespaced so two providers' key spaces cannot collide. Two equivalent envelopes of one identity map to one key — verbatim cookie bytes as the key would fork graph rows on canonicalization differences and discard today's batch-sync canonicalization. | -| **Cluster prefix** (optional capability) | IP-cluster sizing (`cluster_trust_threshold`, implemented as a **KV prefix listing**), pull-sync dedupe, log redaction | A provider declaring cluster support returns a prefix that is a **literal byte prefix of the canonical graph key** — the cluster count lists keys by prefix, so an independently derived hash that is not an actual key prefix silently reports the wrong cluster size. The prefix deliberately collides across identifiers minted from the same client evidence. A provider without the capability declares so, and cluster-dependent gating follows a configured degradation policy (treat cluster size as unknown, with the KV-write decision that implies made explicit in config) instead of counting garbage. | -| **Tombstone** | Withdrawal: expiring the cookie and writing revocation markers | The identifiers eligible for tombstoning are exactly those the provider parses — never a shape-gated subset. | +| Lifecycle operation | Where core uses it today | Contract | +| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Mint** | EC generation on first eligible request | Provider returns the identifier (and only core writes the cookie). | +| **Parse / canonicalize** | Reading `ts-ec` back from the request; deciding `ec_was_present`; batch-sync ingestion | Provider parses a cookie value into its **canonical** identifier, or rejects it. Canonicalization and **equivalence are provider-declared, never imposed globally**: each provider ships equivalence fixtures naming exactly which variants are the same identity — case sensitivity is provider-specific (signed/base64-style envelopes are case-sensitive; even the built-in HMAC id is case-insensitive only in its hex prefix, with a case-preserved suffix). Declared-equivalent values parse to the same canonical identifier (satisfying #778). A value the selected provider does not recognize is treated as absent (but see §6.1 legacy readers). | +| **Canonical graph key** | KV identity-graph row reads/writes | The provider maps a canonical identifier to its graph key: stable, KV-safe (within KV length and character-set limits), collision-free across the provider's identifier space, and namespaced so two providers' key spaces cannot collide. Two equivalent envelopes of one identity map to one key — verbatim cookie bytes as the key would fork graph rows on canonicalization differences and discard today's batch-sync canonicalization. | +| **Cluster prefix** (optional capability) | IP-cluster sizing (`cluster_trust_threshold`, implemented as a **KV prefix listing**), pull-sync dedupe, log redaction | A provider declaring cluster support returns a prefix that is a **literal byte prefix of the canonical graph key** — the cluster count lists keys by prefix, so an independently derived hash that is not an actual key prefix silently reports the wrong cluster size. The prefix deliberately collides across identifiers minted from the same client evidence. A provider without the capability declares so, and cluster-dependent gating follows a configured degradation policy (treat cluster size as unknown, with the KV-write decision that implies made explicit in config) instead of counting garbage. | +| **Tombstone** | Withdrawal: expiring the cookie and writing revocation markers | The identifiers eligible for tombstoning are exactly those the provider parses — never a shape-gated subset. | **Invariant:** for every provider `P` and every identifier `id` minted by `P`, -`P.parse` round-trips `id` (including its case variants and equivalent -envelopes, which all canonicalize to the same identifier and graph key); -where `P` declares cluster support, `cluster_prefix(id)` is a literal prefix -of `graph_key(id)` and is shared by identifiers minted from the same client -evidence; and a withdrawal request carrying `id` tombstones it. A conformance -test suite MUST assert this round-trip for every shipped provider — including -case-variant, equivalent-envelope, cross-provider key-namespace, and KV -length/charset cases — and the suite MUST be written so a future provider -crate can run it against its own implementation. Conformance tests inject -deterministic entropy; probabilistic assertions ("two random suffixes -differ") are not accepted. +`P.parse` round-trips `id` — including every variant `P`'s declared +equivalence fixtures name, all of which canonicalize to the same identifier +and graph key; where `P` declares cluster support, `cluster_prefix(id)` is a +literal prefix of `graph_key(id)` and is shared by identifiers minted from +the same client evidence; and a withdrawal request carrying `id` tombstones +it. A conformance test suite MUST assert this round-trip for every shipped +provider — driven by each provider's equivalence fixtures, plus +cross-provider key-namespace and KV length/charset cases — and the suite +MUST be written so a future provider crate can run it against its own +implementation. Conformance tests inject deterministic entropy; +probabilistic assertions ("two random suffixes differ") are not accepted. + +Three global rules sit above every provider: + +- **Identifier bounds.** A minted identifier obeys a global cookie-safe + alphabet (valid cookie-octets: no separators, whitespace, or control + characters) and a global maximum length — for the identifier itself, not + only the graph key — enforced by core at mint and at parse, so no + provider can emit a value the cookie layer or logs cannot carry. +- **Namespace reservation.** The legacy HMAC grammar `{64hex}.{6alnum}` is + formally **reserved as the `hmac` provider's namespace**. `hmac`'s graph + key is the identifier verbatim and its cluster prefix is the 64-hex + prefix, so every pre-epic row stays reachable and every prefix listing + intact (migration spec §3) — and **no other provider may mint + identifiers or produce graph keys matching that grammar**, which is what + makes verbatim-compatibility and provider-namespacing coexist. + Conformance fixtures include an existing pre-epic row (reachability) and + a prefix-listing case. For `hmac`, the equivalence fixtures pin: + uppercase/lowercase hex-prefix variants are equivalent; suffix case is + preserved and significant. +- **No-cluster behavior is still defined.** A provider without cluster + support deduplicates pull-sync by canonical graph key and redacts logs + with a fixed-length hash of the graph key; `cluster_fallback` (§6.1) + governs only the trust/write decision, not these. ## 4. Trait surface: minimalism rule @@ -147,8 +170,8 @@ pub trait EdgeCookieProvider { /// Mint an identifier from request evidence. fn generate(&self, input: &IdentityInput<'_>) -> Result>; /// Parse and canonicalize a cookie value into this provider's - /// identifier; None when unrecognized. Equivalent envelopes and case - /// variants canonicalize to the same identifier. + /// identifier; None when unrecognized. Values the provider's declared + /// equivalence fixtures name as equivalent canonicalize identically. fn parse(&self, value: &str) -> Option; /// Canonical KV graph key for a parsed identifier. fn graph_key(&self, id: &EcId) -> GraphKey; @@ -176,6 +199,15 @@ spy-provider test pins the split: with `store-on-device` unset, `generate` is never called while a withdrawal request still parses the cookie and writes tombstones. +**A generated identity is not active until its graph row commits.** No +cookie write, no egress, no auction use may observe a minted identifier +before its graph row (with provenance, §6.1) has committed — PR #838 let a +generated EC reach an auction before finalization refused the cookie, +producing an identity that existed for one request and nowhere else. The +normative order is: gate → `generate` → graph-row commit → cookie write → +eligible for egress. A graph-commit failure means the mint never happened: +no cookie, no egress, error logged, the next request retries. + The gate applies to EC providers **only**. Geo and device are ungated for two _different_ reasons, stated separately because only one of them is structural: @@ -209,13 +241,15 @@ one. This spec resolves that by **not having** the method on those traits All validation happens at **settings construction** — a misconfiguration is a startup error, never a request-time error and never a silent behavior change. -| Configuration state | Behavior | -| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `provider` names an unknown key | Startup error listing valid keys. | -| `provider` set, its `[ec.providers.]` block missing | Startup error. | -| `[ec.providers.]` block present, `provider` unset | **Startup error.** (In PR #838 this silently ran stateless — the half-migrated config becomes a production identity outage detected by revenue drop. Rejecting it is the fix.) An operator who genuinely wants stateless deletes the block. | -| `provider` set to an implementation the running adapter cannot satisfy (e.g. a provider requiring host TLS fingerprints on an adapter that has none) | Startup error at adapter wiring time. Adapters declare their host capabilities to the composition root; the root checks the selected provider's needs against them **once**, at startup — not per request. | -| No `provider`, no providers block | Valid: the neutral default for that concern. | +| Configuration state | Behavior | +| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `provider` names an unknown key | Startup error listing valid keys. | +| `provider` set, its `[ec.providers.]` block missing | Startup error. | +| `[ec.providers.]` block present, `provider` unset | **Startup error.** (In PR #838 this silently ran stateless — the half-migrated config becomes a production identity outage detected by revenue drop. Rejecting it is the fix.) An operator who genuinely wants stateless deletes the block. | +| `provider` set to an implementation the running adapter cannot satisfy (e.g. a provider requiring host TLS fingerprints on an adapter that has none) | Startup error at adapter wiring time. Adapters declare their host capabilities to the composition root; the root checks the selected provider's needs against them **once**, at startup — not per request. | +| No `provider`, no providers block | Valid: the neutral default for that concern. | +| `provider = "none"` (explicit stateless) | Valid, and the only way to combine statelessness with `legacy_providers`: minting stops, legacy readers keep existing identities resolvable and **withdrawable** (§6.1). Without this state, `hmac` → stateless would strand every live row in revoke-proof limbo. | +| A minting provider (or any `legacy_providers`) configured, but no identity-graph store configured or openable | **Startup error.** The lifecycle contract assumes graph persistence (§5); discovering its absence at first mint would be a request-time config failure, which this table exists to forbid. | Unknown fields inside every provider config block are rejected (`deny_unknown_fields` on all new settings structs — the pre-existing `Ec` @@ -237,22 +271,64 @@ The contract: tombstoning when the active writer does not recognize a value. Legacy readers never mint. Each listed key must have its `[ec.providers.]` block, validated like the active one (§6 table). +- **Parse order and ambiguity.** The active writer parses first; the first + match wins. Overlapping recognition is not resolved at request time but + **forbidden at startup**: provider namespaces (§3) may not overlap, and + configuring an active/legacy pair whose grammars intersect is a + validation error. +- **The recognizing provider governs.** A legacy-owned identity is gated + by the **legacy provider's** `required_permissions()` for identity use — + the provider that minted under a declared data-use contract is the one + whose contract applies. +- **Provenance is provider- and version-tagged.** Every graph row carries + the minting provider id and its configuration version (this is the same + provenance record the S2S sync authority reads, permission model spec + §7). Same-provider key/passphrase rotation is a version entry, not a + provider switch: parse consults all configured versions of the active + provider. - A cookie recognized by a legacy reader is a live identity for read/withdrawal purposes; whether it is transparently re-minted under the active writer is a per-deployment choice (`[ec] rewrite_legacy = true|false`), and re-minting is subject to the full minting gate of §5. +- **Rewrite is transactional and linking, not fire-and-forget.** Order: + new row commits first, carrying a link to the old row and a copy of the + old row's consent metadata and partner mappings; only then does the + cookie swap; the old row is tombstoned (or link-retired) only after the + new row and cookie are in place. An interrupted rewrite leaves the old + cookie valid and simply retries — no state in which neither identity + works. **Withdrawal of either linked row tombstones both.** - Retiring a legacy reader is the explicit end of those identities: the migration guide documents the cleanup procedure (migration spec §6). - Tests: switch active provider → request with old cookie → identity still - resolves and a withdrawal tombstones it; old cookie with no matching - legacy reader → treated as absent and **never egresses**. + resolves and a withdrawal tombstones it (both linked rows when + rewritten); old cookie with no matching legacy reader → treated as + absent and **never egresses**; interrupted rewrite → old cookie still + live, retry completes; `provider = "none"` + legacy reader → no mints, + withdrawal still works. Cluster degradation config (referenced from §3): when the active writer lacks the cluster capability, `[ec] cluster_fallback = "allow" | "deny"` decides whether KV-backed writes gated on cluster trust proceed; there is no implicit default — the operator chooses. +### 6.2 Runtime failure matrix — normative + +Startup validation (§6) covers configuration; this covers what happens +when a healthy configuration meets an unhealthy runtime. Every row logs at +`error` with a metric; none is silent: + +| Failure | Behavior | +| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | +| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | +| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | +| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | +| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | +| Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | +| Legacy rewrite fails mid-flight | Old cookie remains live; rewrite retries (§6.1) | +| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | +| Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | + ## 7. Composition root and adapter parity Provider construction happens in exactly one place per concern diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 88b2d99a3..2ad71526a 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -32,19 +32,23 @@ and whether that is a preservation or a declared change. **Silent changes are defects.** PR #838 changed six of these without declaring any; each was discoverable only because a deleted test had pinned the old behavior. -| # | Decision (today) | After epic | Status | -| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | -| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | -| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | -| 3 | US-state request, no signals at all → no EC (fail-closed) | Policy decision: the shipped `US` baseline decides. If the baseline grants `store-on-device` without a signal, that is a **declared change** requiring sign-off in the policy review, with rationale in the policy itself | Declared change (if made) | -| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | -| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | -| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | -| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | -| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | -| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | -| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | -| 11 | Raw EC egress (OpenRTB `user.id`, derived request IDs, page bids, proxy/click/Testlight forwarding, identify, pull/batch sync) is gated by the jurisdiction gate today | Gated by the egress inventory (permission spec §7): bidstream egress requires both purposes, first-party identity operations require `store-on-device`, revocation exempt — at least as strict as today for every inventoried path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | +| # | Decision (today) | After epic | Status | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | +| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | +| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | +| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | +| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | +| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | +| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | +| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | +| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | +| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull/batch sync) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | +| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -94,12 +98,22 @@ passphrase = "example-passphrase" Requirements: -1. **Old key fails loud.** `[ec] passphrase` is rejected at startup with a - message naming the new location — not a generic unknown-field error. - Implementation note: `Ec` already carries `deny_unknown_fields`, which - would reject the key generically; producing the actionable message means - keeping a deprecated `passphrase` field whose presence triggers the - custom error. +1. **The transition has a dual-read release; loud rejection comes one + release later.** Today's binary _requires_ `[ec] passphrase` and — via + `deny_unknown_fields` — _rejects_ `[ec] provider` and + `[ec.providers.*]`; a binary that rejects the old shape outright would + mean **no config both binaries accept**, and a config-store fleet + cannot flip config and binaries atomically. Sequence: + - **Release N+1 (dual-read):** accepts the old shape (mapping + `[ec] passphrase` to the `hmac` provider internally, logging a + deprecation warning per startup) _and_ the new shape. Fleet rolls + binaries to N+1 with config unchanged; then config flips to the new + shape via `ts config push`; either order is safe at every instant. + - **Release N+2:** rejects `[ec] passphrase` at startup with a message + naming the new location — not a generic unknown-field error + (implementation note: producing the actionable message means keeping + a deprecated `passphrase` field whose presence triggers the custom + error). 2. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no `provider = "hmac"` selector is a startup error (providers spec §6). In PR #838 this configuration — the exact state an operator following the @@ -136,48 +150,36 @@ The migration guide (a new `docs/guide/` page, linked from the release notes) gives one copy-pasteable recipe per adapter for "keep exactly today's behavior": -The recipe is the **complete recommended policy table from -`trusted-server.example.toml`** — the full GDPR/UK/US jurisdiction rules, -copied, not referenced by omission — plus this delta: - -```toml -[ec] -provider = "hmac" -[ec.providers.hmac] -passphrase = "" - -[device] -provider = "fastly" # Fastly deployments: preserves the JA4 bot gate - -[geo] -provider = "platform" # preserves per-request jurisdiction detection -default_country = "FR" # used only when the host lookup fails (fail-closed - # because FR resolves to the gdpr-eu rule below) - -# ... the full example policy table goes here: gdpr-eu / gdpr-uk / -# us-opt-out groups and their country rules, verbatim ... - -# Delta vs. the protective example: preserve today's treatment of countries -# outside the regulation lists ("non-regulated" → identity allowed). Keep -# the example's `default = "gdpr-eu"` instead to require a signal worldwide -# (§2 row 7). -[permissions.groups.non-regulated] -regime = "none" -default = "granted" - -[permissions.rules] -default = "non-regulated" -``` +The recipe is **one complete, valid TOML fixture, committed to the +repository** (e.g. `docs/guide/fixtures/migration-preserving.toml`) and +included in the guide verbatim — never described as a textual delta +against the example file. (An earlier draft said "copy the example table, +then set `[permissions.rules] default`" — but the copied table already +declares `[permissions.rules]`, and reopening a TOML table is a parse +error; a prose delta cannot be validated, a committed fixture can.) The +fixture contains, in one document: + +- `[ec] provider = "hmac"` with its passphrase block; +- `[device] provider = "fastly"` (Fastly deployments: preserves the JA4 + bot gate); +- `[geo] provider = "platform"` and `default_country = "FR"` (per-request + jurisdiction detection preserved; the default is fail-closed because FR + resolves to the `gdpr-eu` rule); +- the full `gdpr-eu` / `gdpr-uk` / `us-opt-out` groups and country rules + from the example policy (US as `requires_signal` with the grant-signal + class — §2 rows 3–3b), plus the `non-regulated` group with + `rules.default = "non-regulated"` (row 7). Operators who prefer the + protective worldwide default use the example file itself instead. A partial policy is a trap the first draft of this spec fell into: a `[permissions]` section containing **only** the permissive default — with no GDPR/US rules — sends _every_ jurisdiction, France included, to the permissive fallback, because `default_country` selects a -rule like any other country and finds none. The recipe therefore always -carries the full table, and CI pins it: **the exact documented recipe text -is a fixture**, loaded and run through the complete §4.1/§4.2 decision -matrix of the permission spec, asserting per-jurisdiction outcomes match -the pre-epic gate for every preservation row of §2. +rule like any other country and finds none. The committed fixture is +therefore always complete, and CI pins it: **the fixture file itself** is +loaded and run through the complete §4.1/§4.2 decision matrix of the +permission spec, asserting per-jurisdiction outcomes match the pre-epic +gate for every preservation row of §2. The guide separately documents the neutral configuration and what it does _not_ do, states explicitly that `default_country` alone does not replace @@ -194,7 +196,14 @@ global honoring of opt-out signals is unconditional. 2. Before/after deploy, operators watch **EC issuance rate** and EID attachment rate; the migration guide names these as the canary metrics, because the failure mode of a bad migration is a silent drop to zero (or a - silent grant to everyone), not an error rate. + silent grant to everyone), not an error rate. The full metric set, each + with a stated healthy range: geo lookup-failure/fallback rate (permission + spec §5.2), raw-egress denials by path, tombstone family retries, + legacy-reader hit rate, rewrite failures, and cluster-fallback + engagements. Two of these carry thresholds, not just ranges: + legacy-reader hits trending to ~zero is the **retirement-readiness** + signal for a legacy provider, and a nonzero rewrite-failure rate blocks + retirement outright. 3. Startup logs always print: selected provider per concern, whether geo is live, the effective default baseline, and the count of granted-without- signal permissions. One line, greppable, stable format. From 572b104c796928d8abe645c0eb13e622574fada5 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 31 Jul 2026 08:11:36 -0700 Subject: [PATCH 05/24] Address third review: regime-scoped grants, storage migration, and lifecycle repairs Blocking findings from the third review of PR #986: - Grant evidence is now regime- and permission-scoped: gdpr rules accept only TCF consent for the specific purpose, us-privacy accepts TCF or explicit GPP/USP non-opt-out, none accepts any grant class - closing the hole where sale_opt_out=false in France would have authorized identity and partner egress with no TCF. Opt-outs and refusals stay regime-agnostic. - The graph schema change gets an expand-contract rollout: reader/ preserver release (unknown fields preserved through read-modify-write), fleet-convergence gate, then writer activation, with schema versions, lazy backfill, and mixed-version tests. - S2S batch-sync authority is a full recompute of both permissions from stored per-permission, time-bounded evidence (grant basis, timestamp, jurisdiction, policy revision, provider/version) - failing closed on denied, tightened baselines without acceptable stored evidence, expired evidence, or regime-rejected grant sources. Legacy pre-epic rows are hmac-v0 with no grant evidence and fail closed until lazily backfilled. - Pull sync split from batch sync: pull is browser-request-scoped and keeps using the live P1/P4 decision plus revocation state; only batch is provenance-authorized, and its gate is declared hardening (new matrix row 11c; row 11a corrected). - Withdrawal centers on a family revocation record: a stable family ID in every member row, one record written first that is simultaneously the durable intent, the sibling-discovery mechanism, and the fail-closed marker; member tombstones become cleanup; degraded-graph mode fails S2S closed while writes fail; the healthy-graph residual is declared. - Legacy rewrite is confirmed by presentation: both linked rows stay live until a later request presents the new cookie (the server cannot observe Set-Cookie acceptance); linked rows share the revocation family. - The normalization matrix now preserves actual current semantics: whole-record selection by combined P1/P4 eligibility for restrictive/ permissive, LastUpdated with freshness threshold and restrictive tie-break for newest, proxy mode skips decoding, one-valid/one-expired row added, and GPP section fields enumerated normatively. - The auction matrix regains the raw-signal arm: a decodable TCF record applies the gdpr dispatch rule in every regime, so a P1 refusal on US or non-regulated traffic still blocks dispatch. - Provider namespaces become declarative descriptors core can prove pairwise disjoint at startup (opaque parse cannot be); version rotation gets a schema (versions entries, mint_version, newest-first parse, retirement rules). - Client resolve reservations get pending/committed/failed states, lease takeover, retention through token expiry, deterministic graph idempotency, adapter CAS capability - and duplicates never receive Set-Cookie unless the reservation is session-bound, closing the idempotent-replay fixation hole. - The hook invariant pass preserves any pre-hook private/no-store classification (cookieless personalized HTML cannot be made publicly cacheable) and strips CDN directives; panics are declared forbidden and fatal on wasm32-wasip1 (panic=abort - recovery was unimplementable); a cumulative final-response header budget with deterministic rejection order added. - Rollout ordering corrected to strictly reader-first with a convergence gate (the previous either-order claim was false against binaries that reject the new shape); mixed old/new config shapes rejected; rollback sequencing defined; one preserving fixture per adapter since device/geo selections are capability-gated. - Device-provider authorization reconciled with persisted use: new rows stop carrying fingerprint-derived buyer-facing fields (declared change); a field-level graph contract table is a required implementation deliverable. - Non-blockers folded in: adapter-capability matrix, assigned- subdivision region validation, retirement quiet period no shorter than max cookie/row lifetime plus skew, expanded telemetry, and the stale recognize/hashed/eligibility terms corrected. --- ...26-07-30-client-cycle-ec-resolve-design.md | 30 ++- ...integration-response-header-hook-design.md | 24 +- .../2026-07-30-permission-model-design.md | 206 ++++++++++-------- .../2026-07-30-pluggable-providers-design.md | 78 +++++-- ...07-30-provider-migration-rollout-design.md | 109 +++++---- 5 files changed, 295 insertions(+), 152 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index 88b1fa66b..faed2b32d 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -76,7 +76,7 @@ Everything in this spec follows from that. the corresponding graph row is written, mirroring the organic path. Graph unavailable → no cookie, same as organic generation. 4. **Round-trip through the lifecycle contract.** The identifier set here - must be recognized, hashed, and tombstonable by the selected provider + must be parseable, graph-keyed, and tombstonable by the selected provider (providers spec §3). The conformance suite runs against every client-cycle provider. 5. **Exist on every adapter.** Route registration goes through shared route @@ -102,16 +102,30 @@ Everything in this spec follows from that. link, vendor migration) is an explicit linking design this spec does not authorize (own open question, §7). 9. **Make replay consumption and graph persistence one idempotent - sequence.** Neither naive order works: consume-the-nonce-first makes a + sequence — without letting idempotency reinstall the identity + elsewhere.** Neither naive order works: consume-the-nonce-first makes a subsequent graph failure unretryable (the token is spent, the identity never existed); graph-first lets the losers of a replay race leave residual rows. Required shape: consumption is an **atomic single-key - reservation** keyed by the payload's unique id, recording outcome; the - graph write happens under that reservation and is retried under the - same key; duplicate or racing requests observe the reservation and - receive the original outcome — no second identity, no spent-but-unused - token, no orphan row. Tests cover crash-between-steps and two - concurrent requests with the same payload. + reservation** (CAS — an adapter capability the composition root checks, + providers spec §7) keyed by the payload's unique id, with explicit + states: `pending` → `committed` | `failed`. The graph write happens + under the reservation and is retried under the same key; a `pending` + reservation older than its **lease** may be taken over by a retry; + reservations are retained at least through the token's expiry; the + graph write is deterministic under the reservation key so a retry + converges on the same row. + + **A duplicate must never receive the cookie unless the reservation is + session-bound.** "Duplicates observe the recorded outcome" cannot mean + replaying `Set-Cookie` — that would hand a captured token's identity to + a second browser, recreating the fixation §2 exists to prevent. In + one-time mode without session binding, a duplicate gets a terminal + response with **no cookie**; only a requester that proves the original + session binding (the §3.2 nonce) may have the `Set-Cookie` re-emitted. + Tests cover crash-between-steps, lease takeover, two concurrent + requests with the same payload, and a duplicate from a second client + receiving no cookie. ## 4. Requirements on the page script diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 1c6251288..f82f15dfb 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -52,7 +52,14 @@ mutators to the outbound response for HTML document responses it processed. a cookie — a hook applied after that recheck could combine an appended `Set-Cookie` with a replaced public `Cache-Control` into a **shared-cacheable cookie response**. The invariant pass therefore runs - after all mutations, unconditionally. Middle-stage placement also keeps + after all mutations, unconditionally — and it enforces more than the + cookie rule: **any private/no-store classification core assigned before + the hook is preserved** (processed auction HTML is marked private even + when no cookie is emitted — today's final helper returns early without + `Set-Cookie`, so cookie-triggered enforcement alone would let an + integration make cookieless personalized HTML publicly cacheable), and + every CDN/surrogate cache directive is stripped from any response so + classified. Middle-stage placement also keeps the earlier property: an integration mutation is not silently stripped by ordinary core handling — only by the invariant pass, which logs the downgrade it applies. @@ -81,10 +88,19 @@ mutators to the outbound response for HTML document responses it processed. `Set-Cookie` header name outright — cookies go only through `append_set_cookie`, so its validation cannot be bypassed by spelling the header name in a generic op. Per-integration limits bound total - operations, added header count, and added header bytes; exceeding a - limit rejects the excess operations (logged, attributed), never the - response. A mutator that panics or errors is skipped in full — its + operations, added header count, and added header bytes, and a + **cumulative final-response budget** (total header count and bytes) + bounds the sum across integrations — enforced in registration order, so + which operations are rejected when the budget trips is deterministic. + Exceeding a limit rejects the excess operations (logged, attributed), + never the response. A mutator that returns an error is skipped in full — its operations are all-or-nothing — and the response proceeds without it. + **Panics are forbidden and fatal, not recoverable**: the primary target + (`wasm32-wasip1`) builds with `panic = "abort"`, so there is no unwind + boundary to catch at — a spec that promised panic recovery would be + unimplementable there. Mutators are infallible-by-construction or + return `Result`; a panic is a bug that takes the instance down, same as + anywhere else in the request path. ## 3a. Response eligibility — normative diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 820426ab5..f509e8bee 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -64,7 +64,7 @@ The initial vocabulary is therefore exactly: | Identifier | TCF purpose | Enforcement points | | ------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `store-on-device` | 1 | EC provider execution; EC creation; withdrawal/tombstone eligibility | +| `store-on-device` | 1 | EC provider execution; EC creation; input to the §4.2 withdrawal decision (revocation itself is never permission-gated, §7) | | `select-personalised-ads` | 4 | All bidstream and partner identity egress — raw EC in `user.id`, derived request IDs, page bids, EIDs, identify, pull/batch sync (jointly with `store-on-device`; the full path table is §7) | (The identifier strings are the IAB names verbatim, including their original @@ -178,7 +178,7 @@ Validation rejects: - rule keys whose country part is not in the embedded **assigned** ISO 3166-1 alpha-2 list (not merely `[A-Z]{2}` — an unassigned code is almost certainly a typo silently diverting a country to the fallback); - the region part matches `[A-Z0-9]{1,3}`. The `US/CA` slash form is the + the region part must be an assigned ISO 3166-2 subdivision of that country (not merely a shape check — `US/ZZ` would parse but can never match a request), unless the selected geo provider declares its own region vocabulary, in which case validation uses that declaration. The `US/CA` slash form is the house rule-key format corresponding to ISO 3166-2 `US-CA`; - references to permissions outside the enforced vocabulary (§2); - references to undefined groups, and groups missing the `regime` class; @@ -243,10 +243,27 @@ blocked but an **explicit non-opt-out** value grants: consenting to the purpose; an **explicit GPP non-opt-out value** (e.g. `sale_opt_out = false`); a **US Privacy string present and not opting out** — including the "not applicable" flag, which today's tests pin as - allowing. Any grant signal satisfies a `requires_signal` baseline; this - is what lets a `requires_signal` US rule preserve today's "no signal → - block, explicit non-opt-out → allow" behavior, which neither `granted` - nor a TCF-only grant class could express. + allowing. Grant signals are what let a `requires_signal` US rule + preserve today's "no signal → block, explicit non-opt-out → allow" + behavior, which neither `granted` nor a TCF-only grant class could + express. **Which grant evidence a rule accepts is regime- and + permission-scoped** — grant signals are NOT interchangeable across + regimes: + + | Regime of the resolved rule | Evidence accepted as a grant for a `requires_signal` permission | + | --------------------------- | --------------------------------------------------------------- | + | `gdpr` | **Only** a TCF record consenting to that specific purpose | + | `us-privacy` | TCF consent for the purpose, or an explicit GPP/USP non-opt-out | + | `none` | Any grant-class signal | + + Without this scoping, a US-style `sale_opt_out = false` would satisfy a + French `requires_signal` rule — no TCF, both purposes granted, EC minted, + partner egress authorized — contradicting the GDPR preservation row of + the migration matrix. Auction dispatch blocking separately would not + help; identity use would already be authorized. Opt-out signals and + refusals remain regime-agnostic (global), as before: scoping applies + only to what can _grant_, never to what can _revoke_. + - **Refusals**: a decodable TCF record refusing the purpose. A refusal is neither a grant nor an opt-out — it blocks acquisition (precedence 3) and withdraws only per §4.2. @@ -271,11 +288,11 @@ blocked but an **explicit non-opt-out** value grants: jurisdictions — the migration spec's matrix (row 6) records it. Refusal revokes new grants only; whether it also destroys existing identity is governed strictly by §4.2. -4. Grant signal — any grant-class signal (TCF consent, explicit GPP - non-opt-out, present-and-not-opted-out US Privacy) grants the permission - (subject to 1–3: a coexisting TCF refusal beats a non-TCF grant signal, - matching today's US-state ordering where a present TCF record decides - before GPP/USP values are consulted). +4. Grant signal — a grant-class signal **accepted by the resolved rule's + regime for that permission** (table above) grants it (subject to 1–3: a + coexisting TCF refusal beats a non-TCF grant signal, matching today's + US-state ordering where a present TCF record decides before GPP/USP + values are consulted). 5. No signal — the policy baseline decides: `granted` sets it, `requires_signal` leaves it unset. @@ -284,12 +301,12 @@ blocked but an **explicit non-opt-out** value grants: For each enforced permission, with baseline _B_ ∈ {granted, requires_signal, denied}: -| Opt-out present | TCF refusal present | Grant signal present | Result | -| --------------- | ------------------- | -------------------- | ------------------------------------------------ | -| yes | — | — | **unset** (and withdrawal semantics apply, §4.2) | -| no | yes | — | unset (withdrawal per §4.2, trigger 2) | -| no | no | yes | set, unless B = denied | -| no | no | no | set iff B = granted | +| Opt-out present | TCF refusal present | Accepted grant present (regime-scoped) | Result | +| --------------- | ------------------- | -------------------------------------- | ------------------------------------------------ | +| yes | — | — | **unset** (and withdrawal semantics apply, §4.2) | +| no | yes | — | unset (withdrawal per §4.2, trigger 2) | +| no | no | yes | set, unless B = denied | +| no | no | no | set iff B = granted | ### 4.2 Withdrawal vs. absence @@ -331,36 +348,39 @@ behavior had no unit test at all. ### 4.3 Withdrawal durability -Withdrawal is two writes — the KV tombstones and the cookie expiry — and -the contract for partial failure is explicit (PR #838 expired the cookie -first and logged-and-swallowed tombstone-write failures, which can leave a -live graph identity with no browser handle pointing at it): - -- **Order: tombstones first, cookie expiry second.** The cookie is expired - only after the tombstone writes succeed. -- **On tombstone-write failure, the cookie is left in place** and the - failure is logged at `error` with a metric. This is deliberately - self-healing: every withdrawal trigger is durable client-side (GPC is a - browser setting, the TCF record lives in the CMP's storage), so the next - request re-presents the signal and retries the whole withdrawal. No - quarantine queue is needed; the browser is the retry queue. -- **Partial progress is explicit, not atomic.** The tombstone writes are a - **revocation family**: an enumerable, ordered set of independent KV - writes (cookie hash, active-EC hashes), each idempotent, with no - multi-key atomicity assumed — the current storage offers none, and the - spec does not pretend otherwise. A retry resumes the family from the - start (idempotent writes make re-writing completed members harmless). - Withdrawal is **complete** only when every family member is committed; - the cookie expires only then. -- **Reads fail closed on partial families**: every consumer (identify, - batch-sync, pull-sync, egress gates) treats an identity as revoked when - **any** member of its revocation family is present — a partially - withdrawn identity is unusable immediately, even before the family - completes. -- Fault-injection tests cover failure at the **Nth** family write (not - only total failure): first write lands, second fails → cookie untouched, - identity already treated as revoked by readers, error logged; subsequent - request with the same signal → family completes and the cookie expires. +Withdrawal spans multiple KV writes and a cookie expiry; the contract for +partial failure is explicit (PR #838 expired the cookie first and +logged-and-swallowed tombstone-write failures, which can leave a live graph +identity with no browser handle pointing at it). The design centers on one +record that is simultaneously the durable intent, the discovery mechanism, +and the fail-closed marker: + +- **The family revocation record is written first.** Every identity carries + a stable **family ID**, minted with the identity and stored in every + member row (including rows linked by a legacy rewrite, providers spec + §6.1). Revocation writes one record keyed by the family ID. That single + write is the withdrawal: per-member tombstones are cleanup that follows, + idempotent and retried. +- **Every consumer checks the family record, not per-member tombstones.** + A reader arriving through any still-live member row finds the family ID + in the row and the revocation record under it — partial revocation is + discoverable from every member, and the record survives member-tombstone + replacement (which today discards the original row's identity and + metadata, making sibling discovery impossible). +- **The cookie expires only after the family record commits.** +- **If the family-record write itself fails, nothing durable exists** — + the cookie stays and the durable client-side signal (GPC, CMP-stored TCF) + retries the whole withdrawal on the next request. Two mitigations bound + the S2S residual in the meantime: while graph **writes are degraded** + (health signal), S2S partner egress and sync updates fail closed + (providers spec §6.2); and the failure is logged at `error` with a + metric feeding the operational repair path. The residual that remains — + a single failed write on an otherwise healthy graph, for a user who + never returns — is declared here, not hidden. +- Fault-injection tests cover: family-record write fails → cookie + untouched, S2S behavior per degraded mode, retry completes; member + tombstone N fails after the family record → identity already revoked for + every reader, cleanup retries; the same-signal retry path end to end. ### 4.4 Signal normalization — normative matrix @@ -371,18 +391,19 @@ silently. These are the outcomes — decided here, not delegated to the implementation; each row marked **changed** also appears in the migration matrix: -| Input state | Effective record / outcome | Status | -| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | -| Standalone TCF and GPP-embedded TCF disagree per purpose, mode `restrictive` | Per-purpose **synthesis**: a purpose is consented only when **both** records consent (AND) | Preserved (mode semantics pinned against current tests) | -| Same, mode `permissive` | Per-purpose synthesis: consented when **either** record consents (OR) | Preserved (same pinning) | -| Same, mode `newest` | **Whole-record selection** by `Created` timestamp; tie → the GPP-embedded record | Preserved (same pinning) | -| Expired consent record | Treated as **absent entirely** — grants nothing, refuses nothing, withdraws nothing; the baseline applies. Under a `granted` baseline that means the grant stands: an expired refusal is not current evidence and must not revoke indefinitely | Preserved | -| One valid record + a second malformed record of the same family | The **valid record governs**; the malformed one is ignored with a `warn` log. Fail-closed-on-malformed (below) applies only when no valid record of that family exists | Decided here | -| Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | -| Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | Preserved, circularity resolved | -| Proxy/mirror mode (CMP state mirrored server-side) | Mirror-sourced record enters as a live record; when both the mirror and the request carry records, the **request's record wins** (closer to the user) | Decided here | -| GPP opt-out fields | Enumerated per supported section: the sale, sharing, and targeted-advertising opt-out fields each independently constitute an opt-out signal when set; **all present-and-false** constitutes a grant signal (§4); absent/N-A fields contribute nothing. The exact field list per section ID is an appendix of the implementation PR, reviewed against the GPP spec | Decided here | -| Malformed-but-present record, no valid record of that family | **Blocks grants** (fail-closed acquisition — it does not degrade to "absent", which under a `granted` baseline would turn garbage into a grant, the fail-open path in both #838 and the first draft of this spec). Never triggers withdrawal — destruction requires an affirmative, decodable signal (§4.2) | Changed (declared) | +| Input state | Effective record / outcome | Status | +| ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | +| Standalone TCF and GPP-embedded TCF disagree, mode `restrictive` | **Whole-record selection** (today's semantics — an earlier draft specified per-purpose synthesis, which is _not_ what the code does): the record whose combined P1 ∧ P4 eligibility is more restrictive governs in full | Preserved (mode semantics pinned against current tests) | +| Same, mode `permissive` | Whole-record selection: the record whose combined P1 ∧ P4 eligibility is more permissive governs in full | Preserved (same pinning) | +| Same, mode `newest` | Whole-record selection by **`LastUpdated`** (not `Created`), subject to the existing freshness threshold; a tie or inconclusive comparison falls back to the **restrictive** selection | Preserved (same pinning) | +| Expired consent record | Treated as **absent entirely** — grants nothing, refuses nothing, withdraws nothing; the baseline applies. Under a `granted` baseline that means the grant stands: an expired refusal is not current evidence and must not revoke indefinitely | Preserved | +| One valid record + a second malformed record of the same family | The **valid record governs**; the malformed one is ignored with a `warn` log. Fail-closed-on-malformed (below) applies only when no valid record of that family exists | Decided here | +| One valid record + one **expired** record of the same family | The valid record governs; the expired record is absent entirely (consistent with the expiry row) | Decided here | +| Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | +| Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | Preserved, circularity resolved | +| Proxy/mirror mode | **Consent decoding is skipped entirely** — today's behavior, preserved as-is; no mirror-sourced record is synthesized (an earlier draft invented one). Retiring proxy mode, if ever wanted, is its own declared change | Decided here | +| GPP opt-out fields | Normative, not deferred: in the US-National section, `SaleOptOut`, `SharingOptOut`, and `TargetedAdvertisingOptOut` each independently constitute an opt-out signal when set to opted-out; each supported US state section maps its correspondingly named fields identically; a field explicitly set to not-opted-out is grant-class evidence (§4, regime-scoped); absent or N/A fields contribute nothing; **unsupported sections contribute nothing** (neither grant nor revoke). Adding a section is a spec change to this row | Decided here | +| Malformed-but-present record, no valid record of that family | **Blocks grants** (fail-closed acquisition — it does not degrade to "absent", which under a `granted` baseline would turn garbage into a grant, the fail-open path in both #838 and the first draft of this spec). Never triggers withdrawal — destruction requires an affirmative, decodable signal (§4.2) | Changed (declared) | ## 5. Jurisdiction resolution @@ -490,42 +511,57 @@ Consumers of the resolved set in this epic: inventory, normative per path (one test per row; a denylist check proves no ungated egress exists): - | Path | Required permissions | Notes | - | ---------------------------------------------------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | - | OpenRTB `user.id` | `store-on-device` ∧ `select-personalised-ads` | Raw EC is identity in the bidstream — gated exactly as EIDs. PR #838 gated only EIDs, leaving `user.id` reachable with Purpose 4 refused | - | EC-derived auction request IDs | both purposes | Derived values are identity | - | Page-bids path | both purposes | | - | Bidstream EIDs | both purposes | The one gate PR #838 had | - | Proxy / click / Testlight forwarding of the EC cookie or headers | both purposes | **New hardening, declared change** — these paths extract the raw cookie/header without today's jurisdiction gate (migration spec §2 row 11b) | - | Identify endpoint (partner-facing) | both purposes | Partner identity exchange, not a first-party lookup — decided here | - | Pull sync / batch sync (partner identity exchange) | both purposes | Authority source for S2S requests: stored provenance, below | - | Request-scoped graph reads/writes (non-revocation) | `store-on-device` | | - | Revocation paths (tombstones, withdrawal reads) | **exempt** | Must work when permissions are unset | - | Stored consent-state lookup (§4.4) | **exempt**, narrowly scoped | Determining `store-on-device` cannot require `store-on-device` | + | Path | Required permissions | Notes | + | ---------------------------------------------------------------- | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | OpenRTB `user.id` | `store-on-device` ∧ `select-personalised-ads` | Raw EC is identity in the bidstream — gated exactly as EIDs. PR #838 gated only EIDs, leaving `user.id` reachable with Purpose 4 refused | + | EC-derived auction request IDs | both purposes | Derived values are identity | + | Page-bids path | both purposes | | + | Bidstream EIDs | both purposes | The one gate PR #838 had | + | Proxy / click / Testlight forwarding of the EC cookie or headers | both purposes | **New hardening, declared change** — these paths extract the raw cookie/header without today's jurisdiction gate (migration spec §2 row 11b) | + | Identify endpoint (partner-facing) | both purposes | Partner identity exchange, not a first-party lookup — decided here | + | Pull sync (browser-request-scoped partner exchange) | both purposes, from the **live** request resolution | Pull sync is created from a browser request and checks the live `EcContext` today — it keeps using the live P1 ∧ P4 decision plus the family revocation state (§4.3); stored provenance is never a substitute for available live evidence | + | Batch sync (context-free S2S partner exchange) | both purposes, from **stored provenance** | The only truly signal-less path; authority rules below. Today's handler only authenticates and checks row state, so this gate is **declared hardening** (migration spec §2) | + | Request-scoped graph reads/writes (non-revocation) | `store-on-device` | | + | Revocation paths (tombstones, withdrawal reads) | **exempt** | Must work when permissions are unset | + | Stored consent-state lookup (§4.4) | **exempt**, narrowly scoped | Determining `store-on-device` cannot require `store-on-device` | With **no EC provider configured**, identity use fails closed: a cookie value present on the request never egresses anywhere — never vacuously allowed (#838's `ec_allowed` was `is_none_or`, vacuously true with no provider). - **S2S authority (batch/pull sync).** A server-to-server request carries - no user signals, geo, or `EcContext` to resolve permissions from. Its - authority is the identity's **stored provenance**: a record written at - mint/update time carrying the resolved jurisdiction, regime, grant - basis, and provider/version (providers spec §6.1). A sync request - re-validates that provenance against the **current** policy revision: - if the stored jurisdiction now resolves to `denied` for the required - permission, the row is not updated and is flagged for the operational - cleanup of §4.2 trigger 3. Sync never mints authority of its own. + **S2S authority (batch sync).** A context-free server-to-server request + carries no user signals, geo, or `EcContext`. Its authority is the + identity's **stored provenance**: per-permission, time-bounded evidence + written at mint and refreshed on later live requests — grant basis + (which signal class granted, per permission), evidence timestamp, + resolved jurisdiction, policy revision, and provider/version (providers + spec §6.1). A sync request performs a **full recompute of both + permissions** from that stored evidence against the _current_ policy: + it fails closed when the stored jurisdiction's rule is now `denied`, + when a `granted` baseline tightened to `requires_signal` and the stored + evidence contains no accepted grant for that permission, when the + stored evidence has **expired**, or when the regime no longer accepts + the stored grant's source class (§4's regime-scoped table). Any of + these → no update, row flagged for the operational cleanup of §4.2 + trigger 3. Sync never mints authority of its own. + + **Legacy (pre-epic) rows** carry none of these fields. They are treated + as reserved `hmac-v0` provenance with **no stored grant evidence**, so + they **fail closed for partner egress and batch updates** until a live + browser request lazily backfills provenance from a fresh resolution. + Failing open here would grandfather every pre-epic identity past the + permission model indefinitely. 4. **Server-side auction dispatch** — gated on the policy `regime` class, normatively: - | Regime | Dispatch rule | Preserves | - | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- | - | `gdpr` | Dispatch only with a decodable, unexpired TCF record consenting to Purpose 1. Malformed, expired, or absent record → **no bid request leaves** (no-bid response). | Today's GDPR/unknown arm | - | `us-privacy` | Dispatch proceeds in every signal state, including opt-out — the opt-out strips identity (rows above) but the contextual auction runs. | Today's US-state arm | - | `none` | Dispatch proceeds. | Today's non-regulated arm | + | Regime | Dispatch rule | Preserves | + | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | + | `gdpr` | Dispatch only with a decodable, unexpired TCF record consenting to Purpose 1. Malformed, expired, or absent record → **no bid request leaves** (no-bid response). | Today's GDPR/unknown arm | + | `us-privacy` | Dispatch proceeds in every signal state, including opt-out — the opt-out strips identity (rows above) but the contextual auction runs. | Today's US-state arm | + | `none` | Dispatch proceeds. | Today's non-regulated arm | + | **Any regime, decodable TCF record present** | The `gdpr` row applies: dispatch requires that record to consent to Purpose 1 — a raw TCF signal makes the request GDPR-relevant regardless of geolocation, so a US or non-regulated request carrying a Purpose 1 refusal is blocked. | Today's raw-signal arm — **must not regress** | The **compiled-in fallback policy has `regime = "gdpr"`** (§3.1) — the no-policy posture must be the most protective for dispatch too, and a diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index c4dc9cc37..cb17b604d 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -122,8 +122,17 @@ Three global rules sit above every provider: characters) and a global maximum length — for the identifier itself, not only the graph key — enforced by core at mint and at parse, so no provider can emit a value the cookie layer or logs cannot carry. +- **Namespaces are declarative and core-proven.** Disjointness of two + opaque `parse` functions is not provable, so every provider declares a + **static namespace descriptor** in a core-owned declarative form — a set + of literal prefixes and/or fixed-shape grammars (alphabet + length + segments), never arbitrary parser logic. Core proves pairwise + disjointness of all configured descriptors at startup (§6.1), and the + conformance suite asserts each provider's `parse` accepts **only** + values matching its declared descriptor — so the declaration, not the + parser, is the authority the overlap check rests on. - **Namespace reservation.** The legacy HMAC grammar `{64hex}.{6alnum}` is - formally **reserved as the `hmac` provider's namespace**. `hmac`'s graph + formally **reserved as the `hmac` provider's namespace descriptor**. `hmac`'s graph key is the identifier verbatim and its cluster prefix is the 64-hex prefix, so every pre-epic row stays reachable and every prefix listing intact (migration spec §3) — and **no other provider may mint @@ -228,7 +237,17 @@ structural: device provider whose data use goes beyond security classification (for example feeding fingerprints into targeting or identity) is **not authorized by selection alone** and requires a vocabulary extension plus - a gate before it may ship. + a gate before it may ship. This bites immediately, not hypothetically: + today's graph rows persist the JA4 class, an HTTP/2 fingerprint hash, + and buyer-facing quality metadata — persistence and scoring that exceed + security classification. The epic therefore **stops writing + fingerprint-derived buyer-facing fields into new rows** (a declared + change, migration spec §2); the boolean security classification outcome + may be persisted. Re-adding them is the vocabulary-extension route. + Relatedly, the implementation PR must deliver a **field-level graph + contract table** — for every persisted row field: purpose, source, + gating permission, TTL, rewrite behavior, egress paths, and tombstone + scrubbing — reviewed against the egress inventory. PR #838 declared `required_permissions` on all three traits but consulted it only for the EC provider; the geo and device declarations were @@ -273,31 +292,44 @@ The contract: block, validated like the active one (§6 table). - **Parse order and ambiguity.** The active writer parses first; the first match wins. Overlapping recognition is not resolved at request time but - **forbidden at startup**: provider namespaces (§3) may not overlap, and - configuring an active/legacy pair whose grammars intersect is a - validation error. + **forbidden at startup**: the declared namespace descriptors (§3) of the + active writer and every legacy reader must be pairwise disjoint — a + check core can actually perform, because descriptors are declarative; + configuring a pair whose descriptors intersect is a validation error. - **The recognizing provider governs.** A legacy-owned identity is gated by the **legacy provider's** `required_permissions()` for identity use — the provider that minted under a declared data-use contract is the one whose contract applies. -- **Provenance is provider- and version-tagged.** Every graph row carries - the minting provider id and its configuration version (this is the same - provenance record the S2S sync authority reads, permission model spec - §7). Same-provider key/passphrase rotation is a version entry, not a - provider switch: parse consults all configured versions of the active - provider. +- **Provenance is provider- and version-tagged, with a defined rotation + schema.** Every graph row carries the minting provider id, its + configuration version, and the per-permission grant evidence (grant + basis, evidence timestamp, resolved jurisdiction, policy revision) that + the S2S sync authority recomputes from (permission model spec §7). + Same-provider key/passphrase rotation is configuration, not a provider + switch: a provider block may hold multiple `versions` entries + (`[ec.providers.hmac.versions.v2] passphrase = …`) with + `mint_version = "v2"` selecting the writer; `parse` consults versions in + declared order, newest first; removing a version entry is a retirement + subject to the same evidence rules as retiring a legacy reader + (migration spec §6). - A cookie recognized by a legacy reader is a live identity for read/withdrawal purposes; whether it is transparently re-minted under the active writer is a per-deployment choice (`[ec] rewrite_legacy = true|false`), and re-minting is subject to the full minting gate of §5. -- **Rewrite is transactional and linking, not fire-and-forget.** Order: - new row commits first, carrying a link to the old row and a copy of the - old row's consent metadata and partner mappings; only then does the - cookie swap; the old row is tombstoned (or link-retired) only after the - new row and cookie are in place. An interrupted rewrite leaves the old - cookie valid and simply retries — no state in which neither identity - works. **Withdrawal of either linked row tombstones both.** +- **Rewrite is transactional, linking, and confirmed by presentation.** + Order: new row commits first, carrying a link to the old row (sharing + its revocation family ID, permission model spec §4.3) and a copy of the + old row's consent metadata and partner mappings; then the new cookie is + emitted. The server only emits `Set-Cookie` — it cannot observe delivery + or acceptance, so **both linked rows stay live** until a later request + **presents the new cookie** (confirmation by presentation); only then is + the old row retired. A deployment may additionally cap the window with a + grace period no shorter than the old cookie's maximum lifetime plus + rollout skew. An interrupted or unconfirmed rewrite leaves the old + cookie fully valid — no state in which neither identity works. + **Withdrawal of either linked row revokes the shared family, i.e. + both.** - Retiring a legacy reader is the explicit end of those identities: the migration guide documents the cleanup procedure (migration spec §6). - Tests: switch active provider → request with old cookie → identity still @@ -341,6 +373,16 @@ outcomes. Requirements: +- **Adapters declare capabilities against an explicit matrix.** The + capability set the composition root checks selections against is + enumerated, not ad hoc: identity-graph persistence, atomic single-key + reservation (CAS — required by the client-cycle reservation and any + future compare-and-set use), KV prefix listing (cluster support), + platform geo, device host evidence (JA4/HTTP-2), and legacy-rewrite + support. Each adapter's declaration is part of its wiring, and the §6 + capability-mismatch startup error is driven by this matrix. Every §6.2 + runtime-failure row gets fault-injection coverage on every adapter that + declares the corresponding capability. - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in app state; PR #838 rebuilt the provider (cloning the secret into a fresh diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 2ad71526a..99ed06ef7 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -32,23 +32,24 @@ and whether that is a preservation or a declared change. **Silent changes are defects.** PR #838 changed six of these without declaring any; each was discoverable only because a deleted test had pinned the old behavior. -| # | Decision (today) | After epic | Status | -| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | -| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | -| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | -| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | -| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | -| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | -| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | -| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | -| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | -| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | -| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | -| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | -| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | -| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | -| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull/batch sync) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | -| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | +| # | Decision (today) | After epic | Status | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | +| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | +| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | +| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | +| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | +| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | +| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | +| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | +| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | +| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | +| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | +| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -67,8 +68,8 @@ selects `provider = "hmac"` and carries its passphrase over verbatim: vectors: fixed passphrase + IP → exact expected 64-hex prefix, committed so any divergence fails CI rather than rotating the production identity base. -- **Existing cookies stay recognized.** Fixture `ts-ec` values minted by - the pre-epic code pass the provider's `recognize`, and their graph rows +- **Existing cookies stay parseable.** Fixture `ts-ec` values minted by + the pre-epic code pass the provider's `parse`, and their graph rows (keyed by the identifier verbatim) remain reachable — no row is orphaned. - **The hash prefix keeps its semantics.** `ec_hash` remains the 64-hex prefix, preserving both its stability and its deliberate collision across @@ -106,40 +107,63 @@ Requirements: cannot flip config and binaries atomically. Sequence: - **Release N+1 (dual-read):** accepts the old shape (mapping `[ec] passphrase` to the `hmac` provider internally, logging a - deprecation warning per startup) _and_ the new shape. Fleet rolls - binaries to N+1 with config unchanged; then config flips to the new - shape via `ts config push`; either order is safe at every instant. + deprecation warning per startup) _and_ the new shape. Ordering is + **strictly reader-first, never "either order"**: current binaries + reject the new shape (and reject the new `[permissions]` / `[device]` + / `[geo]` additions as unknown fields), so the config may flip only + after **fleet convergence on N+1 is confirmed** — binaries first, + convergence gate, then `ts config push`. A config mixing old and new + fields (`[ec] passphrase` alongside `[ec] provider`) is **rejected** + by N+1, not reconciled. Rollback runs the sequence in reverse: config + back to the old shape first, binaries only after config convergence. + Every new config section introduced by the epic follows this same + compatibility rule, not only `[ec]`. - **Release N+2:** rejects `[ec] passphrase` at startup with a message naming the new location — not a generic unknown-field error (implementation note: producing the actionable message means keeping a deprecated `passphrase` field whose presence triggers the custom error). -2. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no +2. **The graph schema change is expand-contract, in lockstep with the + binary sequence.** New rows carry fields v1 rows never had — provider/ + version, per-permission grant evidence, policy revision, family ID, + rewrite links — and two failure modes must be engineered away: a naive + schema-version bump makes old readers fail closed on new rows, and an + old worker that reads, modifies, and reserializes a row **silently + drops** fields it does not model. Sequence: (a) a **reader/preserver + release** ships first — it understands the new fields and, critically, + preserves unknown fields verbatim through read-modify-write; (b) a + **fleet-convergence gate**; (c) only then does **writer activation** + begin emitting the new fields. Rows carry an explicit schema version; + backfill is lazy via live requests (the same pass that backfills legacy + provenance, permission spec §7). Mixed-version tests are mandatory: + old-reader/new-row, new-reader/old-row, and old-worker + read-modify-write preserving new fields byte-for-byte. +3. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no `provider = "hmac"` selector is a startup error (providers spec §6). In PR #838 this configuration — the exact state an operator following the docs reaches if they miss one line — validated green and silently minted zero ECs. -3. **PR #838-era keys fail loud.** `provider = "host-signals"` (shipped by +4. **PR #838-era keys fail loud.** `provider = "host-signals"` (shipped by PR #838, deliberately not carried into this epic — providers spec §2) and `provider = "client-fixed"` are unknown keys and rejected like any other, so a config written against the PR #838 example cannot silently select a provider that no longer exists. -4. **Provider switches go through legacy readers.** Changing +5. **Provider switches go through legacy readers.** Changing `[ec] provider` on a deployment with live identities requires listing the outgoing provider in `[ec] legacy_providers` (providers spec §6.1) so existing cookies keep resolving and stay withdrawable; the guide documents the switch sequence and the retirement/cleanup step that ends it. -5. **The example config ships the migrated happy path**, uncommented: +6. **The example config ships the migrated happy path**, uncommented: `provider = "hmac"` with its block, `[geo] default_country`, and (for Fastly) the behavior-preserving `[device] provider = "fastly"` and `[geo] provider = "platform"` lines present with a comment stating what removing them changes. PR #838's example shipped the passphrase block uncommented with the selector commented out — steering operators directly into the silent-stateless state. -6. Every misconfiguration in the providers spec §6 table fails at +7. Every misconfiguration in the providers spec §6 table fails at **startup**. Request-time failure for a configuration error is a defect. -7. Config-store payload validation (`ts config push`) applies the same +8. Config-store payload validation (`ts config push`) applies the same rules — including `[permissions]` policy validation (permission spec §3.3) — so a bad config is rejected at push time, before any instance restarts into it. @@ -150,10 +174,15 @@ The migration guide (a new `docs/guide/` page, linked from the release notes) gives one copy-pasteable recipe per adapter for "keep exactly today's behavior": -The recipe is **one complete, valid TOML fixture, committed to the -repository** (e.g. `docs/guide/fixtures/migration-preserving.toml`) and -included in the guide verbatim — never described as a textual delta -against the example file. (An earlier draft said "copy the example table, +The recipe is a **complete, valid TOML fixture per adapter, committed to +the repository** (e.g. `docs/guide/fixtures/migration-preserving-fastly.toml` +and siblings) and included in the guide verbatim — never described as a +textual delta against the example file. Per-adapter because a single +fixture cannot be: `[device] provider = "fastly"` and +`[geo] provider = "platform"` are capability-gated selections that the +Axum/Cloudflare/Spin adapters reject at startup (providers spec §6); each +adapter's fixture carries the selections valid for it, and each is +CI-validated against its adapter. (An earlier draft said "copy the example table, then set `[permissions.rules] default`" — but the copied table already declares `[permissions.rules]`, and reopening a TOML table is a parse error; a prose delta cannot be validated, a committed fixture can.) The @@ -165,7 +194,8 @@ fixture contains, in one document: - `[geo] provider = "platform"` and `default_country = "FR"` (per-request jurisdiction detection preserved; the default is fail-closed because FR resolves to the `gdpr-eu` rule); -- the full `gdpr-eu` / `gdpr-uk` / `us-opt-out` groups and country rules +- (Fastly fixture; other adapters substitute their valid selections) + the full `gdpr-eu` / `gdpr-uk` / `us-opt-out` groups and country rules from the example policy (US as `requires_signal` with the grant-signal class — §2 rows 3–3b), plus the `non-regulated` group with `rules.default = "non-regulated"` (row 7). Operators who prefer the @@ -175,7 +205,7 @@ A partial policy is a trap the first draft of this spec fell into: a `[permissions]` section containing **only** the permissive default — with no GDPR/US rules — sends _every_ jurisdiction, France included, to the permissive fallback, because `default_country` selects a -rule like any other country and finds none. The committed fixture is +rule like any other country and finds none. Each committed fixture is therefore always complete, and CI pins it: **the fixture file itself** is loaded and run through the complete §4.1/§4.2 decision matrix of the permission spec, asserting per-jurisdiction outcomes match the pre-epic @@ -201,9 +231,14 @@ global honoring of opt-out signals is unconditional. spec §5.2), raw-egress denials by path, tombstone family retries, legacy-reader hit rate, rewrite failures, and cluster-fallback engagements. Two of these carry thresholds, not just ranges: - legacy-reader hits trending to ~zero is the **retirement-readiness** - signal for a legacy provider, and a nonzero rewrite-failure rate blocks - retirement outright. + legacy-reader hits at zero for a **quiet period no shorter than the + maximum cookie/row lifetime plus rollout skew** — or provable + rewrite/backfill completion — is the **retirement-readiness** bar for a + legacy provider ("trending to ~zero" is not evidence; a yearly visitor + is not churn), and a nonzero rewrite-failure rate blocks retirement + outright. The telemetry set also includes: graph read/commit failures, + stored-provenance denials, schema-migration failures, and + replay-reservation recoveries. 3. Startup logs always print: selected provider per concern, whether geo is live, the effective default baseline, and the count of granted-without- signal permissions. One line, greppable, stable format. From 2b4d776b65790aea0c5596b7c45acd6ba4f7c1ac Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 31 Jul 2026 12:24:21 -0700 Subject: [PATCH 06/24] Address architecture review: legacy revocation, consent-field semantics, and distributed contracts P0: legacy identities can now enter the family-revocation protocol. Rows lacking a family ID derive one deterministically from (record kind, provider namespace, canonical graph key), so a first-post-upgrade withdrawal is discoverable by every future reader even if the writer crashes before touching the v1 row; random IDs are called out as recreating the orphan the design exists to eliminate. Withdrawal never depends on backfill; tests pin the first-request-is-withdrawal and crash-between-writes cases. P1 groups: - US signal fields get a normative field x value x permission x destructive table (4.5): sale maps to P1+P4 and is destructive, sharing and targeted-advertising map to P4 only and never destroy identity, USP carries no targeted field, absent/N-A grants nothing, state sections override national, opt-out beats grant across sections. The regime evidence table now defers to this mapping. - Normalization is a six-state machine (valid-grant, valid-refusal, opt-out, malformed-present, expired, absent) wired into precedence and the decision matrix (malformed blocks the granted baseline); proxy mode keeps a syntax pass so malformed is distinguishable, blocks record-derived grants, and is declared as a change from today's fail-open skip. - Conflict resolution is deterministic: whole-record selection over the (P1, P4) tuple ordered lexicographically P1-first (split-purpose records decided), newest uses LastUpdated with the freshness threshold and falls back to restrictive; expiry drops sources before conflict resolution - a declared change from today's conflict-first ordering. - The auction raw-signal arm triggers on raw TC-string presence or a GPP section-2 hint before decoding, so malformed raw TCF still blocks dispatch outside GDPR regions. - Stored provenance ages: authoritative timestamp + valid_until per evidence class, re-presentation does not reset age, and every live resolution atomically replaces the full per-permission snapshot so a later refusal clears older positive authority; rewrite provenance is the fresh live resolution, partner mappings keep original expiries. - A per-record-class consistency matrix: replay reservations need linearizable CAS with fencing (Durable-Object-class on Cloudflare, not Workers KV), family revocation records need strong reads plus a declared bounded visibility lag with read-failure failing closed, identity rows may be eventual; retention outlives every dependent lifetime (today's 24h tombstone TTL explicitly does not carry over). - The US policy enumerates US/ rules for configured privacy states with country-level US non-regulated, preserving Wyoming-class traffic; regionless-geo degradation is declared; the states-list consistency test is region-shaped. - The graph field contract is normative in-spec (providers 6.3): every v1 and new field with purpose, source, gating permission, TTL, rewrite, and revocation treatment - including discontinuing fingerprint-derived buyer-facing fields; releases unified as N/N+1/N+2 with semantic (not byte) unknown-field preservation, a hard rollback floor at N+1 after writer activation, and stated mixed-version expectations. - Identity boundaries are structural: core constructs physical graph keys with record-kind/provider/version prefixes (legacy hmac verbatim excepted) and an AuthorizedIdentity newtype - constructible only after parse, permission, graph, and family checks - is the only type outbound serializers accept. - Legacy rewrite aliases to one canonical row via fenced CAS (no dual-write divergence), with confirmation by presentation and a finite retirement deadline. - Client-cycle: session binding is required for production schemes (one-time consumption demoted to defense-in-depth; at-most-once only as an explicitly recorded posture with orphan cleanup); reservations carry owner hash and monotonic lease epochs with fenced transitions; owner-hash retry re-emits the cookie so lost responses do not orphan rows; resolve checks the family revocation record and loses races to revocation. - The hook snapshots all pre-hook cache restrictions (origin-supplied included) and allows only equal-or-stronger privacy; Content-Encoding and Content-Range join the reserved surface; the test set covers origin-private and core-private cookieless HTML, cache hits, Vary, every CDN directive, and body encoding. P2/P3: mint 'cookie write' clarified as scheduled-on-final-response with egress eligibility at graph commit; degraded-health is a per-instance in-memory state machine with hysteresis; HMAC versions resolve from row provenance (untagged = hmac-v0), not parse; client limits are exact (65,536-byte body, content-type allowlist, 256-byte identifier, 128-byte reservation key, per-code statuses); migration matrix gains rows 3d-3g; fixtures include the graph-store config; every rollout metric ships with threshold, window, and action; batch-sync's coverage dip is operationalized with the provenance-coverage metric; FR default relabeled a protective opt-in fallback; device-selection authorization qualified to the opt-in fingerprint provider; the illustrative policy example is labeled as such. --- ...26-07-30-client-cycle-ec-resolve-design.md | 82 +++++--- ...integration-response-header-hook-design.md | 38 ++-- .../2026-07-30-permission-model-design.md | 194 +++++++++++++----- .../2026-07-30-pluggable-providers-design.md | 126 +++++++++--- ...07-30-provider-migration-rollout-design.md | 62 ++++-- 5 files changed, 366 insertions(+), 136 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index faed2b32d..2f27f1ab6 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -65,13 +65,15 @@ Everything in this spec follows from that. that are signed by an expected party, **audience-bound** to this publisher, and **expiring**. Audience binding and expiry alone do not mitigate replay — a captured token installs in another browser for the - whole validity window — so one of the following is additionally - required: **binding to the requesting browser session** (a server-issued - nonce the payload must embed), or **server-side one-time consumption** - (a replay cache on the payload's unique id). A scheme that can support - neither may only ship if its residual replay window is quantified and - explicitly accepted in the feature's issue — "single-use where the - scheme allows" is not a mitigation. + whole validity window. **Production schemes require session binding** + (a server-issued nonce the payload must embed): one-time consumption + alone limits multiplicity but proves nothing about _which_ browser + redeems first — a captured bearer payload can simply win the race — so + it is defense-in-depth, not the mitigation. First-presenter + at-most-once semantics may ship **only** as an explicitly accepted + posture recorded in the feature's issue, together with a specified + orphan-row cleanup path. "Single-use where the scheme allows" is not a + mitigation. 3. **Preserve the identity-graph invariant.** The cookie is set only after the corresponding graph row is written, mirroring the organic path. Graph unavailable → no cookie, same as organic generation. @@ -86,13 +88,18 @@ Everything in this spec follows from that. 6. **Be uncacheable and permission-gated.** `Cache-Control: no-store`; the same `store-on-device` permission gate as organic EC creation runs before any cookie is set. -7. **Bound every input.** A maximum request-body size (order of the 64 KiB - limit PR #838 at least had), enforced by a **bounded read independent of - `Content-Length`** — a missing, false, or chunked length must not bypass - it; a `Content-Type` allowlist; and a length/character-set constraint on - the resulting identifier that keeps it cookie-safe and within the KV - limits of the providers spec §3. Tests exercise the exact 413 boundary - and the missing/false/chunked-length cases. +7. **Bound every input — exact values, testable boundaries.** Request + body: at most **65,536 bytes** (inclusive; byte 65,537 → `413`), + enforced by a bounded read independent of `Content-Length` — a + missing, false, or chunked length must not bypass it. `Content-Type` + allowlist: `text/plain` and `application/json`; anything else → `415`. + The resulting identifier: at most **256 bytes**, cookie-safe alphabet + (providers spec §3 global bounds); violation → `400`. Reservation key + (payload unique id): at most **128 bytes**. Status codes are part of + the contract: `400` malformed payload/identifier, `403` origin/token + rejection, `409` different-identity or revoked-family conflict (§3.8), + `413` body, `415` content type. Tests exercise each boundary at its + exact edge, including the missing/false/chunked-length cases. 8. **Define behavior against an existing identity — no silent replacement.** When the request already carries a recognized EC: resolving to the **same** identity is an idempotent no-op (cookie @@ -109,23 +116,36 @@ Everything in this spec follows from that. residual rows. Required shape: consumption is an **atomic single-key reservation** (CAS — an adapter capability the composition root checks, providers spec §7) keyed by the payload's unique id, with explicit - states: `pending` → `committed` | `failed`. The graph write happens - under the reservation and is retried under the same key; a `pending` - reservation older than its **lease** may be taken over by a retry; - reservations are retained at least through the token's expiry; the - graph write is deterministic under the reservation key so a retry - converges on the same row. - - **A duplicate must never receive the cookie unless the reservation is - session-bound.** "Duplicates observe the recorded outcome" cannot mean - replaying `Set-Cookie` — that would hand a captured token's identity to - a second browser, recreating the fixation §2 exists to prevent. In - one-time mode without session binding, a duplicate gets a terminal - response with **no cookie**; only a requester that proves the original - session binding (the §3.2 nonce) may have the `Set-Cookie` re-emitted. - Tests cover crash-between-steps, lease takeover, two concurrent - requests with the same payload, and a duplicate from a second client - receiving no cookie. + states: `pending` → `committed` | `failed`, each carrying an **owner + hash** (the session binding) and a **monotonic lease epoch**. Takeover + of an expired `pending` lease increments the epoch, and every state + transition is a fenced CAS on (state, epoch) — a stale owner resuming + after its lease expired cannot commit over the takeover's work, because + its epoch no longer matches. `failed` is retryable: the same owner may + supersede it with a fresh `pending` at a higher epoch. The graph write + happens under the reservation and is deterministic under its key, so + any retry converges on the same row; reservations are retained at least + through the token's expiry. + + **A duplicate must never receive the cookie unless it proves the + original session binding.** "Duplicates observe the recorded outcome" + cannot mean replaying `Set-Cookie` — that would hand a captured token's + identity to a second browser, recreating the fixation §2 exists to + prevent. A requester matching the reservation's owner hash **does** + have the `Set-Cookie` re-emitted — which is precisely how a legitimate + browser whose original response was lost recovers on retry, so a + committed graph row never strands as an orphan for the intended + browser; anyone else gets a terminal response with no cookie. In the + explicitly-accepted at-most-once posture (no owner hash), a lost + response is an **orphan row** handled by the specified cleanup path. + The **same-identity no-op of §3.8 first checks the family revocation + record** (permission model spec §4.3): a resolve against a revoked + family is rejected, never refreshed, and a create racing a revocation + loses — revocation wins. Tests cover crash-between-steps, lease + takeover with a stale-epoch commit attempt, two concurrent requests + with the same payload, a duplicate from a second client receiving no + cookie, owner-hash recovery receiving the cookie, and + resolve-vs-revocation races. ## 4. Requirements on the page script diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index f82f15dfb..8ec5b687d 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -53,13 +53,17 @@ mutators to the outbound response for HTML document responses it processed. `Set-Cookie` with a replaced public `Cache-Control` into a **shared-cacheable cookie response**. The invariant pass therefore runs after all mutations, unconditionally — and it enforces more than the - cookie rule: **any private/no-store classification core assigned before - the hook is preserved** (processed auction HTML is marked private even - when no cookie is emitted — today's final helper returns early without - `Set-Cookie`, so cookie-triggered enforcement alone would let an - integration make cookieless personalized HTML publicly cacheable), and - every CDN/surrogate cache directive is stripped from any response so - classified. Middle-stage placement also keeps + cookie rule. Core **snapshots the complete pre-hook cache restriction + state** — whether the restriction came from core's own classification + (processed auction HTML is marked private even when no cookie is + emitted; today's final helper returns early without `Set-Cookie`) **or + from the origin** (an origin-supplied `private, no-store` that core + merely passed through) — and the post-hook response may only be + **equal or stronger** on the privacy axis: integrations can tighten + caching, never loosen it, regardless of which header they replaced. + Every CDN/surrogate cache directive (`Surrogate-Control`, + `CDN-Cache-Control`, host-specific equivalents) is stripped from any + restricted response. Middle-stage placement also keeps the earlier property: an integration mutation is not silently stripped by ordinary core handling — only by the invariant pass, which logs the downgrade it applies. @@ -70,8 +74,11 @@ mutators to the outbound response for HTML document responses it processed. granularities because `Set-Cookie` is multi-valued: (a) reserved header _names_ — HTTP framing and hop-by-hop headers (`Content-Length`, `Transfer-Encoding`, `Connection`, `Trailer`, `Upgrade`, `TE`, - `Keep-Alive`), the `x-ts-*` namespace, and the consent/privacy headers - core emits; (b) reserved cookie _names_ within `Set-Cookie` — `ts-ec`, + `Keep-Alive`), **representation headers coupled to body bytes the hook + cannot see** (`Content-Encoding`, `Content-Range` — relabeling + uncompressed bytes as Brotli, or stripping the encoding from compressed + bytes, corrupts the response), the `x-ts-*` namespace, and the + consent/privacy headers core emits; (b) reserved cookie _names_ within `Set-Cookie` — `ts-ec`, `ts-eids`, and the other `ts-*` cookies core owns. An integration may append its own `Set-Cookie` values; it may not set or expire a reserved cookie name. Violations are rejected at the operation layer (§2) and @@ -135,9 +142,16 @@ processed documents (§6). 6. **Every row of the §3a eligibility matrix has a test** — streaming, cache-hit, pass-through, redirect, error, and 304 each proven to run or not run the hook — not merely one positive header test per adapter. -7. The cache/privacy invariant test: an integration appends a cookie and - replaces `Cache-Control` with a public/surrogate-cacheable value → the - final response is private/no-store with surrogate caching stripped. +7. Cache/privacy invariant tests, one per restriction source and shape: + cookie appended + public `Cache-Control` replacement → private/no-store, + surrogate stripped; **core-private cookieless** processed HTML + + public replacement → restriction preserved; **origin-private + cookieless** pass-through-classified content + public replacement → + restriction preserved; a cache-hit serve re-applying mutations without + weakening the stored classification; a `Vary` mutation neither + dropping core-required values nor bypassing the snapshot; each CDN + directive (`Surrogate-Control`, `CDN-Cache-Control`, host equivalents) + individually stripped; and a rejected `Content-Encoding` mutation. ## 5. Size and sequencing diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index f509e8bee..f47f6673d 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -121,6 +121,10 @@ overrides. Each permission resolves to an **acquisition rule**: - `denied` — never set, even when a signal grants it. ```toml +# Illustrative schema example — NOT the shipped policy. The shipped +# example (trusted-server.example.toml) pairs these groups with the most +# protective rules.default; the permissive default below demonstrates the +# reserved key. [permissions.groups.gdpr-eu] regime = "gdpr" default = "requires_signal" @@ -139,10 +143,15 @@ default = "granted" [permissions.rules] FR = "gdpr-eu" -US = "us-opt-out" +# US privacy gating applies per configured privacy state, matching today's +# state-list behavior; country-level US traffic (a Wyoming request, or one +# whose geo provider yields no region) stays non-regulated. One US/ +# rule per configured privacy state: +"US/CA" = "us-opt-out" +US = "non-regulated" # Overrides name explicit acquisition rules — no +/- sigil syntax; TOML # expresses the target state directly. -"US/CA" = { group = "us-opt-out", overrides = { select-personalised-ads = "requires_signal" } } +"US/CO" = { group = "us-opt-out", overrides = { select-personalised-ads = "requires_signal" } } # Reserved key: countries that resolve but match no rule. Required whenever # the [permissions] section is present. Distinct from [geo] default_country, # which handles requests that resolve no country at all (§5.4). @@ -190,7 +199,9 @@ Validation rejects: the compiled-in fallback omits the section entirely); - duplicate rule keys under case-insensitive comparison (`FR` and `fr`); - a `[geo] default_country` whose country part is not an assigned ISO - code; it accepts either a country (`FR`) or a country/region key + code; it accepts either a country (`FR`) or a country/region key whose + region part is validated as an assigned subdivision exactly like rule + keys (`US/ZZ` is rejected here too) (`US/CA`) — PR #838 supported region defaults, and a no-geo, single-state deployment must be able to select its state rule. It is canonicalized to uppercase, and startup logs which rule (or @@ -213,7 +224,16 @@ The class is never inferred from purpose flags. Where the legacy lists must survive an interim period, a CI test asserts consistency between each list and the policy's regime classes, with deliberate divergences recorded as explicit, commented exceptions in the test — never silent. Both legacy -lists are in scope, not only the GDPR one. +lists are in scope, not only the GDPR one — and the US check is +region-shaped: **every configured `consent.us_privacy.states` entry must +have a matching `US/` rule**, and the country-level `US` rule must +resolve non-regulated (today applies privacy gating only to the configured +states), or the divergence is an explicit commented exception. An adapter +whose geo provider cannot resolve regions degrades **intentionally and +declaredly**: regionless US traffic hits the country rule — non-regulated, +today's behavior for non-privacy-state traffic; an operator preferring +protective country-wide gating writes `US = "us-opt-out"` as their own +declared choice. ### 3.5 Shipped-table coverage @@ -250,11 +270,11 @@ blocked but an **explicit non-opt-out** value grants: permission-scoped** — grant signals are NOT interchangeable across regimes: - | Regime of the resolved rule | Evidence accepted as a grant for a `requires_signal` permission | - | --------------------------- | --------------------------------------------------------------- | - | `gdpr` | **Only** a TCF record consenting to that specific purpose | - | `us-privacy` | TCF consent for the purpose, or an explicit GPP/USP non-opt-out | - | `none` | Any grant-class signal | + | Regime of the resolved rule | Evidence accepted as a grant for a `requires_signal` permission | + | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | + | `gdpr` | **Only** a TCF record consenting to that specific purpose | + | `us-privacy` | TCF consent for the purpose, or GPP/USP evidence **per the §4.5 field mapping** — a field grants only the permissions it maps to | + | `none` | Any grant-class signal | Without this scoping, a US-style `sale_opt_out = false` would satisfy a French `requires_signal` rule — no TCF, both purposes granted, EC minted, @@ -293,20 +313,29 @@ blocked but an **explicit non-opt-out** value grants: coexisting TCF refusal beats a non-TCF grant signal, matching today's US-state ordering where a present TCF record decides before GPP/USP values are consulted). -5. No signal — the policy baseline decides: `granted` sets it, +5. Malformed-present, no valid record of that family (§4.4) — **blocks + the baseline grant**: the permission is unset even under `granted`. + Never withdraws. +6. No signal — the policy baseline decides: `granted` sets it, `requires_signal` leaves it unset. +Normalization (§4.4) reduces each record family to exactly one of six +states — **valid-grant, valid-refusal, opt-out, malformed-present, +expired, absent** — and the precedence above plus the §4.1 matrix are +defined over those states, so no input state is unmapped. + ### 4.1 Decision matrix For each enforced permission, with baseline _B_ ∈ {granted, requires_signal, denied}: -| Opt-out present | TCF refusal present | Accepted grant present (regime-scoped) | Result | -| --------------- | ------------------- | -------------------------------------- | ------------------------------------------------ | -| yes | — | — | **unset** (and withdrawal semantics apply, §4.2) | -| no | yes | — | unset (withdrawal per §4.2, trigger 2) | -| no | no | yes | set, unless B = denied | -| no | no | no | set iff B = granted | +| Opt-out present | TCF refusal present | Accepted grant present (regime-scoped) | Malformed-present | Result | +| --------------- | ------------------- | -------------------------------------- | ----------------- | ------------------------------------------------ | +| yes | — | — | — | **unset** (and withdrawal semantics apply, §4.2) | +| no | yes | — | — | unset (withdrawal per §4.2, trigger 2) | +| no | no | yes | — | set, unless B = denied | +| no | no | no | yes | **unset** (precedence 5 — blocks baseline grant) | +| no | no | no | no | set iff B = granted | ### 4.2 Withdrawal vs. absence @@ -356,11 +385,19 @@ record that is simultaneously the durable intent, the discovery mechanism, and the fail-closed marker: - **The family revocation record is written first.** Every identity carries - a stable **family ID**, minted with the identity and stored in every - member row (including rows linked by a legacy rewrite, providers spec - §6.1). Revocation writes one record keyed by the family ID. That single - write is the withdrawal: per-member tombstones are cleanup that follows, - idempotent and retried. + a stable **family ID**: minted rows store it, and — the case that makes + or breaks the protocol — **rows that lack the field derive it + deterministically** as a function of (record kind, provider namespace, + canonical graph key), e.g. `fam:v0:hmac:`. Determinism is the + point: a withdrawal arriving on the **first post-upgrade request** — a + GPC-carrying visitor whose v1 row has no family field and has never been + backfilled — computes the same family ID that every future reader of + that row computes, so the revocation record is discoverable even if the + writer crashes before ever touching the member row. A **random** ID + would recreate the exact partial-withdrawal orphan this design exists to + eliminate. Revocation writes one record keyed by the family ID; that + single write is the withdrawal. Per-member tombstones are cleanup that + follows, idempotent and retried. - **Every consumer checks the family record, not per-member tombstones.** A reader arriving through any still-live member row finds the family ID in the row and the revocation record under it — partial revocation is @@ -377,10 +414,22 @@ and the fail-closed marker: metric feeding the operational repair path. The residual that remains — a single failed write on an otherwise healthy graph, for a user who never returns — is declared here, not hidden. +- **Consistency and retention are backend contracts**, defined in the + providers spec consistency matrix (§7): revocation-record reads use the + strongest read the backend offers, adapters declare a bounded + revocation-visibility lag (an eventually-consistent store that cannot + bound it fails startup for identity features), a **failed family-record + read fails closed** for egress (revoked-unknown ≠ live), and revocation + records are retained beyond the maximum of cookie lifetime, row TTL, + rewrite grace, and downstream retry horizon — note today's 24-hour + tombstone TTL is far below this bar and does not carry over. - Fault-injection tests cover: family-record write fails → cookie untouched, S2S behavior per degraded mode, retry completes; member tombstone N fails after the family record → identity already revoked for - every reader, cleanup retries; the same-signal retry path end to end. + every reader, cleanup retries; the same-signal retry path end to end; + **first post-upgrade request is a withdrawal** (v1 row, no family field, + derived ID; crash between family record and row write; reader of the + untouched v1 row still sees the revocation). ### 4.4 Signal normalization — normative matrix @@ -389,21 +438,55 @@ record and one effective opt-out state per request. The normalization layer is where today's real-world mess lives, and PR #838 collapsed it silently. These are the outcomes — decided here, not delegated to the implementation; each row marked **changed** also appears in the migration -matrix: - -| Input state | Effective record / outcome | Status | -| ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | -| Standalone TCF and GPP-embedded TCF disagree, mode `restrictive` | **Whole-record selection** (today's semantics — an earlier draft specified per-purpose synthesis, which is _not_ what the code does): the record whose combined P1 ∧ P4 eligibility is more restrictive governs in full | Preserved (mode semantics pinned against current tests) | -| Same, mode `permissive` | Whole-record selection: the record whose combined P1 ∧ P4 eligibility is more permissive governs in full | Preserved (same pinning) | -| Same, mode `newest` | Whole-record selection by **`LastUpdated`** (not `Created`), subject to the existing freshness threshold; a tie or inconclusive comparison falls back to the **restrictive** selection | Preserved (same pinning) | -| Expired consent record | Treated as **absent entirely** — grants nothing, refuses nothing, withdraws nothing; the baseline applies. Under a `granted` baseline that means the grant stands: an expired refusal is not current evidence and must not revoke indefinitely | Preserved | -| One valid record + a second malformed record of the same family | The **valid record governs**; the malformed one is ignored with a `warn` log. Fail-closed-on-malformed (below) applies only when no valid record of that family exists | Decided here | -| One valid record + one **expired** record of the same family | The valid record governs; the expired record is absent entirely (consistent with the expiry row) | Decided here | -| Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | -| Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | Preserved, circularity resolved | -| Proxy/mirror mode | **Consent decoding is skipped entirely** — today's behavior, preserved as-is; no mirror-sourced record is synthesized (an earlier draft invented one). Retiring proxy mode, if ever wanted, is its own declared change | Decided here | -| GPP opt-out fields | Normative, not deferred: in the US-National section, `SaleOptOut`, `SharingOptOut`, and `TargetedAdvertisingOptOut` each independently constitute an opt-out signal when set to opted-out; each supported US state section maps its correspondingly named fields identically; a field explicitly set to not-opted-out is grant-class evidence (§4, regime-scoped); absent or N/A fields contribute nothing; **unsupported sections contribute nothing** (neither grant nor revoke). Adding a section is a spec change to this row | Decided here | -| Malformed-but-present record, no valid record of that family | **Blocks grants** (fail-closed acquisition — it does not degrade to "absent", which under a `granted` baseline would turn garbage into a grant, the fail-open path in both #838 and the first draft of this spec). Never triggers withdrawal — destruction requires an affirmative, decodable signal (§4.2) | Changed (declared) | +matrix. The pipeline order is itself normative: **(1) syntax validation +per source, (2) expiry per source — expired sources drop to absent +_before_ conflict resolution, (3) conflict resolution over the remaining +valid sources.** Current runtime resolves conflicts first and can select +an expired record before clearing both sources; expiry-first is a +**declared change** (migration matrix) that removes that path: + +| Input state | Effective record / outcome | Status | +| ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | +| Standalone TCF and GPP-embedded TCF disagree, mode `restrictive` | Whole-record selection over the **(P1, P4) outcome tuple, compared lexicographically with P1 first** (refusal < grant): the lesser tuple governs in full. Split-purpose records are thereby decided — (grant P1, refuse P4) vs (refuse P1, grant P4) selects the latter. Identical tuples → outcomes are identical; the GPP-embedded record is named for determinism. (An earlier draft specified per-purpose synthesis, which is _not_ what the code does; the tuple order is decided here and pinned against current tests) | Preserved — pinned against current tests | +| Same, mode `permissive` | Same tuple comparison; the **greater** tuple governs in full | Preserved — same pinning | +| Same, mode `newest` | Whole-record selection by **`LastUpdated`** (not `Created`), subject to the existing freshness threshold; a tie, an incomparable pair, or timestamps inside the threshold fall back to the fully deterministic `restrictive` rule above | Preserved — same pinning | +| Expired consent record | Treated as **absent entirely** — grants nothing, refuses nothing, withdraws nothing; the baseline applies. Under a `granted` baseline that means the grant stands: an expired refusal is not current evidence and must not revoke indefinitely | Preserved | +| One valid record + a second malformed record of the same family | The **valid record governs**; the malformed one is ignored with a `warn` log. Fail-closed-on-malformed (below) applies only when no valid record of that family exists | Decided here | +| One valid record + one **expired** record of the same family | The valid record governs — the expired one dropped at pipeline step 2, before conflict resolution ever saw it | **Changed (declared)** — current runtime resolves the conflict first and can select the expired record | +| Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | +| Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | Preserved, circularity resolved | +| Proxy/mirror mode | **Syntax validation still runs; semantic decoding is skipped.** A present record (well- or mal-formed) is _present, undecoded_: it blocks grants (a record TS will not read cannot vouch for consent) and never withdraws; absent → baseline. Header-carried opt-outs (GPC) are unaffected — they need no decoding. Without the syntax pass, malformed-present would be indistinguishable from absent, contradicting the fail-closed rule below | **Changed (declared)**: today proxy mode skips decoding entirely, which under a permissive baseline is fail-open | +| GPP / US Privacy fields | Per the normative field mapping of §4.5 — fields are not interchangeable signals, and absent/N-A fields grant nothing | Decided here (§4.5) | +| Malformed-but-present record, no valid record of that family | **Blocks grants** (fail-closed acquisition — it does not degrade to "absent", which under a `granted` baseline would turn garbage into a grant, the fail-open path in both #838 and the first draft of this spec). Never triggers withdrawal — destruction requires an affirmative, decodable signal (§4.2) | Changed (declared) | + +### 4.5 US signal field mapping — normative + +GPP and US Privacy fields map to specific permissions with specific +effects; they are never interchangeable, a field's absence or N/A value +contributes nothing, and only the fields marked destructive trigger +withdrawal. Section IDs and versions are those of the IAB GPP +specification current at implementation time; adding a section or field is +a change to this table. + +| Source · field | Value | `store-on-device` (P1) | `select-personalised-ads` (P4) | Destructive withdrawal? | +| -------------------------------------------- | ------------- | ---------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------ | +| GPP US section · `SaleOptOut` | opted out | opt-out | opt-out | **Yes** (preserves today) | +| GPP US section · `SaleOptOut` | not opted out | grant | grant | — | +| GPP US section · `SharingOptOut` | opted out | — | opt-out | No | +| GPP US section · `SharingOptOut` | not opted out | — | grant | — | +| GPP US section · `TargetedAdvertisingOptOut` | opted out | — | opt-out | **No** — a targeted-advertising choice must never destroy the stored identity | +| GPP US section · `TargetedAdvertisingOptOut` | not opted out | — | grant | — | +| US Privacy · `opt_out_sale` | `Y` | opt-out | opt-out | **Yes** (preserves today) | +| US Privacy · present, `N` or N/A | — | grant | grant | — (today's tests pin N/A as allowing; USP carries no distinct targeted-advertising field, so it never maps to one) | +| Any field | absent / N-A | — | — | — | + +**Multi-section aggregation:** when both a national and an applicable +state section are present, the state section governs for the fields it +carries; across whatever sections apply, **an opt-out in any applicable +section beats a grant in another** (restrictive aggregation). `SharingOptOut` +and `TargetedAdvertisingOptOut` are new enforcement inputs — current code +consults only the sale field — and are declared as such in the migration +matrix. ## 5. Jurisdiction resolution @@ -533,10 +616,17 @@ Consumers of the resolved set in this epic: **S2S authority (batch sync).** A context-free server-to-server request carries no user signals, geo, or `EcContext`. Its authority is the identity's **stored provenance**: per-permission, time-bounded evidence - written at mint and refreshed on later live requests — grant basis - (which signal class granted, per permission), evidence timestamp, + written at mint and replaced on later live requests — grant basis + (which signal class granted, per permission), the evidence's + **authoritative timestamp and `valid_until`** (per evidence class), resolved jurisdiction, policy revision, and provider/version (providers - spec §6.1). A sync request performs a **full recompute of both + spec §6.1). Two aging rules prevent perpetual renewal: **re-presenting + an unchanged signal does not reset evidence age** — only a record + carrying a newer authoritative timestamp (e.g. TCF `LastUpdated`) does; + and every live resolution **atomically replaces the complete + per-permission snapshot**, never merges — a refusal, opt-out, malformed + or absent state in the fresh resolution clears prior positive authority + for its scope, so an old P4 grant cannot survive a later P4 refusal. A sync request performs a **full recompute of both permissions** from that stored evidence against the _current_ policy: it fails closed when the stored jurisdiction's rule is now `denied`, when a `granted` baseline tightened to `requires_signal` and the stored @@ -556,12 +646,12 @@ Consumers of the resolved set in this epic: 4. **Server-side auction dispatch** — gated on the policy `regime` class, normatively: - | Regime | Dispatch rule | Preserves | - | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | - | `gdpr` | Dispatch only with a decodable, unexpired TCF record consenting to Purpose 1. Malformed, expired, or absent record → **no bid request leaves** (no-bid response). | Today's GDPR/unknown arm | - | `us-privacy` | Dispatch proceeds in every signal state, including opt-out — the opt-out strips identity (rows above) but the contextual auction runs. | Today's US-state arm | - | `none` | Dispatch proceeds. | Today's non-regulated arm | - | **Any regime, decodable TCF record present** | The `gdpr` row applies: dispatch requires that record to consent to Purpose 1 — a raw TCF signal makes the request GDPR-relevant regardless of geolocation, so a US or non-regulated request carrying a Purpose 1 refusal is blocked. | Today's raw-signal arm — **must not regress** | + | Regime | Dispatch rule | Preserves | + | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | + | `gdpr` | Dispatch only with a decodable, unexpired TCF record consenting to Purpose 1. Malformed, expired, or absent record → **no bid request leaves** (no-bid response). | Today's GDPR/unknown arm | + | `us-privacy` | Dispatch proceeds in every signal state, including opt-out — the opt-out strips identity (rows above) but the contextual auction runs. | Today's US-state arm | + | `none` | Dispatch proceeds. | Today's non-regulated arm | + | **Any regime, raw TCF signal present** — a TC string on the request or a GPP section-2 hint, detected **before decoding** | The `gdpr` row applies: dispatch requires the _effective_ record to be decodable, unexpired, and consenting to Purpose 1. A **malformed or expired** raw signal therefore blocks dispatch — today a malformed raw TCF blocks, and gating this arm on decodability would have silently relaxed that. A US or non-regulated request carrying a Purpose 1 refusal is likewise blocked. | Today's raw-signal arm — **must not regress** | The **compiled-in fallback policy has `regime = "gdpr"`** (§3.1) — the no-policy posture must be the most protective for dispatch too, and a @@ -591,9 +681,15 @@ further consumer if and when it proceeds. (consent, opt-out, malformed, expired, absent), including the no-policy fallback regime, asserting both the dispatch decision and that a blocked dispatch emits no outbound request. -- The §7 S2S authority path: sync against stored provenance, including - the policy-tightened-to-denied case (no update, flagged for cleanup) - and the exempt consent-state lookup. +- The §7 S2S authority path: **every denial reason individually** — + denied rule, tightened baseline without acceptable stored evidence, + expired evidence, regime-rejected grant source — plus the exempt + consent-state lookup, stale-evidence re-presentation (age must not + reset), and legacy-row fail-closed-then-backfill. +- The full cross-product **regime × permission × evidence source** from + §4's acceptance table and §4.5's field mapping, including multi-section + aggregation conflicts. +- Legacy-row withdrawal end to end (§4.3's derived family ID). - §4.3 fault-injection cases. - Policy validation tests for every §3.3 rejection, exercised through both acceptance paths (push-time and startup). diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index cb17b604d..8c934bc31 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -213,9 +213,24 @@ cookie write, no egress, no auction use may observe a minted identifier before its graph row (with provenance, §6.1) has committed — PR #838 let a generated EC reach an auction before finalization refused the cookie, producing an identity that existed for one request and nowhere else. The -normative order is: gate → `generate` → graph-row commit → cookie write → -eligible for egress. A graph-commit failure means the mint never happened: -no cookie, no egress, error logged, the next request retries. +normative order is: gate → `generate` → graph-row commit → cookie +scheduled → eligible for egress. "Cookie scheduled" means queued onto the +final response — `Set-Cookie` is physically emitted after first-request +processing, so egress eligibility begins at **graph commit**, not at +header emission; the identity exists durably from that moment. A +graph-commit failure means the mint never happened: no cookie, no egress, +error logged, the next request retries. + +**Egress is typed, not policed.** The inventory-and-denylist test +(permission model spec §7) is a backstop, but conventions do not survive +new code — the ungated proxy/click/Testlight paths happened precisely +because raw EC values circulate as ordinary strings. Core therefore +introduces an **`AuthorizedIdentity`** newtype constructible only by core, +only after parse + permission gate + graph/family-revocation check; +outbound serializers (ORTB builder, page bids, sync, identify, forwarding) +accept `AuthorizedIdentity`, never `&str`/`EcId`. A future bypass then +requires deliberately reconstructing the raw string — visible in review — +rather than passing along what was already in hand. The gate applies to EC providers **only**. Geo and device are ungated for two _different_ reasons, stated separately because only one of them is @@ -244,10 +259,8 @@ structural: fingerprint-derived buyer-facing fields into new rows** (a declared change, migration spec §2); the boolean security classification outcome may be persisted. Re-adding them is the vocabulary-extension route. - Relatedly, the implementation PR must deliver a **field-level graph - contract table** — for every persisted row field: purpose, source, - gating permission, TTL, rewrite behavior, egress paths, and tombstone - scrubbing — reviewed against the egress inventory. + The field-level graph contract itself is normative in this spec — + §6.3 — not deferred to the implementation. PR #838 declared `required_permissions` on all three traits but consulted it only for the EC provider; the geo and device declarations were @@ -317,19 +330,26 @@ The contract: active writer is a per-deployment choice (`[ec] rewrite_legacy = true|false`), and re-minting is subject to the full minting gate of §5. -- **Rewrite is transactional, linking, and confirmed by presentation.** - Order: new row commits first, carrying a link to the old row (sharing - its revocation family ID, permission model spec §4.3) and a copy of the - old row's consent metadata and partner mappings; then the new cookie is - emitted. The server only emits `Set-Cookie` — it cannot observe delivery - or acceptance, so **both linked rows stay live** until a later request - **presents the new cookie** (confirmation by presentation); only then is - the old row retired. A deployment may additionally cap the window with a - grace period no shorter than the old cookie's maximum lifetime plus - rollout skew. An interrupted or unconfirmed rewrite leaves the old - cookie fully valid — no state in which neither identity works. - **Withdrawal of either linked row revokes the shared family, i.e. - both.** +- **Rewrite aliases to one canonical row — no dual-write window.** Order: + the new canonical row commits first, sharing the old row's revocation + family ID (permission model spec §4.3); its provenance is the **current + live resolution** (rewrite happens on a live request — copying old + consent evidence would rejuvenate stale authority), while partner + mappings copy **with their original per-field timestamps and expiry**. + Then a fenced CAS **replaces the old row with an alias record** pointing + at the canonical row; from that moment every read or update through + either cookie chases the alias (single hop) to the one canonical row — + a concurrent pull/batch/identify update cannot land on a row about to + be discarded, because after the CAS there is only one row to land on, + and an update racing the CAS itself retries against the canonical. Then + the new cookie is emitted. The server cannot observe `Set-Cookie` + acceptance, so the **alias stays live** until a later request presents + the new cookie (confirmation by presentation), and in any case until a + **finite retirement deadline** no shorter than the old cookie's maximum + lifetime plus rollout skew. An interrupted rewrite leaves the old + cookie resolving (directly or via the alias) — no state in which + neither identity works. **Withdrawal through either cookie revokes the + shared family.** - Retiring a legacy reader is the explicit end of those identities: the migration guide documents the cleanup procedure (migration spec §6). - Tests: switch active provider → request with old cookie → identity still @@ -361,6 +381,43 @@ when a healthy configuration meets an unhealthy runtime. Every row logs at | Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | | Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | +The **degraded-graph health signal** referenced above and by the +withdrawal contract is a defined state machine, not a vibe: it is +**per-instance and in-memory** (no shared propagation, no stored health +record whose own read could fail), entered when graph-write failures cross +a sliding-window threshold (N failures within window W), and exited with +hysteresis after M consecutive successes. While degraded: S2S partner +egress and sync updates fail closed; organic requests continue stateless. +The thresholds ship as constants with the implementation and are printed +in the startup log. + +### 6.3 Graph row contract — normative + +The per-field contract for identity rows, covering today's v1 fields and +the fields this epic adds. Serialization is JSON with the existing `v` +schema-version discriminator; from release N+1 onward (migration spec §4), +readers round-trip unknown keys **semantically** (values preserved through +read-modify-write; byte-identical output is not required and not +achievable through a structured serializer). + +| Field | Purpose | Source | Gating permission (egress) | TTL / refresh | Rewrite | On revocation | +| ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | +| key (v1: identifier verbatim; v2: core-constructed, §4) | Row identity | Provider/core | — | Row TTL (1 y today) | New canonical row; old key becomes alias | Family record governs; member tombstone as cleanup | +| `v` | Schema discriminator | Core | — | — | Written at current version | Retained | +| `created` | Row age | Core | P1 (first-party ops) | Never refreshed | Preserved (no rejuvenation) | Retained in tombstone | +| `consent.tcf` / `consent.gpp` | Raw signal snapshot for audit; superseded as authority by provenance | Request | Never egressed to partners | Replaced on live resolution (§7 snapshot rule, permission spec) | Fresh live values | Scrubbed | +| `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | +| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision, provider/version) | S2S authority | Live resolution only | Read by S2S recompute | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | +| New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | +| `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | +| `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | +| `pub_properties` (origin/seen domains) | Creation context | Core at mint | P1 | Write-once | Preserved | Scrubbed | +| `device.*` (JA4 class, H2 hash, quality metadata) | **Discontinued for new rows** (§5): fingerprint-derived, buyer-facing — beyond security-classification authorization. v1 rows retain them read-only; they are never egressed post-epic and are dropped at rewrite | Fastly device provider | None grants egress | Write-once (v1) | **Dropped** | Scrubbed | +| New: security classification outcome (boolean) | Bot-gate result | Device provider | — (never egressed) | Written at mint | Fresh | Scrubbed | +| `network.*` | Cluster disambiguation | Platform at mint | P1 | Write-once | Fresh | Scrubbed | +| `ids` (partner → UID map) | Partner identity graph | Pixel/pull/batch sync | P1 ∧ P4 (partner egress) | Per-mapping timestamps; bounded count/length | Copied **with original timestamps/expiry** | Scrubbed | +| New: alias record kind | Rewrite indirection (§6.1) | Core | — | Retirement deadline | Is the mechanism | Family-revoked like any member | + ## 7. Composition root and adapter parity Provider construction happens in exactly one place per concern @@ -373,16 +430,25 @@ outcomes. Requirements: -- **Adapters declare capabilities against an explicit matrix.** The - capability set the composition root checks selections against is - enumerated, not ad hoc: identity-graph persistence, atomic single-key - reservation (CAS — required by the client-cycle reservation and any - future compare-and-set use), KV prefix listing (cluster support), - platform geo, device host evidence (JA4/HTTP-2), and legacy-rewrite - support. Each adapter's declaration is part of its wiring, and the §6 - capability-mismatch startup error is driven by this matrix. Every §6.2 - runtime-failure row gets fault-injection coverage on every adapter that - declares the corresponding capability. +- **Adapters declare capabilities against an explicit matrix — with + consistency semantics, not just feature bits.** The capability set: + identity-graph persistence, atomic single-key reservation, KV prefix + listing (cluster support), platform geo, device host evidence + (JA4/HTTP-2), and legacy-rewrite support. Persistence capabilities carry + **per-record-class consistency requirements**, because "has KV" says + nothing about whether revocation is observable: + + | Record class | Required semantics | + | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | + | Family revocation records | Strongest read the backend offers, plus a **declared, bounded revocation-visibility lag** (Workers KV documents up to ~60 s eventual propagation — that bound must be declared, and the residual it implies stated in operator docs). Unboundable lag → startup failure for identity features. A **failed or erroring revocation-record read fails closed** for egress | + | Identity rows | Eventual consistency acceptable — rows are accretive, and the family-record check (strong, per above) governs use | + + Each adapter's declaration is part of its wiring, drives the §6 + capability-mismatch startup error, and every §6.2 runtime-failure row + gets fault-injection coverage on every adapter declaring the + corresponding capability. + - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in app state; PR #838 rebuilt the provider (cloning the secret into a fresh diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 99ed06ef7..eaf6bb133 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -40,6 +40,10 @@ discoverable only because a deleted test had pinned the old behavior. | 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | | 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | | 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | +| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | +| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — strictly more protective | +| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | +| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | | 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | | 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | | 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | @@ -129,15 +133,26 @@ Requirements: rewrite links — and two failure modes must be engineered away: a naive schema-version bump makes old readers fail closed on new rows, and an old worker that reads, modifies, and reserializes a row **silently - drops** fields it does not model. Sequence: (a) a **reader/preserver - release** ships first — it understands the new fields and, critically, - preserves unknown fields verbatim through read-modify-write; (b) a - **fleet-convergence gate**; (c) only then does **writer activation** - begin emitting the new fields. Rows carry an explicit schema version; - backfill is lazy via live requests (the same pass that backfills legacy - provenance, permission spec §7). Mixed-version tests are mandatory: - old-reader/new-row, new-reader/old-row, and old-worker - read-modify-write preserving new fields byte-for-byte. + drops** fields it does not model. The sequence shares the config + release names: **N+1 is the reader/preserver release** — it understands + the new fields and preserves unknown keys **semantically** through + read-modify-write (values round-trip; byte-identical JSON is neither + required nor achievable through a structured serializer — and a + genuinely pre-N+1 worker cannot preserve at all, which is exactly why + the floor exists); after the **fleet-convergence gate**, **N+2 + activates the writer** and begins emitting the new fields. **Rollback + below N+1 is prohibited once any new-format row exists** — a pre-floor + binary would silently strip the new fields from every row it touches. + Rows carry the existing `v` schema discriminator; backfill is lazy via + live requests (the same pass that backfills legacy provenance, + permission spec §7) — and, critically, **withdrawal never depends on + backfill**: the family ID for an untouched v1 row is derived + deterministically (permission spec §4.3), so a first-post-upgrade + GPC request withdraws correctly with zero migrated state. Mixed-version tests with stated expected results: + N+1-reader/old-row → full function; old-reader/new-row → v1 semantics, + new fields untouched if read-only, preserved semantically if + read-modify-write on N+1, **test-proven lost on pre-N+1** (documenting + why the floor is a floor); N+2-reader/N+1-written-row → full function. 3. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no `provider = "hmac"` selector is a startup error (providers spec §6). In PR #838 this configuration — the exact state an operator following the @@ -188,12 +203,16 @@ declares `[permissions.rules]`, and reopening a TOML table is a parse error; a prose delta cannot be validated, a committed fixture can.) The fixture contains, in one document: -- `[ec] provider = "hmac"` with its passphrase block; +- `[ec] provider = "hmac"` with its passphrase block, **and the + identity-graph store configuration** — selecting a minting provider + without an openable graph store is a startup error (providers spec §6), + so a fixture omitting it would not start; - `[device] provider = "fastly"` (Fastly deployments: preserves the JA4 bot gate); - `[geo] provider = "platform"` and `default_country = "FR"` (per-request - jurisdiction detection preserved; the default is fail-closed because FR - resolves to the `gdpr-eu` rule); + jurisdiction detection preserved; the FR default is a **protective + opt-in fallback**, not fail-closed — valid TCF consent still grants, + where today's unresolved-geo path always denies); - (Fastly fixture; other adapters substitute their valid selections) the full `gdpr-eu` / `gdpr-uk` / `us-opt-out` groups and country rules from the example policy (US as `requires_signal` with the grant-signal @@ -238,11 +257,26 @@ global honoring of opt-out signals is unconditional. is not churn), and a nonzero rewrite-failure rate blocks retirement outright. The telemetry set also includes: graph read/commit failures, stored-provenance denials, schema-migration failures, and - replay-reservation recoveries. + replay-reservation recoveries. **Each rollout-gate metric ships with a + threshold, an evaluation window, and a named action** (pause rollout / + roll back / block retirement) in the migration guide — a metric with a + "healthy range" but no action is dashboard decoration; the two already + specified (legacy-reader quiet period, rewrite failures) are the + pattern the rest follow. 3. Startup logs always print: selected provider per concern, whether geo is live, the effective default baseline, and the count of granted-without- signal permissions. One line, greppable, stable format. -4. Rollback is config-only where possible: reverting to the previous +4. **The batch-sync coverage dip is operationalized, not discovered.** + Because legacy rows fail closed for batch updates until backfilled + (permission spec §7), batch-sync acceptance drops toward zero at + cutover and recovers along the live-traffic backfill curve. The + **provenance-coverage metric** (share of active rows carrying + provenance) is the tracking signal; the migration guide states the + expected recovery shape, tells operators to notify batch-sync partners + of the transient rejection rate, and defines no fail-open shortcut — + the alternative (grandfathering pre-epic identities past the + permission model) is rejected in the permission spec. +5. Rollback is config-only where possible: reverting to the previous config version restores the previous behavior on the previous binary. The one irreversible artifact is withdrawal tombstones — which is why the withdrawal triggers (permission spec §4.2) are exhaustive, why partial From de70ca931138ed21f7d7a3a44fe9c70aecac3f94 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 31 Jul 2026 14:20:41 -0700 Subject: [PATCH 07/24] Address fifth review: opt-out subclasses, GPP applicability, consistency eligibility, and rollout closure P1 fixes: - Opt-out signals split into destructive (GPC, sale, USP - withdraw) and non-destructive (sharing, targeted-advertising - revoke P4 only, never tombstone) subclasses assigned by the 4.5 mapping, resolving the 4.2-vs-4.5 contradiction. - Proxy mode performs minimal opt-out extraction (the 4.5-mapped fields and USP only) so globally authoritative opt-outs are never suppressed; still no record-derived grants; declared as a change from today's opt-out-blind skip. - GPP applicability is an ordered algorithm with a pinned jurisdiction -> section-ID map (usnat 7, usca 8, usva 9, usco 10, usut 11, usct 12): applicability from resolved jurisdiction, state-over-national per field, restrictive aggregation; foreign and non-applicable sections contribute nothing; N/A preserved as not-opted-out (declared, correcting the earlier contributes-nothing rule). - TCF conflict selection reverts to today's algorithm - P1-and-P4 conjunction comparison with standalone winning equal conjunctions (including split-purpose) - replacing the invented lexicographic tuple and keeping the Preserved label honest. - S2S freshness is a per-evidence-class contract: TCF ages by LastUpdated, GPP/USP by first-seen with an equality digest (re-presentation keeps original first-seen), baseline grants re-derive from the current policy revision; clock-skew clamping. - Degraded-mode protection is declared local-only with the cross-instance residual quantified (bounded by user return latency, metered), instead of implying fleet-wide fail-closed. - Family revocation records require a strongly consistent primitive; Workers KV is explicitly ineligible ('60 seconds or more' is not a bound); alias/rewrite records join reservations in the linearizable CAS class and rewrite_legacy is rejected without it. - The trait now really returns graph_key_suffix (a round-4 batch loss), core owns the 6.3 physical key grammar (id/, alias/, fam/, rwx/, resv/ prefixes + reserved legacy grammar) with wire schemas and TTLs per record class. - HMAC version attribution really resolves from immutable row provenance (also a round-4 batch loss); parse identifies namespace only. - AuthorizedIdentity is scope-parameterized (GraphOps vs PartnerEgress) so a P1-only identity cannot reach an ORTB serializer. - The provider contract gains acquisition modes (ServerMint / ClientResolve carrying resolve_from_client and the JS module id); the resolve endpoint enforces the provider's full required_permissions. - Revocation-wins at the resolve endpoint holds because the family check runs through the linearizable class client-cycle already requires. - Rewrite is a persistent fenced transaction: pinned target on retry, reconciliation of updates that won the old-row CAS, orphan GC by absent transaction, and fenced alias retargeting keeping chains single-hop. - Release protocol: rollback is binaries-first (N+2 -> N+1 keeping the new config); N+1 rejects provider/version selections it cannot encode; a pre-N+1 graph-store readiness step plus matrix row 12 covers graphless HMAC deployments (breaking, declared). - The abstract capability list becomes a concrete adapter matrix (Fastly/Axum/Cloudflare/Spin) with honest cells - including that Cloudflare supports platform geo country-only (the migration text claiming it rejects platform geo was wrong) and that no CAS-class primitive is currently wired anywhere but the dev adapter. - Integration cookies enter the permission model: registration-declared names with purpose and retention, persistent cookies gated on store-on-device, session cookies as the narrow exemption. - Cache monotonicity is a defined lattice (no-store > no-cache > private > public, shrink-only ages, snapshot-gated stale directives, protected Vary union) in the contract, not the tests. P2/P3: persisted-KV consent now flows through the full pipeline (declared change); provenance transition and mid-replacement fault tests; the recipe renamed minimal-divergence with its unavoidable divergences enumerated; batch-sync coverage is a gated stage with thresholds and a pause action; policy-revision activation defined (stamped revisions, bounded mixing, no tombstone resurrection); representation surface extended (Content-Type, ETag, Last-Modified, Accept-Ranges, digests); append restricted to list-valued headers; exact budgets and snapshot read semantics; reservation namespacing and per-state ownership conflicts; media-type matching ignores parameters; the pass-through test wording fixed; and a product-decision sign-off list (9 items) added to the migration spec for explicit maintainer ratification. --- ...26-07-30-client-cycle-ec-resolve-design.md | 42 +++- ...integration-response-header-hook-design.md | 60 ++++-- .../2026-07-30-permission-model-design.md | 126 ++++++++---- .../2026-07-30-pluggable-providers-design.md | 186 +++++++++++++----- ...07-30-provider-migration-rollout-design.md | 113 ++++++++--- 5 files changed, 398 insertions(+), 129 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index 2f27f1ab6..1eeb61e14 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -61,7 +61,11 @@ Everything in this spec follows from that. enough to set identity. Requests with no `Origin` and no valid token are rejected. 2. **Verify the payload cryptographically per provider — including against - replay.** The provider's `resolve_from_client` accepts only payloads + replay.** `resolve_from_client` is the client-resolve acquisition mode + of the provider contract (providers spec §4 + `Acquisition::ClientResolve`, which also carries the JS module the + page leg needs) — no longer an undeclared method this spec invents. It + accepts only payloads that are signed by an expected party, **audience-bound** to this publisher, and **expiring**. Audience binding and expiry alone do not mitigate replay — a captured token installs in another browser for the @@ -85,14 +89,20 @@ Everything in this spec follows from that. wiring; the parity suite asserts the endpoint's presence and behavior on all four adapters. (PR #838 registered it on Fastly only, so the same config on the Axum dev server proxied the POST to the publisher origin.) -6. **Be uncacheable and permission-gated.** `Cache-Control: no-store`; - the same `store-on-device` permission gate as organic EC creation runs - before any cookie is set. +6. **Be uncacheable and permission-gated on the provider's full + declaration.** `Cache-Control: no-store`; before any cookie is set, + the endpoint enforces the selected provider's complete + `required_permissions()` — not a hard-coded `store-on-device` check; a + client-resolve provider declaring more than P1 gets all of it + enforced, exactly as organic minting does (providers spec §5). 7. **Bound every input — exact values, testable boundaries.** Request body: at most **65,536 bytes** (inclusive; byte 65,537 → `413`), enforced by a bounded read independent of `Content-Length` — a missing, false, or chunked length must not bypass it. `Content-Type` - allowlist: `text/plain` and `application/json`; anything else → `415`. + allowlist: `text/plain` and `application/json`, matched on the media + type alone — case-insensitively, ignoring parameters, so the browser's + default `text/plain;charset=UTF-8` passes; duplicate `Content-Type` + headers → `400`; anything else → `415`. The resulting identifier: at most **256 bytes**, cookie-safe alphabet (providers spec §3 global bounds); violation → `400`. Reservation key (payload unique id): at most **128 bytes**. Status codes are part of @@ -124,8 +134,16 @@ Everything in this spec follows from that. its epoch no longer matches. `failed` is retryable: the same owner may supersede it with a fresh `pending` at a higher epoch. The graph write happens under the reservation and is deterministic under its key, so - any retry converges on the same row; reservations are retained at least - through the token's expiry. + any retry converges on the same row; reservations are retained at + least through the token's expiry. Reservation keys are **namespaced** + per the providers spec §6.3 grammar + (`resv////`), so + payloads cannot collide across publishers, providers, or versions. + Ownership conflicts are terminal per state: a non-owner hitting + `pending` gets `409` (retry only after lease expiry); a non-owner + hitting `committed`/`failed` gets the no-cookie terminal response; an + owner hitting `failed` may supersede it (higher epoch); cleanup + deletes reservations after retention, never before token expiry. **A duplicate must never receive the cookie unless it proves the original session binding.** "Duplicates observe the recorded outcome" @@ -139,9 +157,13 @@ Everything in this spec follows from that. explicitly-accepted at-most-once posture (no owner hash), a lost response is an **orphan row** handled by the specified cleanup path. The **same-identity no-op of §3.8 first checks the family revocation - record** (permission model spec §4.3): a resolve against a revoked - family is rejected, never refreshed, and a create racing a revocation - loses — revocation wins. Tests cover crash-between-steps, lease + record** (permission model spec §4.3), and it does so **through the + linearizable primitive class this feature already requires** for + reservations (providers spec §7 matrix) — which is what makes + "revocation wins" true rather than aspirational: on an eventually + consistent read, a racing create could observe a stale absence and + emit a cookie for a revoked family. A resolve against a revoked family + is rejected, never refreshed, and a create racing a revocation loses. Tests cover crash-between-steps, lease takeover with a stale-epoch commit attempt, two concurrent requests with the same payload, a duplicate from a second client receiving no cookie, owner-hash recovery receiving the cookie, and diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 8ec5b687d..23d1241f2 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -61,9 +61,17 @@ mutators to the outbound response for HTML document responses it processed. merely passed through) — and the post-hook response may only be **equal or stronger** on the privacy axis: integrations can tighten caching, never loosen it, regardless of which header they replaced. - Every CDN/surrogate cache directive (`Surrogate-Control`, - `CDN-Cache-Control`, host-specific equivalents) is stripped from any - restricted response. Middle-stage placement also keeps + "Equal or stronger" is a defined merge, not a vibe: restriction + strength is ordered `no-store` > `no-cache` > `private` > `public`, + and the final value per axis is the **stronger of snapshot and + mutation**; `max-age`/`s-maxage` may only shrink relative to the + snapshot; `stale-while-revalidate`/`stale-if-error` may appear only if + the snapshot had them; every CDN/surrogate directive + (`Surrogate-Control`, `CDN-Cache-Control`, host-specific equivalents) + is stripped from any restricted response; and **core-required `Vary` + members are protected in the contract, not just the tests** — the + final `Vary` is the union of the snapshot's required members and the + mutation. Middle-stage placement also keeps the earlier property: an integration mutation is not silently stripped by ordinary core handling — only by the invariant pass, which logs the downgrade it applies. @@ -75,18 +83,31 @@ mutators to the outbound response for HTML document responses it processed. _names_ — HTTP framing and hop-by-hop headers (`Content-Length`, `Transfer-Encoding`, `Connection`, `Trailer`, `Upgrade`, `TE`, `Keep-Alive`), **representation headers coupled to body bytes the hook - cannot see** (`Content-Encoding`, `Content-Range` — relabeling - uncompressed bytes as Brotli, or stripping the encoding from compressed - bytes, corrupts the response), the `x-ts-*` namespace, and the + cannot see** (`Content-Encoding`, `Content-Range`, `Content-Type`, + `ETag`, `Last-Modified`, `Accept-Ranges`, and digest headers — + relabeling uncompressed bytes as Brotli, or advertising a validator or + digest for bytes the hook never saw, corrupts responses or poisons + caches), the `x-ts-*` namespace, and the consent/privacy headers core emits; (b) reserved cookie _names_ within `Set-Cookie` — `ts-ec`, - `ts-eids`, and the other `ts-*` cookies core owns. An integration may - append its own `Set-Cookie` values; it may not set or expire a reserved - cookie name. Violations are rejected at the operation layer (§2) and + `ts-eids`, and the other `ts-*` cookies core owns. Integration cookies are **inside the permission model, not beside it** + (product sign-off item 9, migration spec §8) — otherwise the hook is a + door around the EC gate: an integration could write a durable + identifier while `store-on-device` is denied. `append_set_cookie` + therefore requires the cookie name to be **declared at registration** + with a stated purpose and maximum retention; a **persistent** cookie + (any `Max-Age`/`Expires`) is applied only when the request's resolved + permissions include `store-on-device`, while **session cookies** (no + persistence attributes) are the narrow, documented exemption. + Undeclared cookie names are rejected like reserved ones. An integration + may never set or expire a reserved cookie name. Violations are rejected at the operation layer (§2) and logged at `warn` with the integration id. The reserved lists are single constants next to the definitions they protect, not duplicated in the hook. - For non-reserved headers, the mutator API distinguishes **append** from - **replace** explicitly; the default is append (for `Set-Cookie`, append is + **replace** explicitly; **append is valid only for genuinely + list-valued headers** (a singleton header accepts only replace — two + values of a singleton header by append is a malformed response, not a + merge); the default is append where legal (for `Set-Cookie`, append is the only non-reserved operation — replace is not offered). Replacing a header the origin set is a deliberate act, visible in the mutator's code. - Later registrations see earlier mutations (order = registration order, @@ -95,10 +116,15 @@ mutators to the outbound response for HTML document responses it processed. `Set-Cookie` header name outright — cookies go only through `append_set_cookie`, so its validation cannot be bypassed by spelling the header name in a generic op. Per-integration limits bound total - operations, added header count, and added header bytes, and a - **cumulative final-response budget** (total header count and bytes) - bounds the sum across integrations — enforced in registration order, so - which operations are rejected when the budget trips is deterministic. + operations (≤ 32), added headers (≤ 16), and added bytes (≤ 8 KiB), and + a **cumulative final-response budget** (≤ 128 headers / ≤ 32 KiB total, + counting `name: value` plus separators, within any lower adapter + ceiling) bounds the sum across integrations — enforced in registration + order, so which operations are rejected when a budget trips is + deterministic. Each mutator receives an **immutable snapshot of the + response head** (status and headers as of its turn, prior integrations' + accepted operations applied) as its read context; it never holds a + mutable reference (§2). Exceeding a limit rejects the excess operations (logged, attributed), never the response. A mutator that returns an error is skipped in full — its operations are all-or-nothing — and the response proceeds without it. @@ -145,9 +171,9 @@ processed documents (§6). 7. Cache/privacy invariant tests, one per restriction source and shape: cookie appended + public `Cache-Control` replacement → private/no-store, surrogate stripped; **core-private cookieless** processed HTML + - public replacement → restriction preserved; **origin-private - cookieless** pass-through-classified content + public replacement → - restriction preserved; a cache-hit serve re-applying mutations without + public replacement → restriction preserved; **origin-private cookieless** processed HTML that retained the + origin's cache restrictions + public replacement → restriction + preserved (pass-through responses never run the hook, §3a); a cache-hit serve re-applying mutations without weakening the stored classification; a `Vary` mutation neither dropping core-required values nor bypassing the snapshot; each CDN directive (`Surrogate-Control`, `CDN-Cache-Control`, host equivalents) diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index f47f6673d..6154de7c7 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -251,9 +251,12 @@ Signals are classified into three classes — a two-class model (TCF grant / opt-out) cannot reproduce today's US behavior, where no-signal traffic is blocked but an **explicit non-opt-out** value grants: -- **Opt-out signals** (affirmative withdrawal): GPC header; GPP sections - carrying a sale/sharing opt-out; US Privacy opt-out. Opt-out signals are - honored **globally**, not only in the jurisdictions whose law defines +- **Opt-out signals**, in two subclasses assigned by the §4.5 mapping: + **destructive** opt-outs (GPC; sale opt-outs; USP opt-out) revoke and + trigger withdrawal; **non-destructive** opt-outs (sharing, + targeted-advertising) revoke the permissions they map to but never + destroy the stored identity — a targeted-ads choice must not tombstone. + Both subclasses are honored **globally**, not only in the jurisdictions whose law defines them — a deliberate, more-protective simplification: scoping a browser's explicit opt-out to a geolocation guess would honor it for some visitors and ignore it for others based on IP evidence. (For jurisdictions outside @@ -347,9 +350,12 @@ group label, since a group can mix rules across permissions. The triggers, exhaustively — nothing else withdraws: -1. **An opt-out signal withdraws in every jurisdiction, whatever the - baseline.** (For US states this preserves today's behavior; elsewhere it - is the declared change of §4's global-opt-out rule.) +1. **A destructive opt-out signal (per §4.5's destructive column: GPC, + sale opt-outs, USP opt-out) withdraws in every jurisdiction, whatever + the baseline.** Non-destructive opt-outs (sharing, + targeted-advertising) never trigger this — they revoke acquisition + only. (For US states this preserves today's behavior; elsewhere it is + the declared change of §4's global-opt-out rule.) 2. **A TCF record refusing `store-on-device` withdraws iff the baseline is `requires_signal` or `denied`.** Where the baseline is `granted`, refusal blocks _new_ grants but never tombstones: tombstones are @@ -445,19 +451,19 @@ valid sources.** Current runtime resolves conflicts first and can select an expired record before clearing both sources; expiry-first is a **declared change** (migration matrix) that removes that path: -| Input state | Effective record / outcome | Status | -| ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | -| Standalone TCF and GPP-embedded TCF disagree, mode `restrictive` | Whole-record selection over the **(P1, P4) outcome tuple, compared lexicographically with P1 first** (refusal < grant): the lesser tuple governs in full. Split-purpose records are thereby decided — (grant P1, refuse P4) vs (refuse P1, grant P4) selects the latter. Identical tuples → outcomes are identical; the GPP-embedded record is named for determinism. (An earlier draft specified per-purpose synthesis, which is _not_ what the code does; the tuple order is decided here and pinned against current tests) | Preserved — pinned against current tests | -| Same, mode `permissive` | Same tuple comparison; the **greater** tuple governs in full | Preserved — same pinning | -| Same, mode `newest` | Whole-record selection by **`LastUpdated`** (not `Created`), subject to the existing freshness threshold; a tie, an incomparable pair, or timestamps inside the threshold fall back to the fully deterministic `restrictive` rule above | Preserved — same pinning | -| Expired consent record | Treated as **absent entirely** — grants nothing, refuses nothing, withdraws nothing; the baseline applies. Under a `granted` baseline that means the grant stands: an expired refusal is not current evidence and must not revoke indefinitely | Preserved | -| One valid record + a second malformed record of the same family | The **valid record governs**; the malformed one is ignored with a `warn` log. Fail-closed-on-malformed (below) applies only when no valid record of that family exists | Decided here | -| One valid record + one **expired** record of the same family | The valid record governs — the expired one dropped at pipeline step 2, before conflict resolution ever saw it | **Changed (declared)** — current runtime resolves the conflict first and can select the expired record | -| Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | -| Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | Preserved, circularity resolved | -| Proxy/mirror mode | **Syntax validation still runs; semantic decoding is skipped.** A present record (well- or mal-formed) is _present, undecoded_: it blocks grants (a record TS will not read cannot vouch for consent) and never withdraws; absent → baseline. Header-carried opt-outs (GPC) are unaffected — they need no decoding. Without the syntax pass, malformed-present would be indistinguishable from absent, contradicting the fail-closed rule below | **Changed (declared)**: today proxy mode skips decoding entirely, which under a permissive baseline is fail-open | -| GPP / US Privacy fields | Per the normative field mapping of §4.5 — fields are not interchangeable signals, and absent/N-A fields grant nothing | Decided here (§4.5) | -| Malformed-but-present record, no valid record of that family | **Blocks grants** (fail-closed acquisition — it does not degrade to "absent", which under a `granted` baseline would turn garbage into a grant, the fail-open path in both #838 and the first draft of this spec). Never triggers withdrawal — destruction requires an affirmative, decodable signal (§4.2) | Changed (declared) | +| Input state | Effective record / outcome | Status | +| ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Standalone TCF and GPP-embedded TCF disagree, mode `restrictive` | **Whole-record selection comparing the P1 ∧ P4 conjunction only** — today's algorithm, preserved (an earlier draft's lexicographic (P1, P4) tuple would have changed split-purpose outcomes): if exactly one record's conjunction is false, `restrictive` selects it; **equal conjunctions — including split-purpose disagreements — keep the standalone record**, as current code does | Preserved — pinned against current tests | +| Same, mode `permissive` | Same conjunction comparison, selecting the record whose conjunction is true; equal conjunctions keep the standalone record | Preserved — same pinning | +| Same, mode `newest` | Whole-record selection by **`LastUpdated`** subject to the existing freshness threshold; a tie, an incomparable pair, or timestamps inside the threshold fall back to the `restrictive` rule above (itself deterministic) | Preserved — same pinning | +| Expired consent record | Treated as **absent entirely** — grants nothing, refuses nothing, withdraws nothing; the baseline applies. Under a `granted` baseline that means the grant stands: an expired refusal is not current evidence and must not revoke indefinitely | Preserved | +| One valid record + a second malformed record of the same family | The **valid record governs**; the malformed one is ignored with a `warn` log. Fail-closed-on-malformed (below) applies only when no valid record of that family exists | Decided here | +| One valid record + one **expired** record of the same family | The valid record governs — the expired one dropped at pipeline step 2, before conflict resolution ever saw it | **Changed (declared)** — current runtime resolves the conflict first and can select the expired record | +| Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | +| Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**, then flows through the full normalization pipeline (syntax, expiry, conflict) like any live record; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | **Changed (declared)**: current code returns immediately after the KV load, bypassing expiry and conflict normalization | +| Proxy/mirror mode | **Minimal opt-out extraction still runs; full semantic decoding is skipped.** Because opt-outs are globally authoritative (§4), proxy mode must not suppress them: the §4.5-mapped opt-out fields (GPP US sections) and the US Privacy string are decoded — nothing else — alongside syntax validation, so a valid SaleOptOut or USP opt-out revokes and withdraws exactly as outside proxy mode. No grants are ever derived from records in proxy mode; a present record otherwise blocks grants (fail-closed); absent → baseline. GPC needs no decoding | **Changed (declared)**: today proxy mode skips decoding entirely — fail-open under permissive baselines and, worse, opt-out-blind | +| GPP / US Privacy fields | Per the normative field mapping of §4.5 — fields are not interchangeable signals, and absent/N-A fields grant nothing | Decided here (§4.5) | +| Malformed-but-present record, no valid record of that family | **Blocks grants** (fail-closed acquisition — it does not degrade to "absent", which under a `granted` baseline would turn garbage into a grant, the fail-open path in both #838 and the first draft of this spec). Never triggers withdrawal — destruction requires an affirmative, decodable signal (§4.2) | Changed (declared) | ### 4.5 US signal field mapping — normative @@ -480,13 +486,36 @@ a change to this table. | US Privacy · present, `N` or N/A | — | grant | grant | — (today's tests pin N/A as allowing; USP carries no distinct targeted-advertising field, so it never maps to one) | | Any field | absent / N-A | — | — | — | -**Multi-section aggregation:** when both a national and an applicable -state section are present, the state section governs for the fields it -carries; across whatever sections apply, **an opt-out in any applicable -section beats a grant in another** (restrictive aggregation). `SharingOptOut` -and `TargetedAdvertisingOptOut` are new enforcement inputs — current code -consults only the sale field — and are declared as such in the migration -matrix. +**N/A vs absent:** a field explicitly set to _Not Applicable_ is treated +as not-opted-out (grant-class) — pinned by today's USP tests and matching +current GPP `NotApplicable` handling, and declared as such since an +earlier draft said N/A contributes nothing. A field **absent** from an +applicable section, or any field of a non-applicable section, contributes +nothing. + +**Applicability and aggregation — ordered algorithm:** + +1. **Section map (normative, pinned here — not "whatever GPP is current"):** + `US` national ↔ GPP section 7 (usnat); `US/CA` ↔ 8 (usca); `US/VA` ↔ 9 + (usva); `US/CO` ↔ 10 (usco); `US/UT` ↔ 11 (usut); `US/CT` ↔ 12 (usct). + Section versions are those published at this spec's date; adding a + section or version is a change to this map. +2. **Determine applicability from the resolved jurisdiction:** the + national section is applicable to any `us-privacy`-regime request; a + state section is applicable iff it maps to the resolved `US/`. + Foreign-state sections (a `usca` string on a `US/CO` request) and all + sections on non-`us-privacy` requests are **not applicable** and + contribute nothing. Regionless US traffic: national section only. +3. **State-over-national, per field:** where an applicable state section + carries a field, it governs that field; the national section fills only + fields the state section lacks. +4. **Aggregate across what remains applicable:** an opt-out (of either + subclass) in any applicable field beats a grant from another — + restrictive aggregation. + +`SharingOptOut` and `TargetedAdvertisingOptOut` are new enforcement +inputs — current code consults only the sale field — and are declared as +such in the migration matrix. ## 5. Jurisdiction resolution @@ -549,6 +578,22 @@ different states, and pre-epic behavior treated them differently (fail closed vs. non-regulated) — collapsing them is what made PR #838's migration story unresolvable (migration spec §2, rows 5 and 7). +### 5.5 Policy revision activation + +A policy edit propagates through the config store, so a fleet briefly +mixes revisions. The contract: instances stamp every resolution and every +provenance write with the policy revision they used (already required by +§7); the mixing window is bounded by config propagation and observable via +the config-version metric; and mixed revisions cannot cause irreversible +harm, because **destructive withdrawal triggers are user signals, never +policy** (§4.2 trigger 3) — the one revision-sensitive destructive case +(trigger 2 under a now-`denied` baseline) requires an affirmative user +refusal at the evaluating instance, which is safe under either revision. +S2S recomputation always evaluates against the instance's current +revision and records it. Rolling a policy revision back restores +acquisition rules but **cannot resurrect tombstoned identities**; the +migration guide says so where operators will read it. + ## 6. Failure-mode matrix — normative | Condition | Resolution behavior | @@ -620,13 +665,21 @@ Consumers of the resolved set in this epic: (which signal class granted, per permission), the evidence's **authoritative timestamp and `valid_until`** (per evidence class), resolved jurisdiction, policy revision, and provider/version (providers - spec §6.1). Two aging rules prevent perpetual renewal: **re-presenting - an unchanged signal does not reset evidence age** — only a record - carrying a newer authoritative timestamp (e.g. TCF `LastUpdated`) does; - and every live resolution **atomically replaces the complete - per-permission snapshot**, never merges — a refusal, opt-out, malformed - or absent state in the fresh resolution clears prior positive authority - for its scope, so an old P4 grant cannot survive a later P4 refusal. A sync request performs a **full recompute of both + spec §6.1). Freshness is a **per-evidence-class contract**, because not + every source carries a timestamp: + + | Evidence class | Authoritative timestamp | Age reset | Max age | + | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------- | + | TCF consent | The record's `LastUpdated` | Only a record with a **newer** `LastUpdated` | Existing TCF expiry TTL | + | GPP / USP values (no intrinsic timestamp) | **First-seen**: when TS first observed this exact normalized value (equality digest stored in provenance) | Re-presenting an identical digest **keeps the original first-seen**; a different value is new evidence with a new first-seen | Consent TTL (same as TCF) | + | Policy-baseline grant (`granted` rule, no signal) | The policy revision that granted | Re-derived on every recompute against the current revision — policy is not user evidence and does not age; it changes | n/a | + + Timestamps are compared with bounded clock-skew tolerance and + future-dated values are clamped to receipt time. And every live + resolution **atomically replaces the complete per-permission + snapshot**, never merges — a refusal, opt-out, malformed or absent + state in the fresh resolution clears prior positive authority for its + scope, so an old P4 grant cannot survive a later P4 refusal. A sync request performs a **full recompute of both permissions** from that stored evidence against the _current_ policy: it fails closed when the stored jurisdiction's rule is now `denied`, when a `granted` baseline tightened to `requires_signal` and the stored @@ -688,7 +741,12 @@ further consumer if and when it proceeds. reset), and legacy-row fail-closed-then-backfill. - The full cross-product **regime × permission × evidence source** from §4's acceptance table and §4.5's field mapping, including multi-section - aggregation conflicts. + aggregation conflicts and the applicability algorithm's foreign-section + and regionless rows. +- Provenance snapshot-replacement transitions per permission: prior grant + → refusal, → opt-out, → malformed, → absent — plus a mid-replacement + fault proving the surviving state is the complete old **or** complete + new snapshot, never a merged mixture. - Legacy-row withdrawal end to end (§4.3's derived family ID). - §4.3 fault-injection cases. - Policy validation tests for every §3.3 rejection, exercised through both diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index 8c934bc31..2d3d58e58 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -98,7 +98,7 @@ through the selected provider: | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Mint** | EC generation on first eligible request | Provider returns the identifier (and only core writes the cookie). | | **Parse / canonicalize** | Reading `ts-ec` back from the request; deciding `ec_was_present`; batch-sync ingestion | Provider parses a cookie value into its **canonical** identifier, or rejects it. Canonicalization and **equivalence are provider-declared, never imposed globally**: each provider ships equivalence fixtures naming exactly which variants are the same identity — case sensitivity is provider-specific (signed/base64-style envelopes are case-sensitive; even the built-in HMAC id is case-insensitive only in its hex prefix, with a case-preserved suffix). Declared-equivalent values parse to the same canonical identifier (satisfying #778). A value the selected provider does not recognize is treated as absent (but see §6.1 legacy readers). | -| **Canonical graph key** | KV identity-graph row reads/writes | The provider maps a canonical identifier to its graph key: stable, KV-safe (within KV length and character-set limits), collision-free across the provider's identifier space, and namespaced so two providers' key spaces cannot collide. Two equivalent envelopes of one identity map to one key — verbatim cookie bytes as the key would fork graph rows on canonicalization differences and discard today's batch-sync canonicalization. | +| **Canonical graph key** | KV identity-graph row reads/writes | The provider supplies a canonical key **suffix**; **core constructs the physical key** per the §6.3 key grammar (legacy-HMAC verbatim keys excepted), so cross-provider and cross-record-kind isolation is enforced by construction rather than promised by provider code. Suffixes are stable, KV-safe (length and character-set limits), and collision-free within the provider's space. Two equivalent envelopes of one identity map to one key — verbatim cookie bytes as the key would fork graph rows on canonicalization differences and discard today's batch-sync canonicalization. | | **Cluster prefix** (optional capability) | IP-cluster sizing (`cluster_trust_threshold`, implemented as a **KV prefix listing**), pull-sync dedupe, log redaction | A provider declaring cluster support returns a prefix that is a **literal byte prefix of the canonical graph key** — the cluster count lists keys by prefix, so an independently derived hash that is not an actual key prefix silently reports the wrong cluster size. The prefix deliberately collides across identifiers minted from the same client evidence. A provider without the capability declares so, and cluster-dependent gating follows a configured degradation policy (treat cluster size as unknown, with the KV-write decision that implies made explicit in config) instead of counting garbage. | | **Tombstone** | Withdrawal: expiring the cookie and writing revocation markers | The identifiers eligible for tombstoning are exactly those the provider parses — never a shape-gated subset. | @@ -176,19 +176,33 @@ pub trait EdgeCookieProvider { /// Permissions this provider's data use requires. Enforced by core for /// minting and identity use — never for parse/tombstone (§5). fn required_permissions(&self) -> PermissionSet; - /// Mint an identifier from request evidence. - fn generate(&self, input: &IdentityInput<'_>) -> Result>; /// Parse and canonicalize a cookie value into this provider's - /// identifier; None when unrecognized. Values the provider's declared + /// identifier; None when unrecognized. Identifies the provider + /// NAMESPACE only — never a configuration version (§6.1: versions + /// resolve from row provenance). Values the provider's declared /// equivalence fixtures name as equivalent canonicalize identically. fn parse(&self, value: &str) -> Option; - /// Canonical KV graph key for a parsed identifier. - fn graph_key(&self, id: &EcId) -> GraphKey; - /// Cluster capability: a literal byte prefix of `graph_key(id)`, shared - /// across identifiers minted from the same client evidence. None when - /// the provider does not support IP-cluster semantics (§3). + /// Canonical graph-key SUFFIX (bounded length, KV-safe). Core — not + /// the provider — constructs the physical key (§6.3 key grammar), so + /// cross-provider and cross-record-kind isolation is structural. + /// Sole exception: hmac v0 keys are the identifier verbatim. + fn graph_key_suffix(&self, id: &EcId) -> GraphKeySuffix; + /// Cluster capability: a literal byte prefix of the physical graph + /// key, shared across identifiers minted from the same client + /// evidence. None when the provider lacks IP-cluster semantics (§3). fn cluster_prefix(&self, id: &EcId) -> Option; + /// Acquisition mode — exactly one: + fn acquisition(&self) -> Acquisition<'_>; } + +/// How a provider's identifiers come into being. Server-mint providers +/// generate from request evidence; client-resolve providers verify a +/// browser-posted payload (client-cycle spec) and declare the JS module +/// their page leg needs. One provider implements exactly one mode. +pub enum Acquisition<'a> { + ServerMint(&'a dyn ServerMint), // fn generate(&IdentityInput) -> EcId + ClientResolve(&'a dyn ClientResolve),// fn resolve_from_client(&Payload) -> EcId +} // + fn js_module_id() -> &str ``` (Names indicative; the shape is normative. `required_permissions` joins the @@ -225,10 +239,14 @@ error logged, the next request retries. (permission model spec §7) is a backstop, but conventions do not survive new code — the ungated proxy/click/Testlight paths happened precisely because raw EC values circulate as ordinary strings. Core therefore -introduces an **`AuthorizedIdentity`** newtype constructible only by core, -only after parse + permission gate + graph/family-revocation check; -outbound serializers (ORTB builder, page bids, sync, identify, forwarding) -accept `AuthorizedIdentity`, never `&str`/`EcId`. A future bypass then +introduces a **scope-parameterized `AuthorizedIdentity`**, +constructible only by core, only after the checks _for that exact scope_: +`AuthorizedIdentity` after parse + `store-on-device` + +family-revocation check; `AuthorizedIdentity` additionally +after `select-personalised-ads`. Outbound serializers (ORTB builder, page +bids, sync, identify, forwarding) accept `AuthorizedIdentity` +and nothing weaker — an unparameterized wrapper would let a P1-only +identity flow into an ORTB request. A future bypass then requires deliberately reconstructing the raw string — visible in review — rather than passing along what was already in hand. @@ -247,8 +265,11 @@ structural: JA4/HTTP-2 fingerprints) for **security classification** — the bot gate protecting KV-backed identity writes — which must run precisely for traffic that has granted nothing. The authorization for that processing - is the operator's explicit `[device] provider` selection, and this spec - records that as the decision, with its privacy implication stated: a + is the operator's explicit `[device] provider` selection — a statement + about the **opt-in host-fingerprint provider**; the `builtin` UA-only + default processes nothing beyond the User-Agent every request already + carries and needs no such authorization — and this spec records that as + the decision, with its privacy implication stated: a device provider whose data use goes beyond security classification (for example feeding fingerprints into targeting or identity) is **not authorized by selection alone** and requires a vocabulary extension plus @@ -321,35 +342,55 @@ The contract: Same-provider key/passphrase rotation is configuration, not a provider switch: a provider block may hold multiple `versions` entries (`[ec.providers.hmac.versions.v2] passphrase = …`) with - `mint_version = "v2"` selecting the writer; `parse` consults versions in - declared order, newest first; removing a version entry is a retirement - subject to the same evidence rules as retiring a legacy reader - (migration spec §6). + `mint_version = "v2"` selecting the writer. **`parse` cannot identify a + version** — every HMAC version shares one grammar and `parse` returns no + version — so the mint version lives in **immutable row provenance** + (rows without a tag are `hmac-v0`), and cryptographic verification + consults configured versions newest-first only where provenance is + unavailable (a cookie with no reachable row). Removing a version entry + is a retirement subject to the same evidence rules as retiring a legacy + reader (migration spec §6). - A cookie recognized by a legacy reader is a live identity for read/withdrawal purposes; whether it is transparently re-minted under the active writer is a per-deployment choice (`[ec] rewrite_legacy = true|false`), and re-minting is subject to the full minting gate of §5. -- **Rewrite aliases to one canonical row — no dual-write window.** Order: - the new canonical row commits first, sharing the old row's revocation - family ID (permission model spec §4.3); its provenance is the **current - live resolution** (rewrite happens on a live request — copying old - consent evidence would rejuvenate stale authority), while partner - mappings copy **with their original per-field timestamps and expiry**. - Then a fenced CAS **replaces the old row with an alias record** pointing - at the canonical row; from that moment every read or update through - either cookie chases the alias (single hop) to the one canonical row — - a concurrent pull/batch/identify update cannot land on a row about to - be discarded, because after the CAS there is only one row to land on, - and an update racing the CAS itself retries against the canonical. Then - the new cookie is emitted. The server cannot observe `Set-Cookie` - acceptance, so the **alias stays live** until a later request presents - the new cookie (confirmation by presentation), and in any case until a +- **Rewrite is a persistent fenced transaction aliasing to one canonical + row — no dual-write window, no duplicate targets, no lost updates.** + The steps, each resumable because the transaction record (its own + linearizable record class, §7 matrix) is written **first** and pins the + chosen target key and fencing epoch: + 1. **Transaction record** commits: source key, target key, epoch, + state. A crashed rewrite retried later reads it and resumes with + the **same** target — a fresh random target (and an orphaned first + one) cannot exist, and any target row without a committed transaction + pointing at it is garbage-collectable by that absence. + 2. **Canonical row** commits under the pinned target key, sharing the + old row's revocation family ID (permission model spec §4.3); + provenance is the **current live resolution** (copying old consent + evidence would rejuvenate stale authority); partner mappings copy + with their **original per-field timestamps and expiry**, and the + copy point is recorded in the transaction. + 3. **Fenced CAS replaces the old row with an alias record** targeting + the canonical. If the CAS loses to a concurrent pull/batch/identify + update, the rewrite **re-runs a reconciliation pass** under its + epoch — merging updates newer than the recorded copy point into the + canonical — and retries the CAS; an update that won the old row is + therefore never lost. + 4. The new cookie is emitted; the transaction marks complete. + + From step 3 on, every read or update through either cookie chases the + alias (one hop) to the single canonical row. **Chains stay single-hop**: + a later rewrite B→C retargets every alias pointing at B (the canonical + row records its inbound aliases; alias records are in the linearizable + class, so retargeting is fenced) so A points directly at C. The server + cannot observe `Set-Cookie` acceptance, so the alias stays live until a + later request **presents the new cookie**, and in any case until a **finite retirement deadline** no shorter than the old cookie's maximum - lifetime plus rollout skew. An interrupted rewrite leaves the old - cookie resolving (directly or via the alias) — no state in which - neither identity works. **Withdrawal through either cookie revokes the - shared family.** + lifetime plus rollout skew. An interrupted rewrite at any step leaves + the old cookie resolving — no state in which neither identity works. + **Withdrawal through either cookie revokes the shared family.** + - Retiring a legacy reader is the explicit end of those identities: the migration guide documents the cleanup procedure (migration spec §6). - Tests: switch active provider → request with old cookie → identity still @@ -391,7 +432,49 @@ egress and sync updates fail closed; organic requests continue stateless. The thresholds ship as constants with the implementation and are printed in the startup log. -### 6.3 Graph row contract — normative +Its protection is therefore **local-only, and the spec says so**: a +backend-wide outage degrades every instance through its own observations +within one window, but an instance-local family-write failure leaves +other instances — which have no record to find, and healthy backends of +their own — serving S2S egress until the browser's durable signal retries +successfully. That residual is bounded by the user's return latency, is +counted (failed family writes are a first-class metric), and is accepted +in place of a deployment-wide shared fail-closed channel, whose own +availability and freshness would be a harder problem than the one it +solves. + +### 6.3 Storage contract — normative + +**Physical key grammar.** Core constructs every key; providers supply only +the bounded suffix: + +| Record class | Key | Notes | +| ----------------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------- | +| Identity row (v2+) | `id///` | Suffix from `graph_key_suffix`, ≤ 128 bytes, KV-safe alphabet | +| Identity row (legacy hmac-v0) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar; no other class or provider may produce a matching key | +| Alias | `alias///` | Same suffix as the row it replaced | +| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | +| Rewrite transaction | `rwx/` | One in-flight rewrite per family | +| Replay reservation | `resv////` | Client-cycle spec; payload id ≤ 128 bytes | + +Grammars are pairwise non-intersecting by their literal prefixes (plus the +reserved legacy grammar), which is what makes cross-class collision +impossible rather than unlikely. + +**Wire schemas** (JSON, like identity rows; every class carries a schema +version): the **alias record** holds target key, created-at, retirement +deadline, and fencing epoch; the **family revocation record** holds the +family ID, revoked-at, triggering signal class (§4.5 destructive column), +and epoch — deliberately no identity data, so it can outlive its members; +the **rewrite transaction** holds source key, target key, copy point, +state, and epoch; the **reservation** holds state, owner hash, lease +epoch, outcome, and created-at (client-cycle spec). Field validation and +TTLs: aliases live to their retirement deadline; family records to the +§7 retention rule (beyond every member, cookie, rewrite, and retry +lifetime); transactions to completion plus an audit window; reservations +at least through token expiry. + +#### Graph row contract The per-field contract for identity rows, covering today's v1 fields and the fields this epic adds. Serialization is JSON with the existing `v` @@ -438,16 +521,29 @@ Requirements: **per-record-class consistency requirements**, because "has KV" says nothing about whether revocation is observable: - | Record class | Required semantics | - | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | - | Family revocation records | Strongest read the backend offers, plus a **declared, bounded revocation-visibility lag** (Workers KV documents up to ~60 s eventual propagation — that bound must be declared, and the residual it implies stated in operator docs). Unboundable lag → startup failure for identity features. A **failed or erroring revocation-record read fails closed** for egress | - | Identity rows | Eventual consistency acceptable — rows are accretive, and the family-record check (strong, per above) governs use | + | Record class | Required semantics | + | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | + | Family revocation records | **Strongly consistent (read-after-write) primitive required.** Cloudflare Workers KV is **not eligible** — its documentation says propagation may take "60 seconds or more", an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | + | Alias / rewrite-transaction records | **Linearizable fenced CAS required** (same primitive class as reservations). `rewrite_legacy = true` is rejected at startup on adapters lacking it (§6) | + | Identity rows | Eventual consistency acceptable — rows are accretive, and the family-record check (strong, per above) governs use | Each adapter's declaration is part of its wiring, drives the §6 capability-mismatch startup error, and every §6.2 runtime-failure row gets fault-injection coverage on every adapter declaring the - corresponding capability. + corresponding capability. The **concrete per-adapter values** — the + actual matrix, not the abstract capability list — as known today; a + cell marked _verify_ must be established before the depending feature + is selectable on that adapter, and the filled matrix is normative: + + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | + | Graph persistence (eventual OK) | KV Store: yes | Local store: yes (dev-grade) | Workers KV: yes (eventually consistent) | Key-value: yes | + | Prefix listing (cluster) | Yes (used today) | Yes | Yes (eventual) | _verify_ | + | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | Yes (in-process) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | + | Linearizable fenced CAS (reservations, alias/rewrite) | **Not currently available** — client-cycle and `rewrite_legacy` unselectable until a primitive exists | Yes (in-process) | Durable Objects: possible, not wired | **No** | + | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | + | Device host evidence (JA4/H2) | Yes | No | No | No | - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index eaf6bb133..54f7fe4d4 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -54,6 +54,7 @@ discoverable only because a deleted test had pinned the old behavior. | 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | | 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | | 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | +| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -118,16 +119,32 @@ Requirements: after **fleet convergence on N+1 is confirmed** — binaries first, convergence gate, then `ts config push`. A config mixing old and new fields (`[ec] passphrase` alongside `[ec] provider`) is **rejected** - by N+1, not reconciled. Rollback runs the sequence in reverse: config - back to the old shape first, binaries only after config convergence. - Every new config section introduced by the epic follows this same - compatibility rule, not only `[ec]`. + by N+1, not reconciled. **Rollback is binaries-first too, in the + other direction**: N+2 → N+1 binaries roll back **keeping the new + config** (N+1 reads it fully — reverting config first would hand the + old shape to N+2 binaries that reject it). N+1 additionally + **rejects provider or version selections whose provenance it cannot + yet encode** — new-provider adoption waits for N+2, so no row is + minted that N+2 would misclassify. Every new config section + introduced by the epic follows this same compatibility rule, not + only `[ec]`. - **Release N+2:** rejects `[ec] passphrase` at startup with a message naming the new location — not a generic unknown-field error (implementation note: producing the actionable message means keeping a deprecated `passphrase` field whose presence triggers the custom error). -2. **The graph schema change is expand-contract, in lockstep with the +2. **Graph-store readiness precedes everything.** Today the graph store + is optional and EC generation succeeds without one; the epic's + no-active-until-commit invariant (providers spec §5) makes it + mandatory wherever a minting provider is configured — so a currently + valid graphless HMAC deployment would **startup-fail on N+1's + dual-read mapping** without a preparatory step. The migration + therefore begins with a **pre-N+1 readiness step**: provision and + verify an openable graph store (and confirm the adapter's capability + row supports the features in use, providers spec §7) _before_ rolling + N+1. This is a **declared breaking change** for graphless deployments + (matrix row 12), not a side effect discovered at boot. +3. **The graph schema change is expand-contract, in lockstep with the binary sequence.** New rows carry fields v1 rows never had — provider/ version, per-permission grant evidence, policy revision, family ID, rewrite links — and two failure modes must be engineered away: a naive @@ -153,49 +170,62 @@ Requirements: new fields untouched if read-only, preserved semantically if read-modify-write on N+1, **test-proven lost on pre-N+1** (documenting why the floor is a floor); N+2-reader/N+1-written-row → full function. -3. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no +4. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no `provider = "hmac"` selector is a startup error (providers spec §6). In PR #838 this configuration — the exact state an operator following the docs reaches if they miss one line — validated green and silently minted zero ECs. -4. **PR #838-era keys fail loud.** `provider = "host-signals"` (shipped by +5. **PR #838-era keys fail loud.** `provider = "host-signals"` (shipped by PR #838, deliberately not carried into this epic — providers spec §2) and `provider = "client-fixed"` are unknown keys and rejected like any other, so a config written against the PR #838 example cannot silently select a provider that no longer exists. -5. **Provider switches go through legacy readers.** Changing +6. **Provider switches go through legacy readers.** Changing `[ec] provider` on a deployment with live identities requires listing the outgoing provider in `[ec] legacy_providers` (providers spec §6.1) so existing cookies keep resolving and stay withdrawable; the guide documents the switch sequence and the retirement/cleanup step that ends it. -6. **The example config ships the migrated happy path**, uncommented: +7. **The example config ships the migrated happy path**, uncommented: `provider = "hmac"` with its block, `[geo] default_country`, and (for Fastly) the behavior-preserving `[device] provider = "fastly"` and `[geo] provider = "platform"` lines present with a comment stating what removing them changes. PR #838's example shipped the passphrase block uncommented with the selector commented out — steering operators directly into the silent-stateless state. -7. Every misconfiguration in the providers spec §6 table fails at +8. Every misconfiguration in the providers spec §6 table fails at **startup**. Request-time failure for a configuration error is a defect. -8. Config-store payload validation (`ts config push`) applies the same +9. Config-store payload validation (`ts config push`) applies the same rules — including `[permissions]` policy validation (permission spec §3.3) — so a bad config is rejected at push time, before any instance restarts into it. -## 5. Behavior-preserving migration recipe (operator-facing) +## 5. Minimal-divergence migration recipe (operator-facing) + +"Keep exactly today's behavior" is not fully achievable, and the recipe's +name says so. The unavoidable divergences, enumerated (each also a matrix +row): global opt-out honoring (row 8); refusal blocking new grants +everywhere (row 6); newly enforced GPP sharing/targeted fields, which can +also **grant** P4 where nothing granted before (row 3e); the FR +unresolved-geo fallback, where valid TCF consent can grant while today's +unresolved-geo path always denies (row 5); malformed-present blocking +acquisition (§4.4); proxy-mode opt-out extraction; and the batch-sync +provenance gate (row 11c). Everything else the recipe preserves. The migration guide (a new `docs/guide/` page, linked from the release notes) -gives one copy-pasteable recipe per adapter for "keep exactly today's -behavior": +gives one copy-pasteable recipe per adapter for the minimal-divergence +posture: The recipe is a **complete, valid TOML fixture per adapter, committed to the repository** (e.g. `docs/guide/fixtures/migration-preserving-fastly.toml` and siblings) and included in the guide verbatim — never described as a textual delta against the example file. Per-adapter because a single -fixture cannot be: `[device] provider = "fastly"` and -`[geo] provider = "platform"` are capability-gated selections that the -Axum/Cloudflare/Spin adapters reject at startup (providers spec §6); each +fixture cannot be: `[device] provider = "fastly"` is Fastly-only, and +`[geo] provider = "platform"` varies by host — Cloudflare **does** +support platform geo but resolves **country only, no region** (per the +providers spec adapter matrix), which changes state-level US privacy +outcomes and engages the declared regionless degradation; Axum and Spin +have no platform geo and reject the selection (providers spec §6). Each adapter's fixture carries the selections valid for it, and each is CI-validated against its adapter. (An earlier draft said "copy the example table, then set `[permissions.rules] default`" — but the copied table already @@ -266,16 +296,24 @@ global honoring of opt-out signals is unconditional. 3. Startup logs always print: selected provider per concern, whether geo is live, the effective default baseline, and the count of granted-without- signal permissions. One line, greppable, stable format. -4. **The batch-sync coverage dip is operationalized, not discovered.** +4. **The batch-sync coverage dip is a gated rollout stage, not a + notification.** Provenance-coverage thresholds are normative gate + criteria: the guide defines a target coverage level and evaluation + window; recovery stalling below threshold for the window triggers the + **pause action** — investigate backfill (traffic mix, dormant rows), + never disable the gate; and staging is explicit: provenance + **writing** begins the moment N+2 activates, enforcement is already + in force (there is no fail-open stage), so the only stageable knob is + partner communication and the cleanup cadence for rows that never + recover. Because legacy rows fail closed for batch updates until backfilled (permission spec §7), batch-sync acceptance drops toward zero at cutover and recovers along the live-traffic backfill curve. The **provenance-coverage metric** (share of active rows carrying - provenance) is the tracking signal; the migration guide states the - expected recovery shape, tells operators to notify batch-sync partners - of the transient rejection rate, and defines no fail-open shortcut — - the alternative (grandfathering pre-epic identities past the - permission model) is rejected in the permission spec. + provenance) is the gate signal; operators notify batch-sync partners + of the transient rejection rate. There is no fail-open shortcut — the + alternative (grandfathering pre-epic identities past the permission + model) is rejected in the permission spec. 5. Rollback is config-only where possible: reverting to the previous config version restores the previous behavior on the previous binary. The one irreversible artifact is withdrawal tombstones — which is why the @@ -301,3 +339,32 @@ global honoring of opt-out signals is unconditional. spec verbatim — operator docs and normative spec must not diverge on precedence, and prose like "signals are mapped as a grant or a revoke" without stating which wins is insufficient. + +## 8. Product decisions requiring explicit sign-off + +These are decisions this spec set makes that #838 had not already made (or +made differently). Each must be ratified by maintainers before +implementation — an unratified row reverts to open, not to silently +implemented: + +1. Opt-outs are honored globally and destructive ones irreversibly + withdraw identities outside the jurisdiction defining the signal + (permission spec §4, §4.2). +2. Sale opt-outs (GPP and USP) control both P1 and P4 and destroy the + identity (§4.5). +3. Sharing / targeted-advertising opt-outs remove P4 but intentionally + retain the stored identity (§4.5). +4. US contextual auctions continue during opt-out, with identity removed + (permission spec §7 dispatch matrix). +5. Regionless US traffic is treated as non-regulated unless the operator + chooses country-wide gating (permission spec §3.4). +6. Full consent strings continue downstream, and raw consent snapshots + are retained in graph rows for audit (providers spec §6.3). +7. Legacy batch-sync traffic is rejected until live-browser provenance + backfill occurs (§6.4 of this spec; permission spec §7). +8. Proxy / click / Testlight forwarding becomes newly gated by P1 ∧ P4 + (§2 row 11b). +9. Integration-owned response cookies are inside the permission model: + persistent cookies require `store-on-device` at apply time and a + declared registration; session cookies are the narrow exemption + (response-hook spec §3). From c8b4b849edb110fb94ae6fb433e6038fa35c3bc7 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 31 Jul 2026 19:44:28 -0700 Subject: [PATCH 08/24] Address sixth review: global opt-out aggregation, negative authority, and storage-protocol coherence P1 fixes: - GPP applicability now gates grants only: mapped opt-out fields (either subclass) aggregate globally from any section on any request, resolving the section-4-vs-4.5 contradiction where a French visitor's usnat SaleOptOut was simultaneously mandatory and ignored. - The section map covers everything current code recognizes (7-23, through usmn), not 7-12; Texas section 16 named as what the truncated map would have silently lost; states without a state section (MD, IN, KY, RI) use the national section. - Destructive TCF-refusal withdrawal requires the refusal to be carried by the live request; persisted-KV records participate in acquisition only - closing the path where a years-old stored refusal tombstones on the first signal-less request after a policy tightens to denied, and repairing the mixed-revision safety claim. - Negative authority gets its own record: a permission-exempt, strongly consistent suppression record (sup/) with per-permission entries that every S2S recompute and partner-egress check consults - resolving the circularity where clearing P1 provenance required the P1 the refusal just unset, and the eventual-row edge where a stale replica restored P4 after a targeted-advertising opt-out. - The consistency requirement has one normative home (the providers matrix, strong read-after-write); the permission spec's bounded-lag leftover is gone. - The never-returning-visitor residual is stated as unbounded and becomes sign-off item 11, replacing the false bounded-by-return- latency claim. - Aliases live at the source identity key with a kind discriminator in the value envelope - a separate alias/ address could neither be found by old-cookie lookups nor installed by a single-key CAS. - Identity keys drop the version segment (id//); version lives in the row envelope, killing the read-the-row-to-learn- how-to-read-the-row circularity. All HMAC versions stay on the verbatim key scheme, keeping the 64-hex cluster prefix a literal key prefix for every HMAC row; rotation-induced cluster splits are declared as inherent to rotation. - Rewrite requires the row store itself to provide per-key CAS with read-your-writes (alias installs happen there); adapters with purely eventual row stores cannot host rewrite. Chains use bounded traversal (4 hops, cycle detection, fail closed) with opportunistic path compression instead of an undefined inbound-alias index. - Revocation-wins at the resolve endpoint is an explicit linearization point: family records carry an epoch and the cookie-emitting commit is a CAS conditioned on it - linearizable reads alone lose the race. - resolve_from_client takes a core-built ClientResolveContext (canonical audience, verified session owner, clock, bounded payload) and returns a verified identity with reservation id and expiry. - The mutator snapshot is redacted: all Set-Cookie values and reserved identity/consent/privacy header values withheld, so the hook cannot leak the raw EC around AuthorizedIdentity. - The cache merge is over independent sticky directives per RFC 9111 (no-cache and private are orthogonal; the ordered-lattice version could make a personalized response shared-storable), and the complete origin Vary set is preserved, not only core-required members. - Hook cookie coupling acknowledged: the persistent-cookie gate is a listed enforcement point in the permission spec inventory; cookie operations activate only after the permission model lands; a typed cookie builder enforces declared lifetime/scope/security attributes; deletion cookies work when P1 is denied. - N+1 is a full semantic reader and enforcer for every N+2 record kind (aliases, family revocation, suppression, provenance fail-closed), with rollback tests on N+1 against N+2 data; binaries-first rollback gains its precondition (converge to an N+1-compatible config first after N+2-only adoption, retaining new-provider secrets as legacy readers rather than reverting config). - Adapters without revocation-eligible storage migrate with explicitly stateless fixtures (sign-off item 12) instead of invalid HMAC fixtures. P2: explicit NotApplicable rows (grant-class, preserved) separated from absent; per-permission first-seen digests over only applicable aggregated fields; consent.us_states.privacy_states path corrected; provider-switch rollback keeps the new provider as a legacy reader; rewrite_legacy with a client-resolve writer is a startup error; client parity redefined as identical startup rejection on ungated adapters; the capability matrix distinguishes platform availability from wiring (Spin: available, not wired); row 3e's effects classified in both directions; and the sign-off list is a ratification table (owner/status per row, implementation blocked while any row is open) extended with items 10-14. --- ...26-07-30-client-cycle-ec-resolve-design.md | 32 ++-- ...integration-response-header-hook-design.md | 59 ++++--- .../2026-07-30-permission-model-design.md | 132 ++++++++++----- .../2026-07-30-pluggable-providers-design.md | 126 ++++++++------ ...07-30-provider-migration-rollout-design.md | 157 ++++++++++-------- 5 files changed, 314 insertions(+), 192 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index 1eeb61e14..3cfeb09d4 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -85,10 +85,15 @@ Everything in this spec follows from that. must be parseable, graph-keyed, and tombstonable by the selected provider (providers spec §3). The conformance suite runs against every client-cycle provider. -5. **Exist on every adapter.** Route registration goes through shared route - wiring; the parity suite asserts the endpoint's presence and behavior on - all four adapters. (PR #838 registered it on Fastly only, so the same - config on the Axum dev server proxied the POST to the publisher origin.) +5. **Exist on every adapter — where parity means identical behavior, + including identical refusal.** Route registration goes through shared + route wiring. On adapters whose capability matrix rows are green + (today only the dev adapter has the required CAS class — providers + spec §7), the parity suite asserts identical endpoint behavior; on + adapters without them, parity means **identical startup rejection of + the client-cycle selection** — not a proxied 404 (PR #838's failure + mode: Fastly-only registration let the Axum dev server proxy the POST + to the publisher origin), and not a silently absent route. 6. **Be uncacheable and permission-gated on the provider's full declaration.** `Cache-Control: no-store`; before any cookie is set, the endpoint enforces the selected provider's complete @@ -157,13 +162,18 @@ Everything in this spec follows from that. explicitly-accepted at-most-once posture (no owner hash), a lost response is an **orphan row** handled by the specified cleanup path. The **same-identity no-op of §3.8 first checks the family revocation - record** (permission model spec §4.3), and it does so **through the - linearizable primitive class this feature already requires** for - reservations (providers spec §7 matrix) — which is what makes - "revocation wins" true rather than aspirational: on an eventually - consistent read, a racing create could observe a stale absence and - emit a cookie for a revoked family. A resolve against a revoked family - is rejected, never refreshed, and a create racing a revocation loses. Tests cover crash-between-steps, lease + record**, and "revocation wins" is enforced by an explicit + linearization point, not by read strength alone — a linearizable read + followed by a separate commit still loses the race (read "not + revoked" → withdrawal commits → resolve emits a cookie for a revoked + family). The family record carries an **epoch** (providers spec + §6.3), bumped by every revocation-state change; the resolve reads + epoch _e_ at the start, and its cookie-emitting commit is a **CAS in + the strong class conditioned on the family epoch still being _e_**. + A withdrawal landing between read and commit bumps the epoch, the + commit fails, and the resolve is rejected — the CAS is the + linearization point. A resolve against an already-revoked family is + rejected, never refreshed. Tests cover crash-between-steps, lease takeover with a stale-epoch commit attempt, two concurrent requests with the same payload, a duplicate from a second client receiving no cookie, owner-hash recovery receiving the cookie, and diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 23d1241f2..5d35f23a5 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -61,17 +61,22 @@ mutators to the outbound response for HTML document responses it processed. merely passed through) — and the post-hook response may only be **equal or stronger** on the privacy axis: integrations can tighten caching, never loosen it, regardless of which header they replaced. - "Equal or stronger" is a defined merge, not a vibe: restriction - strength is ordered `no-store` > `no-cache` > `private` > `public`, - and the final value per axis is the **stronger of snapshot and - mutation**; `max-age`/`s-maxage` may only shrink relative to the - snapshot; `stale-while-revalidate`/`stale-if-error` may appear only if - the snapshot had them; every CDN/surrogate directive - (`Surrogate-Control`, `CDN-Cache-Control`, host-specific equivalents) - is stripped from any restricted response; and **core-required `Vary` - members are protected in the contract, not just the tests** — the - final `Vary` is the union of the snapshot's required members and the - mutation. Middle-stage placement also keeps + "Equal or stronger" is a defined merge over **independent sticky + directives, not a totally ordered lattice** — `no-cache` and `private` + are orthogonal constraints (RFC 9111: `no-cache` permits shared + storage subject to revalidation; `private` forbids shared storage), so + "replace `private` with the stronger `no-cache`" would make a + personalized response shared-storable. The merge: each of `no-store`, + `no-cache`, `private` is **sticky** — present in the snapshot or the + mutation ⇒ present in the final response, independently; `public` is + dropped whenever any restriction is present; `max-age`/`s-maxage` may + only shrink relative to the snapshot; `stale-while-revalidate`/ + `stale-if-error` may appear only if the snapshot had them; every + CDN/surrogate directive (`Surrogate-Control`, `CDN-Cache-Control`, + host equivalents) is stripped from any restricted response; and the + final `Vary` is the **union of the complete snapshot `Vary` set** — + origin-supplied members included, not only core-required ones — and + the mutation. Middle-stage placement also keeps the earlier property: an integration mutation is not silently stripped by ordinary core handling — only by the invariant pass, which logs the downgrade it applies. @@ -98,8 +103,14 @@ mutators to the outbound response for HTML document responses it processed. (any `Max-Age`/`Expires`) is applied only when the request's resolved permissions include `store-on-device`, while **session cookies** (no persistence attributes) are the narrow, documented exemption. - Undeclared cookie names are rejected like reserved ones. An integration - may never set or expire a reserved cookie name. Violations are rejected at the operation layer (§2) and + Cookie operations go through a **typed cookie builder** that enforces + the declared lifetime ceiling, domain/path scope, and security + attributes (`Secure`, `SameSite`) — not a free-form string; **deletion + cookies (expiry of the integration's own declared names) remain + possible when `store-on-device` is denied**, since removing state must + never require the permission to keep it. Undeclared cookie names are + rejected like reserved ones. An integration may never set or expire a + reserved cookie name. Violations are rejected at the operation layer (§2) and logged at `warn` with the integration id. The reserved lists are single constants next to the definitions they protect, not duplicated in the hook. @@ -121,10 +132,17 @@ mutators to the outbound response for HTML document responses it processed. counting `name: value` plus separators, within any lower adapter ceiling) bounds the sum across integrations — enforced in registration order, so which operations are rejected when a budget trips is - deterministic. Each mutator receives an **immutable snapshot of the + deterministic. Each mutator receives an **immutable, redacted snapshot of the response head** (status and headers as of its turn, prior integrations' accepted operations applied) as its read context; it never holds a - mutable reference (§2). + mutable reference (§2). Redaction is a security boundary, not + tidiness: the hook runs after core queues the EC `Set-Cookie`, so an + unredacted view would hand a mutator the raw EC to copy into + `X-Vendor-Identity` or its own cookie — walking around + `AuthorizedIdentity` entirely. The snapshot therefore + **excludes every `Set-Cookie` value and every reserved identity, + consent, and privacy header value** (names may be listed as present; + values are withheld). Exceeding a limit rejects the excess operations (logged, attributed), never the response. A mutator that returns an error is skipped in full — its operations are all-or-nothing — and the response proceeds without it. @@ -181,10 +199,13 @@ processed documents (§6). ## 5. Size and sequencing -This is a ~150-line feature plus tests, with zero coupling to the provider -architecture or the permission model. It lands as its own small PR **when -its first real consumer is identified** (§4.2) — at any point in the epic's -sequence, blocking nothing and blocked by nothing. If no consumer +This is a modest feature plus tests, with zero coupling to the provider +architecture — but its **cookie operations are coupled to the permission +model** (§3; the gate is a listed enforcement point in the permission +spec §7 inventory), so the claim of total independence is retired: the +header-mutation portion may land whenever its first real consumer is +identified (§4.2), while `append_set_cookie` activates only **after** the +permission model PR, and registers as unavailable before it. If no consumer materializes, it does not land; being unblocked is not a reason to ship scaffolding. diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 6154de7c7..8a0818166 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -210,7 +210,7 @@ Validation rejects: ### 3.4 One source of jurisdiction truth Today, `detect_jurisdiction` — driven by the runtime lists -`consent.gdpr.applies_in` and `consent.us_privacy.states` — is the sole +`consent.gdpr.applies_in` and `consent.us_states.privacy_states` — is the sole jurisdiction source for **both** the auction consent gate and the EC gate. The permission model replaces the EC side; if the auction gate keeps reading the old lists while EC reads policy rules, the two will drift (adding a @@ -225,7 +225,7 @@ survive an interim period, a CI test asserts consistency between each list and the policy's regime classes, with deliberate divergences recorded as explicit, commented exceptions in the test — never silent. Both legacy lists are in scope, not only the GDPR one — and the US check is -region-shaped: **every configured `consent.us_privacy.states` entry must +region-shaped: **every configured `consent.us_states.privacy_states` entry must have a matching `US/` rule**, and the country-level `US` rule must resolve non-regulated (today applies privacy gating only to the configured states), or the divergence is an explicit commented exception. An adapter @@ -356,8 +356,15 @@ The triggers, exhaustively — nothing else withdraws: targeted-advertising) never trigger this — they revoke acquisition only. (For US states this preserves today's behavior; elsewhere it is the declared change of §4's global-opt-out rule.) -2. **A TCF record refusing `store-on-device` withdraws iff the baseline is - `requires_signal` or `denied`.** Where the baseline is `granted`, +2. **A TCF record refusing `store-on-device` withdraws iff the baseline + is `requires_signal` or `denied` — and only when the refusal is + carried by the live request.** A persisted-KV consent record + participates in acquisition only and **never triggers withdrawal**: + without this, a refusal stored years ago under a `granted` policy + would destructively fire on the first signal-less request after the + policy tightens to `denied` — a policy edit tombstoning by proxy, + which trigger 3 forbids, and the counterexample to §5.5's + mixed-revision safety claim. Where the baseline is `granted`, refusal blocks _new_ grants but never tombstones: tombstones are irreversible, and PR #838 wrote them for visitors in unregulated jurisdictions whose global CMP emitted a purpose-refusing string — @@ -410,25 +417,43 @@ and the fail-closed marker: discoverable from every member, and the record survives member-tombstone replacement (which today discards the original row's identity and metadata, making sibling discovery impossible). +- **Negative authority has its own permission-exempt record.** A live + refusal or non-destructive opt-out must clear prior positive + provenance — but the row write that would do it requires `store-on-device`, + which the refusal just unset, and identity rows may be eventually + consistent, so a stale replica could resurrect a P4 grant after a + targeted-advertising opt-out. The fix is a **suppression record** in + the strongly consistent class (providers spec §6.3: `sup/`), + carrying per-permission suppression entries with timestamps. Writing it + is **permission-exempt** (clearing authority is protective, like + revocation), and **every S2S recompute and partner-egress check + consults it**: a suppressed permission is unset whatever the row's + provenance says, so no eventual-consistency edge can restore it. - **The cookie expires only after the family record commits.** - **If the family-record write itself fails, nothing durable exists** — - the cookie stays and the durable client-side signal (GPC, CMP-stored TCF) - retries the whole withdrawal on the next request. Two mitigations bound - the S2S residual in the meantime: while graph **writes are degraded** - (health signal), S2S partner egress and sync updates fail closed - (providers spec §6.2); and the failure is logged at `error` with a - metric feeding the operational repair path. The residual that remains — - a single failed write on an otherwise healthy graph, for a user who - never returns — is declared here, not hidden. -- **Consistency and retention are backend contracts**, defined in the - providers spec consistency matrix (§7): revocation-record reads use the - strongest read the backend offers, adapters declare a bounded - revocation-visibility lag (an eventually-consistent store that cannot - bound it fails startup for identity features), a **failed family-record + the cookie stays and the durable client-side signal (GPC, CMP-stored + TCF) retries the whole withdrawal on the next request. Mitigations: + while graph **writes are degraded** (health signal), S2S partner egress + and sync updates fail closed on that instance (providers spec §6.2); + the failure is logged at `error` with a metric feeding the operational + repair path. The residual that remains — a single failed write on an + otherwise healthy graph, for a visitor who **never returns** — is + **unbounded**, not "bounded by return latency": return latency has no + bound for a non-returning visitor, and the per-instance breaker does + not reach other instances. Accepting this residual instead of building + a durable external retry queue is **product sign-off item 11** + (migration spec §8), not a footnote. +- **Consistency and retention are backend contracts with a single + normative home**: the providers spec consistency matrix (§7). It — not + this spec — states the requirement, and it requires a **strongly + consistent (read-after-write) primitive** for revocation records; no + bounded-lag alternative exists (an earlier draft here permitted one, + which contradicted the matrix — an adapter with a two-second lag would + have passed one spec and failed the other). A **failed family-record read fails closed** for egress (revoked-unknown ≠ live), and revocation records are retained beyond the maximum of cookie lifetime, row TTL, - rewrite grace, and downstream retry horizon — note today's 24-hour - tombstone TTL is far below this bar and does not carry over. + rewrite grace, and downstream retry horizon — today's 24-hour tombstone + TTL is far below this bar and does not carry over. - Fault-injection tests cover: family-record write fails → cookie untouched, S2S behavior per degraded mode, retry completes; member tombstone N fails after the family record → identity already revoked for @@ -474,17 +499,18 @@ withdrawal. Section IDs and versions are those of the IAB GPP specification current at implementation time; adding a section or field is a change to this table. -| Source · field | Value | `store-on-device` (P1) | `select-personalised-ads` (P4) | Destructive withdrawal? | -| -------------------------------------------- | ------------- | ---------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------ | -| GPP US section · `SaleOptOut` | opted out | opt-out | opt-out | **Yes** (preserves today) | -| GPP US section · `SaleOptOut` | not opted out | grant | grant | — | -| GPP US section · `SharingOptOut` | opted out | — | opt-out | No | -| GPP US section · `SharingOptOut` | not opted out | — | grant | — | -| GPP US section · `TargetedAdvertisingOptOut` | opted out | — | opt-out | **No** — a targeted-advertising choice must never destroy the stored identity | -| GPP US section · `TargetedAdvertisingOptOut` | not opted out | — | grant | — | -| US Privacy · `opt_out_sale` | `Y` | opt-out | opt-out | **Yes** (preserves today) | -| US Privacy · present, `N` or N/A | — | grant | grant | — (today's tests pin N/A as allowing; USP carries no distinct targeted-advertising field, so it never maps to one) | -| Any field | absent / N-A | — | — | — | +| Source · field | Value | `store-on-device` (P1) | `select-personalised-ads` (P4) | Destructive withdrawal? | +| -------------------------------------------- | --------------------------- | -------------------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | +| GPP US section · `SaleOptOut` | opted out | opt-out | opt-out | **Yes** (preserves today) | +| GPP US section · `SaleOptOut` | not opted out | grant | grant | — | +| GPP US section · `SharingOptOut` | opted out | — | opt-out | No | +| GPP US section · `SharingOptOut` | not opted out | — | grant | — | +| GPP US section · `TargetedAdvertisingOptOut` | opted out | — | opt-out | **No** — a targeted-advertising choice must never destroy the stored identity | +| GPP US section · `TargetedAdvertisingOptOut` | not opted out | — | grant | — | +| US Privacy · `opt_out_sale` | `Y` | opt-out | opt-out | **Yes** (preserves today) | +| US Privacy · present, `N` or N/A | — | grant | grant | — (today's tests pin N/A as allowing; USP carries no distinct targeted-advertising field, so it never maps to one) | +| Any field | explicitly _Not Applicable_ | as the field's not-opted-out row above | as the field's not-opted-out row above | — | +| Any field | absent | — | — | — | **N/A vs absent:** a field explicitly set to _Not Applicable_ is treated as not-opted-out (grant-class) — pinned by today's USP tests and matching @@ -495,17 +521,29 @@ nothing. **Applicability and aggregation — ordered algorithm:** -1. **Section map (normative, pinned here — not "whatever GPP is current"):** - `US` national ↔ GPP section 7 (usnat); `US/CA` ↔ 8 (usca); `US/VA` ↔ 9 - (usva); `US/CO` ↔ 10 (usco); `US/UT` ↔ 11 (usut); `US/CT` ↔ 12 (usct). - Section versions are those published at this spec's date; adding a - section or version is a change to this map. -2. **Determine applicability from the resolved jurisdiction:** the - national section is applicable to any `us-privacy`-regime request; a - state section is applicable iff it maps to the resolved `US/`. - Foreign-state sections (a `usca` string on a `US/CO` request) and all - sections on non-`us-privacy` requests are **not applicable** and - contribute nothing. Regionless US traffic: national section only. +1. **Section map (normative, pinned here — not "whatever GPP is + current"), covering every section current code recognizes (7–23), not + a subset:** `US` national ↔ 7 (usnat); then the state sections — + `US/CA` ↔ 8, `US/VA` ↔ 9, `US/CO` ↔ 10, `US/UT` ↔ 11, `US/CT` ↔ 12, + `US/FL` ↔ 13, `US/MT` ↔ 14, `US/OR` ↔ 15, `US/TX` ↔ 16, `US/DE` ↔ 17, + `US/IA` ↔ 18, `US/NE` ↔ 19, `US/NH` ↔ 20, `US/NJ` ↔ 21, `US/TN` ↔ 22, + `US/MN` ↔ 23. Dropping to 7–12 would silently lose, e.g., a Texas + (section 16) sale opt-out. The implementation PR cross-checks this + list against the current decoder's section set; versions are those + published at this spec's date; adding a section or version is a change + to this map. +2. **Applicability gates grants only — never opt-outs.** A mapped + **opt-out** field (either subclass) is honored from **any** section on + **any** request, whatever the regime — this is §4's global-opt-out + rule, and filtering it by jurisdiction would make a French visitor's + `usnat SaleOptOut` simultaneously mandatory (§4) and ignored (here). + For **grants**: the national section is applicable to any + `us-privacy`-regime request; a state section is applicable iff it maps + to the resolved `US/`; foreign-state sections and all sections + on non-`us-privacy` requests grant nothing. Regionless US traffic: + national section only. A configured privacy state with no + state-specific section (e.g. MD, IN, KY, RI today) uses the national + section alone. 3. **State-over-national, per field:** where an applicable state section carries a field, it governs that field; the national section fills only fields the state section lacks. @@ -652,6 +690,8 @@ Consumers of the resolved set in this epic: | Request-scoped graph reads/writes (non-revocation) | `store-on-device` | | | Revocation paths (tombstones, withdrawal reads) | **exempt** | Must work when permissions are unset | | Stored consent-state lookup (§4.4) | **exempt**, narrowly scoped | Determining `store-on-device` cannot require `store-on-device` | + | Integration persistent response cookies (hook spec §3) | `store-on-device` | Applied at mutation time from the request's resolved permissions; session cookies are the declared exemption (sign-off items 9–10) | + | Suppression-record writes (§4.3) | **exempt** | Clearing authority is protective, like revocation | With **no EC provider configured**, identity use fails closed: a cookie value present on the request never egresses anywhere — never vacuously @@ -668,11 +708,11 @@ Consumers of the resolved set in this epic: spec §6.1). Freshness is a **per-evidence-class contract**, because not every source carries a timestamp: - | Evidence class | Authoritative timestamp | Age reset | Max age | - | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------- | - | TCF consent | The record's `LastUpdated` | Only a record with a **newer** `LastUpdated` | Existing TCF expiry TTL | - | GPP / USP values (no intrinsic timestamp) | **First-seen**: when TS first observed this exact normalized value (equality digest stored in provenance) | Re-presenting an identical digest **keeps the original first-seen**; a different value is new evidence with a new first-seen | Consent TTL (same as TCF) | - | Policy-baseline grant (`granted` rule, no signal) | The policy revision that granted | Re-derived on every recompute against the current revision — policy is not user evidence and does not age; it changes | n/a | + | Evidence class | Authoritative timestamp | Age reset | Max age | + | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------- | + | TCF consent | The record's `LastUpdated` | Only a record with a **newer** `LastUpdated` | Existing TCF expiry TTL | + | GPP / USP values (no intrinsic timestamp) | **First-seen**: when TS first observed this exact normalized value (a **per-permission equality digest computed over only the applicable, aggregated §4.5 fields for that permission** — never the whole GPP record, or a CMP touching an unrelated notice field would mint a new digest and reset first-seen forever) | Re-presenting an identical digest **keeps the original first-seen**; a different value is new evidence with a new first-seen | Consent TTL (same as TCF) | + | Policy-baseline grant (`granted` rule, no signal) | The policy revision that granted | Re-derived on every recompute against the current revision — policy is not user evidence and does not age; it changes | n/a | Timestamps are compared with bounded clock-skew tolerance and future-dated values are clamped to receipt time. And every live diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index 2d3d58e58..d34382907 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -201,8 +201,14 @@ pub trait EdgeCookieProvider { /// their page leg needs. One provider implements exactly one mode. pub enum Acquisition<'a> { ServerMint(&'a dyn ServerMint), // fn generate(&IdentityInput) -> EcId - ClientResolve(&'a dyn ClientResolve),// fn resolve_from_client(&Payload) -> EcId -} // + fn js_module_id() -> &str + ClientResolve(&'a dyn ClientResolve), +} +// ClientResolve::resolve_from_client(&ClientResolveContext) -> Result +// ctx is core-built: canonical publisher audience, verified session +// owner hash, clock, and the bounded payload — a bare payload could +// not verify audience binding, session binding, or expiry. +// VerifiedIdentity carries the identifier, reservation id, and expiry. +// ClientResolve::js_module_id() -> &str ``` (Names indicative; the shape is normative. `required_permissions` joins the @@ -294,15 +300,16 @@ one. This spec resolves that by **not having** the method on those traits All validation happens at **settings construction** — a misconfiguration is a startup error, never a request-time error and never a silent behavior change. -| Configuration state | Behavior | -| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `provider` names an unknown key | Startup error listing valid keys. | -| `provider` set, its `[ec.providers.]` block missing | Startup error. | -| `[ec.providers.]` block present, `provider` unset | **Startup error.** (In PR #838 this silently ran stateless — the half-migrated config becomes a production identity outage detected by revenue drop. Rejecting it is the fix.) An operator who genuinely wants stateless deletes the block. | -| `provider` set to an implementation the running adapter cannot satisfy (e.g. a provider requiring host TLS fingerprints on an adapter that has none) | Startup error at adapter wiring time. Adapters declare their host capabilities to the composition root; the root checks the selected provider's needs against them **once**, at startup — not per request. | -| No `provider`, no providers block | Valid: the neutral default for that concern. | -| `provider = "none"` (explicit stateless) | Valid, and the only way to combine statelessness with `legacy_providers`: minting stops, legacy readers keep existing identities resolvable and **withdrawable** (§6.1). Without this state, `hmac` → stateless would strand every live row in revoke-proof limbo. | -| A minting provider (or any `legacy_providers`) configured, but no identity-graph store configured or openable | **Startup error.** The lifecycle contract assumes graph persistence (§5); discovering its absence at first mint would be a request-time config failure, which this table exists to forbid. | +| Configuration state | Behavior | +| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `provider` names an unknown key | Startup error listing valid keys. | +| `provider` set, its `[ec.providers.]` block missing | Startup error. | +| `[ec.providers.]` block present, `provider` unset | **Startup error.** (In PR #838 this silently ran stateless — the half-migrated config becomes a production identity outage detected by revenue drop. Rejecting it is the fix.) An operator who genuinely wants stateless deletes the block. | +| `provider` set to an implementation the running adapter cannot satisfy (e.g. a provider requiring host TLS fingerprints on an adapter that has none) | Startup error at adapter wiring time. Adapters declare their host capabilities to the composition root; the root checks the selected provider's needs against them **once**, at startup — not per request. | +| No `provider`, no providers block | Valid: the neutral default for that concern. | +| `rewrite_legacy = true` with a **client-resolve** active writer | **Startup error.** An organic request carries no signed client payload to mint from, and a later resolve POST meeting a different existing identity is a `409` by the client-cycle spec — the combination is incoherent until an authenticated linking/migration flow is specified. | +| `provider = "none"` (explicit stateless) | Valid, and the only way to combine statelessness with `legacy_providers`: minting stops, legacy readers keep existing identities resolvable and **withdrawable** (§6.1). Without this state, `hmac` → stateless would strand every live row in revoke-proof limbo. | +| A minting provider (or any `legacy_providers`) configured, but no identity-graph store configured or openable | **Startup error.** The lifecycle contract assumes graph persistence (§5); discovering its absence at first mint would be a request-time config failure, which this table exists to forbid. | Unknown fields inside every provider config block are rejected (`deny_unknown_fields` on all new settings structs — the pre-existing `Ec` @@ -371,19 +378,28 @@ The contract: evidence would rejuvenate stale authority); partner mappings copy with their **original per-field timestamps and expiry**, and the copy point is recorded in the transaction. - 3. **Fenced CAS replaces the old row with an alias record** targeting - the canonical. If the CAS loses to a concurrent pull/batch/identify - update, the rewrite **re-runs a reconciliation pass** under its - epoch — merging updates newer than the recorded copy point into the - canonical — and retries the CAS; an update that won the old row is - therefore never lost. + 3. **A per-key CAS on the source identity key replaces the row value + with the alias** (same address, `kind` discriminator — §6.3). This + is why rewrite requires the row store itself to offer CAS with + read-your-writes (§7 matrix): the participants must share + transactional primitives, or a source update landing through an + eventual replica after the copy could be silently dropped. If the + CAS loses to a concurrent pull/batch/identify update, the rewrite + **re-runs a reconciliation pass** under its epoch — re-reading the + source with the store's strongest read, merging updates newer than + the recorded copy point into the canonical — and retries; an update + that won the old row is therefore never lost. 4. The new cookie is emitted; the transaction marks complete. From step 3 on, every read or update through either cookie chases the - alias (one hop) to the single canonical row. **Chains stay single-hop**: - a later rewrite B→C retargets every alias pointing at B (the canonical - row records its inbound aliases; alias records are in the linearizable - class, so retargeting is fenced) so A points directly at C. The server + alias to the single canonical row. Chains are handled by **bounded + traversal with path compression**, not an inbound-alias index (an + index would need its own fenced schema, bounds, and concurrency rules + that nothing defined): traversal follows at most **4** hops with + visited-set cycle detection (deeper or cyclic → treated as row-read + failure, fail closed per §6.2); whenever a traversal crosses more than + one hop, it opportunistically CASes the first alias to point at the + final canonical, so chains converge to one hop without coordination. The server cannot observe `Set-Cookie` acceptance, so the alias stays live until a later request **presents the new cookie**, and in any case until a **finite retirement deadline** no shorter than the old cookie's maximum @@ -448,24 +464,31 @@ solves. **Physical key grammar.** Core constructs every key; providers supply only the bounded suffix: -| Record class | Key | Notes | -| ----------------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------- | -| Identity row (v2+) | `id///` | Suffix from `graph_key_suffix`, ≤ 128 bytes, KV-safe alphabet | -| Identity row (legacy hmac-v0) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar; no other class or provider may produce a matching key | -| Alias | `alias///` | Same suffix as the row it replaced | -| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | -| Rewrite transaction | `rwx/` | One in-flight rewrite per family | -| Replay reservation | `resv////` | Client-cycle spec; payload id ≤ 128 bytes | - -Grammars are pairwise non-intersecting by their literal prefixes (plus the -reserved legacy grammar), which is what makes cross-class collision -impossible rather than unlikely. +| Record class | Key | Notes | +| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ 128 bytes, KV-safe alphabet. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | +| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | +| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | +| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | +| Suppression (negative authority) | `sup/` | Per-permission suppression entries + timestamps; permission-exempt writes; consulted by every S2S recompute and partner-egress check (permission spec §4.3) | +| Rewrite transaction | `rwx/` | One in-flight rewrite per family | +| Replay reservation | `resv////` | Client-cycle spec; payload id ≤ 128 bytes | + +Grammars are pairwise non-intersecting by their literal prefixes (plus +the reserved hmac grammar), and every record value carries a `kind` +discriminator alongside its schema version — so a reader always knows +what it fetched, including where two classes deliberately share an +address (row vs. alias). **Wire schemas** (JSON, like identity rows; every class carries a schema version): the **alias record** holds target key, created-at, retirement deadline, and fencing epoch; the **family revocation record** holds the family ID, revoked-at, triggering signal class (§4.5 destructive column), -and epoch — deliberately no identity data, so it can outlive its members; +and a **family epoch** bumped on every revocation-state change (the +client-cycle commit CAS is conditioned on it) — deliberately no identity +data, so it can outlive its members; the **suppression record** holds +per-permission suppression entries with timestamps (strong class, +permission-exempt writes, permission model spec §4.3); the **rewrite transaction** holds source key, target key, copy point, state, and epoch; the **reservation** holds state, owner hash, lease epoch, outcome, and created-at (client-cycle spec). Field validation and @@ -521,12 +544,14 @@ Requirements: **per-record-class consistency requirements**, because "has KV" says nothing about whether revocation is observable: - | Record class | Required semantics | - | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | - | Family revocation records | **Strongly consistent (read-after-write) primitive required.** Cloudflare Workers KV is **not eligible** — its documentation says propagation may take "60 seconds or more", an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | - | Alias / rewrite-transaction records | **Linearizable fenced CAS required** (same primitive class as reservations). `rewrite_legacy = true` is rejected at startup on adapters lacking it (§6) | - | Identity rows | Eventual consistency acceptable — rows are accretive, and the family-record check (strong, per above) governs use | + | Record class | Required semantics | + | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | + | Family revocation records | **Strongly consistent (read-after-write) primitive required.** Cloudflare Workers KV is **not eligible** — its documentation says propagation may take "60 seconds or more", an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | + | Family suppression records | Same strong class as family revocation — negative authority must not lose races to stale replicas | + | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | + | Alias installs | Row-store **per-key CAS with read-your-writes** — the alias lives at the source identity key (§6.3), so the _row store itself_ must supply the CAS; a purely eventual row store cannot host rewrite. `rewrite_legacy = true` is rejected at startup unless both this and the transaction class are available (§6) | + | Identity rows | Eventual consistency acceptable — rows are accretive, and the family-record check (strong, per above) governs use | Each adapter's declaration is part of its wiring, drives the §6 capability-mismatch startup error, and every §6.2 runtime-failure row @@ -536,14 +561,19 @@ Requirements: cell marked _verify_ must be established before the depending feature is selectable on that adapter, and the filled matrix is normative: - | Capability | Fastly | Axum (dev) | Cloudflare | Spin | - | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | - | Graph persistence (eventual OK) | KV Store: yes | Local store: yes (dev-grade) | Workers KV: yes (eventually consistent) | Key-value: yes | - | Prefix listing (cluster) | Yes (used today) | Yes | Yes (eventual) | _verify_ | - | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | Yes (in-process) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | - | Linearizable fenced CAS (reservations, alias/rewrite) | **Not currently available** — client-cycle and `rewrite_legacy` unselectable until a primitive exists | Yes (in-process) | Durable Objects: possible, not wired | **No** | - | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | - | Device host evidence (JA4/H2) | Yes | No | No | No | + Cells distinguish **platform availability** (the host offers a + primitive) from **wired** (Trusted Server integrates it) — conflating + them is how a "yes" cell hides an unusable feature. Feature eligibility + requires wired, not merely available: + + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | + | Graph persistence (eventual OK) | KV Store: available + wired | Local store: available + wired (dev-grade) | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | + | Prefix listing (cluster) | Yes (used today) | Yes | Yes (eventual) | _verify_ | + | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | Yes (in-process) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | + | Linearizable fenced CAS (reservations, alias/rewrite) | **Not currently available** — client-cycle and `rewrite_legacy` unselectable until a primitive exists | Yes (in-process) | Durable Objects: possible, not wired | **No** | + | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | + | Device host evidence (JA4/H2) | Yes | No | No | No | - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 54f7fe4d4..ccd48ac0b 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -32,29 +32,29 @@ and whether that is a preservation or a declared change. **Silent changes are defects.** PR #838 changed six of these without declaring any; each was discoverable only because a deleted test had pinned the old behavior. -| # | Decision (today) | After epic | Status | -| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | -| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | -| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | -| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | -| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | -| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | -| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | -| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | -| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — strictly more protective | -| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | -| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | -| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | -| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | -| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | -| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | -| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | -| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | -| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | -| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | -| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | -| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | -| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | +| # | Decision (today) | After epic | Status | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | +| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | +| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | +| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | +| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | +| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — opt-out effects are more protective; the same fields' not-opted-out values can also **newly grant P4**, which is not (both effects classified in permission spec §4.5) | +| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | +| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | +| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | +| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | +| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | +| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | +| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | +| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | +| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | +| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | +| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -119,21 +119,47 @@ Requirements: after **fleet convergence on N+1 is confirmed** — binaries first, convergence gate, then `ts config push`. A config mixing old and new fields (`[ec] passphrase` alongside `[ec] provider`) is **rejected** - by N+1, not reconciled. **Rollback is binaries-first too, in the - other direction**: N+2 → N+1 binaries roll back **keeping the new - config** (N+1 reads it fully — reverting config first would hand the - old shape to N+2 binaries that reject it). N+1 additionally - **rejects provider or version selections whose provenance it cannot - yet encode** — new-provider adoption waits for N+2, so no row is - minted that N+2 would misclassify. Every new config section - introduced by the epic follows this same compatibility rule, not - only `[ec]`. + by N+1, not reconciled. **N+1 is a full semantic reader and + enforcer for every N+2 record kind — not a field preserver.** + Preserving unknown JSON does not chase aliases, consult family + revocations, honor suppression records, or fail closed on + provenance; an N+1 that merely preserved would, after rollback, + treat aliased rows as missing and revoked identities as live. + Rollback tests therefore run the alias, family-revocation, + suppression, and provenance paths **on N+1** against N+2-written + data. **Rollback is binaries-first too, in the other direction** — + N+2 → N+1 binaries roll back keeping the new config (N+1 reads it + fully; reverting config first would hand the old shape to N+2 + binaries that reject it) — **with one precondition**: if an + N+2-only provider or version has been adopted, the fleet must first + converge on an N+1-compatible new-shape config (deselecting what + N+1 rejects, retaining the new provider's secrets as a **legacy + reader** so its minted identities keep resolving and stay + withdrawable until they expire — never "revert to the previous + config", which would strand them); only then do binaries roll back. + N+1 additionally **rejects provider or version selections whose + provenance it cannot yet encode** — new-provider adoption waits for + N+2, so no row is minted that N+2 would misclassify. Every new + config section introduced by the epic follows this same + compatibility rule, not only `[ec]`. - **Release N+2:** rejects `[ec] passphrase` at startup with a message naming the new location — not a generic unknown-field error (implementation note: producing the actionable message means keeping a deprecated `passphrase` field whose presence triggers the custom error). -2. **Graph-store readiness precedes everything.** Today the graph store +2. **Revocation-eligible storage is a per-adapter gate, and ungated + adapters migrate stateless.** Identity features require the adapter's + strong-consistency rows in the capability matrix (providers spec §7) + to be green: today that means Fastly must _verify_ its KV read + semantics, Cloudflare must wire a Durable-Object-class primitive, and + Spin must wire storage at all. Until an adapter passes the gate, its + migration fixture is **explicitly stateless** (`provider = "none"`, + no `[permissions]`-gated identity features) — calling an HMAC fixture + "valid" on an adapter that must reject identity features at startup + would make the required fixtures self-contradictory. Whether ungated + adapters go stateless or block the release is product sign-off + item 12. +3. **Graph-store readiness precedes everything.** Today the graph store is optional and EC generation succeeds without one; the epic's no-active-until-commit invariant (providers spec §5) makes it mandatory wherever a minting provider is configured — so a currently @@ -144,7 +170,7 @@ Requirements: row supports the features in use, providers spec §7) _before_ rolling N+1. This is a **declared breaking change** for graphless deployments (matrix row 12), not a side effect discovered at boot. -3. **The graph schema change is expand-contract, in lockstep with the +4. **The graph schema change is expand-contract, in lockstep with the binary sequence.** New rows carry fields v1 rows never had — provider/ version, per-permission grant evidence, policy revision, family ID, rewrite links — and two failure modes must be engineered away: a naive @@ -170,35 +196,35 @@ Requirements: new fields untouched if read-only, preserved semantically if read-modify-write on N+1, **test-proven lost on pre-N+1** (documenting why the floor is a floor); N+2-reader/N+1-written-row → full function. -4. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no +5. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no `provider = "hmac"` selector is a startup error (providers spec §6). In PR #838 this configuration — the exact state an operator following the docs reaches if they miss one line — validated green and silently minted zero ECs. -5. **PR #838-era keys fail loud.** `provider = "host-signals"` (shipped by +6. **PR #838-era keys fail loud.** `provider = "host-signals"` (shipped by PR #838, deliberately not carried into this epic — providers spec §2) and `provider = "client-fixed"` are unknown keys and rejected like any other, so a config written against the PR #838 example cannot silently select a provider that no longer exists. -6. **Provider switches go through legacy readers.** Changing +7. **Provider switches go through legacy readers.** Changing `[ec] provider` on a deployment with live identities requires listing the outgoing provider in `[ec] legacy_providers` (providers spec §6.1) so existing cookies keep resolving and stay withdrawable; the guide documents the switch sequence and the retirement/cleanup step that ends it. -7. **The example config ships the migrated happy path**, uncommented: +8. **The example config ships the migrated happy path**, uncommented: `provider = "hmac"` with its block, `[geo] default_country`, and (for Fastly) the behavior-preserving `[device] provider = "fastly"` and `[geo] provider = "platform"` lines present with a comment stating what removing them changes. PR #838's example shipped the passphrase block uncommented with the selector commented out — steering operators directly into the silent-stateless state. -8. Every misconfiguration in the providers spec §6 table fails at +9. Every misconfiguration in the providers spec §6 table fails at **startup**. Request-time failure for a configuration error is a defect. -9. Config-store payload validation (`ts config push`) applies the same - rules — including `[permissions]` policy validation (permission spec - §3.3) — so a bad config is rejected at push time, before any instance - restarts into it. +10. Config-store payload validation (`ts config push`) applies the same + rules — including `[permissions]` policy validation (permission spec + §3.3) — so a bad config is rejected at push time, before any instance + restarts into it. ## 5. Minimal-divergence migration recipe (operator-facing) @@ -343,28 +369,23 @@ global honoring of opt-out signals is unconditional. ## 8. Product decisions requiring explicit sign-off These are decisions this spec set makes that #838 had not already made (or -made differently). Each must be ratified by maintainers before -implementation — an unratified row reverts to open, not to silently -implemented: - -1. Opt-outs are honored globally and destructive ones irreversibly - withdraw identities outside the jurisdiction defining the signal - (permission spec §4, §4.2). -2. Sale opt-outs (GPP and USP) control both P1 and P4 and destroy the - identity (§4.5). -3. Sharing / targeted-advertising opt-outs remove P4 but intentionally - retain the stored identity (§4.5). -4. US contextual auctions continue during opt-out, with identity removed - (permission spec §7 dispatch matrix). -5. Regionless US traffic is treated as non-regulated unless the operator - chooses country-wide gating (permission spec §3.4). -6. Full consent strings continue downstream, and raw consent snapshots - are retained in graph rows for audit (providers spec §6.3). -7. Legacy batch-sync traffic is rejected until live-browser provenance - backfill occurs (§6.4 of this spec; permission spec §7). -8. Proxy / click / Testlight forwarding becomes newly gated by P1 ∧ P4 - (§2 row 11b). -9. Integration-owned response cookies are inside the permission model: - persistent cookies require `store-on-device` at apply time and a - declared registration; session cookies are the narrow exemption - (response-hook spec §3). +made differently). **Implementation is blocked while any row is `open`**; +each row needs an owner, a status, and a link to its decision record — +an unratified row reverts to open, not to silently implemented. + +| # | Decision | Where | Owner | Status | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | --------------------- | ------ | +| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | +| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | +| 3 | Sharing / targeted-advertising opt-outs remove P4 but retain the stored identity | permission §4.5 | maintainers + legal | open | +| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | +| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | +| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | +| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | +| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | +| 9 | Integration persistent cookies inside the permission model (P1-gated, declared registration) | hook §3; permission §7 | maintainers | open | +| 10 | Session cookies exempt from `store-on-device` even when they carry a stable identifier | hook §3 | maintainers + legal | open | +| 11 | A single failed destructive withdrawal may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | +| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | +| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | From 09e54e96a97562bc9f5af6b72b67c1b5081cc1af Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 31 Jul 2026 22:28:47 -0700 Subject: [PATCH 09/24] Address seventh review: descope the epic periphery, close core findings, add a review ledger The recurring theme - browser-side acquisition, integration-owned identifiers, rewrite, and pre-existing state generating blockers while the core holds - is answered structurally this round: Descope (sign-off item 15, ratify or veto): - The client-cycle spec is demoted to a deferred informative draft: no production adapter has its CAS-class primitive, it has no consumer, and its findings no longer block core ratification. Within it: the ownerless first-presenter mode is removed outright (risk acceptance does not make a security invariant true, and its orphan cleanup was unimplementable - the server cannot observe Set-Cookie acceptance); the page leg is permission-gated before the module executes; the cross-key commit atomicity gap is recorded as open question 0. - rewrite_legacy is cut from the epic into a recorded deferral carrying its open problems (retention lineage, eventual-store visibility, chain stranding, cluster inflation) as the entry bar for a future spec; provider switching is served by legacy readers alone; the key is rejected as unknown. - The hook ships headers-only: the write-side cookie gate never modeled reading, using, forwarding, or withdrawing an integration cookie (or its P4 nature), so cookie operations defer to a follow-up spec with that full model as entry bar; sign-off items 9/10 updated. Core fixes (new P1/P2): - Recognized rowless legacy cookies (graphless deployments) get a permission-gated, race-safe adoption transaction; no egress before adoption; withdrawal needs no adoption (derived family ID); matrix row 13. - Unreferenced [ec.providers.*] blocks are startup errors - a dropped legacy_providers entry must not silently strand identities. - Physical key delimiters are backend-safe and validated per adapter (Fastly forbids / in prefix queries); cluster eligibility requires a queryable physical prefix, checked at startup. - Cluster size means live identity rows: kind/liveness filtering, short-TTL tombstone inflation declared conservative. - Validation split into structural (push + startup) and deployment (startup; optional push pre-check via a machine-readable capability profile) - 'same validation at push' was unimplementable. - The rollback floor is N+2 writer activation itself, recorded as a durable schema-floor marker - not an unobservable first-row fact. - Integration IDs are startup-unique. Previously-open items closed: - GPP map completed against the official registry: section 6 (US Privacy as GPP section) and 24-27 (MD/IN/KY/RI - the earlier claim they had no sections was wrong). - State-over-national applies to grants only; a national opt-out can never be erased by a state field. - Suppression records completed: full negative-state coverage, monotonic per-permission ordering, re-consent clearing, write-failure semantics. - The raw-TCF dispatch arm triggers on TCF-sourced effective records including the persisted-KV fallback. - N+1 writes the safety-critical record kinds (family, suppression) and accepts N+2-only providers as legacy readers, making the rollout boundary safe in both directions. - Request-side integration views are identity-redacted; the legacy RequestFilterEffects.response_headers channel is folded into the hook. - must-understand and friends join the sticky directive set; the Axum matrix cell is honest (in-process, non-durable, dev-only). Process: docs/superpowers/specs/pr986-review-ledger.md records the disposition of every finding from all seven review rounds (fixed / reapplied-after-batch-loss / partial-refixed / superseded / deferred / open), so coverage is auditable per finding rather than claimed in summaries. --- ...26-07-30-client-cycle-ec-resolve-design.md | 51 ++++-- ...integration-response-header-hook-design.md | 63 ++++---- .../2026-07-30-permission-model-design.md | 99 +++++++----- .../2026-07-30-pluggable-providers-design.md | 149 +++++++++--------- ...07-30-provider-migration-rollout-design.md | 74 +++++---- docs/superpowers/specs/pr986-review-ledger.md | 132 ++++++++++++++++ 6 files changed, 385 insertions(+), 183 deletions(-) create mode 100644 docs/superpowers/specs/pr986-review-ledger.md diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index 3cfeb09d4..c1ea840ed 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -1,7 +1,12 @@ # Design Spec: Client-Cycle Edge Cookie Providers and the Resolve Endpoint -**Status:** Draft — **prerequisites unmet; do not implement against this spec -until its open questions (§7) are resolved in a dedicated issue** +**Status:** **Deferred — informative draft, not part of the epic's +normative set.** No production adapter has the required CAS-class +primitive (providers spec §7 matrix), the feature has no concrete +consumer, and successive reviews keep finding open protocol questions +(§7). It re-enters the epic only through its own dedicated issue, with +this document as the starting bar — findings against this spec do not +block ratification of the core specs. **Author:** Engineering **Issue references:** none yet (this spec exists to force one; #778 does not cover this feature) @@ -69,15 +74,19 @@ Everything in this spec follows from that. that are signed by an expected party, **audience-bound** to this publisher, and **expiring**. Audience binding and expiry alone do not mitigate replay — a captured token installs in another browser for the - whole validity window. **Production schemes require session binding** - (a server-issued nonce the payload must embed): one-time consumption - alone limits multiplicity but proves nothing about _which_ browser - redeems first — a captured bearer payload can simply win the race — so - it is defense-in-depth, not the mitigation. First-presenter - at-most-once semantics may ship **only** as an explicitly accepted - posture recorded in the feature's issue, together with a specified - orphan-row cleanup path. "Single-use where the scheme allows" is not a - mitigation. + whole validity window. **Session binding is required, with no ownerless + escape hatch** (a server-issued nonce the payload must embed): + one-time consumption alone limits multiplicity but proves nothing + about _which_ browser redeems first — a captured bearer payload can + simply win the race — so it is defense-in-depth, not the mitigation. + An earlier draft allowed a "first-presenter, product-accepted" + ownerless mode; it is **removed**: risk acceptance does not make a + security invariant true, and the promised orphan cleanup was + unimplementable anyway — the server cannot observe `Set-Cookie` + acceptance, so it cannot distinguish a lost-response orphan from a + successful-but-dormant identity. A scheme that cannot embed the + session nonce cannot ship. "Single-use where the scheme allows" is + not a mitigation. 3. **Preserve the identity-graph invariant.** The cookie is set only after the corresponding graph row is written, mirroring the organic path. Graph unavailable → no cookie, same as organic generation. @@ -158,10 +167,7 @@ Everything in this spec follows from that. have the `Set-Cookie` re-emitted — which is precisely how a legitimate browser whose original response was lost recovers on retry, so a committed graph row never strands as an orphan for the intended - browser; anyone else gets a terminal response with no cookie. In the - explicitly-accepted at-most-once posture (no owner hash), a lost - response is an **orphan row** handled by the specified cleanup path. - The **same-identity no-op of §3.8 first checks the family revocation + browser; anyone else gets a terminal response with no cookie. The **same-identity no-op of §3.8 first checks the family revocation record**, and "revocation wins" is enforced by an explicit linearization point, not by read strength alone — a linearizable read followed by a separate commit still loses the race (read "not @@ -181,6 +187,16 @@ Everything in this spec follows from that. ## 4. Requirements on the page script +**The page leg is permission-gated before it executes.** The browser +module obtains or derives a vendor identity — vendor contact, stable +identifier in hand — so injecting it whenever the provider is merely +_selected_ would run identity code for a visitor who denied everything, +with only the later POST refused. The module is injected/activated only +when the request's resolved permissions already satisfy the provider's +complete `required_permissions()`, and the page leg is a listed row in +the permission spec's §7 enforcement inventory (deferred alongside this +feature). + - The re-post guard must not depend on reading an HttpOnly cookie. Either the server injects a "resolved" marker the script _can_ read (a non-identity companion cookie or an injected page variable), or the @@ -219,6 +235,11 @@ sentence as the only guardrail. ## 7. Open questions — to be settled in the feature's issue before any code +0. The commit path spans keys (reservation, identity row, family-epoch + CAS): the cross-key atomicity or saga/compensation design is + **undefined** — the single-key CAS steps are specified, their + composition is not. + 1. Which concrete vendor scheme is the first real consumer, and does its envelope format satisfy §3.2 (audience binding, expiry)? If no concrete consumer exists, the feature waits — the demo provider is not a diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 5d35f23a5..a8a7ef7cb 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -67,7 +67,8 @@ mutators to the outbound response for HTML document responses it processed. storage subject to revalidation; `private` forbids shared storage), so "replace `private` with the stronger `no-cache`" would make a personalized response shared-storable. The merge: each of `no-store`, - `no-cache`, `private` is **sticky** — present in the snapshot or the + `no-cache`, `private`, `must-revalidate`, `proxy-revalidate`, + `must-understand`, and `no-transform` is **sticky** — present in the snapshot or the mutation ⇒ present in the final response, independently; `public` is dropped whenever any restriction is present; `max-age`/`s-maxage` may only shrink relative to the snapshot; `stale-while-revalidate`/ @@ -94,22 +95,21 @@ mutators to the outbound response for HTML document responses it processed. digest for bytes the hook never saw, corrupts responses or poisons caches), the `x-ts-*` namespace, and the consent/privacy headers core emits; (b) reserved cookie _names_ within `Set-Cookie` — `ts-ec`, - `ts-eids`, and the other `ts-*` cookies core owns. Integration cookies are **inside the permission model, not beside it** - (product sign-off item 9, migration spec §8) — otherwise the hook is a - door around the EC gate: an integration could write a durable - identifier while `store-on-device` is denied. `append_set_cookie` - therefore requires the cookie name to be **declared at registration** - with a stated purpose and maximum retention; a **persistent** cookie - (any `Max-Age`/`Expires`) is applied only when the request's resolved - permissions include `store-on-device`, while **session cookies** (no - persistence attributes) are the narrow, documented exemption. - Cookie operations go through a **typed cookie builder** that enforces - the declared lifetime ceiling, domain/path scope, and security - attributes (`Secure`, `SameSite`) — not a free-form string; **deletion - cookies (expiry of the integration's own declared names) remain - possible when `store-on-device` is denied**, since removing state must - never require the permission to keep it. Undeclared cookie names are - rejected like reserved ones. An integration may never set or expire a + `ts-eids`, and the other `ts-*` cookies core owns. **Cookie operations are deferred out of the v1 hook — headers only.** + The write-side gate alone ("persistent cookies require P1") was shown + insufficient: it never modeled reading, using, forwarding, or + withdrawing the cookie — a P1-granted-then-withdrawn integration + cookie would keep arriving on every request with nothing required to + expire, hide, or stop egressing it, and an advertising-identifier + cookie needs P4 the contract never expressed. Rather than ship + "inside the permission model" as a claim the model does not back, + `append_set_cookie` and the typed cookie builder are **deferred** to a + follow-up spec whose entry bar is: declared per-cookie required + permissions, a typed authorized request-side view, stripping from + unauthorized integration/proxy inputs, mandatory expiry on destructive + P1 withdrawal, and startup-unique (name, domain, path) ownership. + Integration IDs are startup-unique regardless. Until then the + operation set is headers-only, and `Set-Cookie` is fully reserved. reserved cookie name. Violations are rejected at the operation layer (§2) and logged at `warn` with the integration id. The reserved lists are single constants next to the definitions they protect, not duplicated in the @@ -174,19 +174,24 @@ processed documents (§6). ## 4. Done-when (from #782, sharpened) 1. Trait + builder + registry application, each public item documented. -2. **At least one real consumer ships in the same PR** — an existing +2. **The pre-existing `RequestFilterEffects.response_headers` channel is + folded into the hook in the same PR** — its outputs become hook + operations subject to the same validation, reserved surface, budgets, + and invariant pass, or the channel is removed; a second, unvalidated + header path bypassing the hook defeats every rule above. +3. **At least one real consumer ships in the same PR** — an existing integration registering a mutator for a real need (or, failing a real need, the feature waits; scaffolding with only self-referential tests is dead code and will be removed). -3. Every adapter applies mutations on its outbound path, with a per-adapter +4. Every adapter applies mutations on its outbound path, with a per-adapter route test asserting an integration-set header appears in the response. -4. A parity-suite case asserts identical mutation behavior across adapters. -5. Reserved-surface, append/replace, operation-limit, and erroring-mutator +5. A parity-suite case asserts identical mutation behavior across adapters. +6. Reserved-surface, append/replace, operation-limit, and erroring-mutator semantics covered by unit tests. -6. **Every row of the §3a eligibility matrix has a test** — streaming, +7. **Every row of the §3a eligibility matrix has a test** — streaming, cache-hit, pass-through, redirect, error, and 304 each proven to run or not run the hook — not merely one positive header test per adapter. -7. Cache/privacy invariant tests, one per restriction source and shape: +8. Cache/privacy invariant tests, one per restriction source and shape: cookie appended + public `Cache-Control` replacement → private/no-store, surrogate stripped; **core-private cookieless** processed HTML + public replacement → restriction preserved; **origin-private cookieless** processed HTML that retained the @@ -199,13 +204,11 @@ processed documents (§6). ## 5. Size and sequencing -This is a modest feature plus tests, with zero coupling to the provider -architecture — but its **cookie operations are coupled to the permission -model** (§3; the gate is a listed enforcement point in the permission -spec §7 inventory), so the claim of total independence is retired: the -header-mutation portion may land whenever its first real consumer is -identified (§4.2), while `append_set_cookie` activates only **after** the -permission model PR, and registers as unavailable before it. If no consumer +This is a modest feature plus tests with zero coupling to the provider +architecture or, in its v1 headers-only form (§3), to the permission +model. It lands whenever its first real consumer is identified (§4.2); +cookie operations arrive only with their own follow-up spec (§3) and its +permission-model coupling. If no consumer materializes, it does not land; being unblocked is not a reason to ship scaffolding. diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 8a0818166..4a3feb629 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -423,12 +423,23 @@ and the fail-closed marker: which the refusal just unset, and identity rows may be eventually consistent, so a stale replica could resurrect a P4 grant after a targeted-advertising opt-out. The fix is a **suppression record** in - the strongly consistent class (providers spec §6.3: `sup/`), - carrying per-permission suppression entries with timestamps. Writing it - is **permission-exempt** (clearing authority is protective, like - revocation), and **every S2S recompute and partner-egress check - consults it**: a suppressed permission is unset whatever the row's - provenance says, so no eventual-consistency edge can restore it. + the strongly consistent class (providers spec §6.3: `sup/`). + Its semantics are complete, not sketched: entries are **per permission** + with the triggering state (refusal or non-destructive opt-out — the + full negative-state coverage; absence writes nothing) and an + authoritative timestamp; ordering is **monotonic per permission** — + a write with an older timestamp than the stored entry is a no-op, so + replays cannot regress the state; **re-consent clears**: a live + resolution carrying an accepted grant with a newer authoritative + timestamp than the suppression entry supersedes it (recorded as a + clear entry in the same record — still an exempt write, since it only + ever reflects the live resolution); and **write failure fails closed + for the live request** (the refusal's effect stands for this response) + while S2S may transiently honor prior authority until the retry lands — + logged, metered, and covered by the degraded-mode rule. Every S2S + recompute and partner-egress check consults the record: a suppressed + permission is unset whatever the row's provenance says, so no + eventual-consistency edge can restore it. - **The cookie expires only after the family record commits.** - **If the family-record write itself fails, nothing durable exists** — the cookie stays and the durable client-side signal (GPC, CMP-stored @@ -522,16 +533,19 @@ nothing. **Applicability and aggregation — ordered algorithm:** 1. **Section map (normative, pinned here — not "whatever GPP is - current"), covering every section current code recognizes (7–23), not - a subset:** `US` national ↔ 7 (usnat); then the state sections — - `US/CA` ↔ 8, `US/VA` ↔ 9, `US/CO` ↔ 10, `US/UT` ↔ 11, `US/CT` ↔ 12, - `US/FL` ↔ 13, `US/MT` ↔ 14, `US/OR` ↔ 15, `US/TX` ↔ 16, `US/DE` ↔ 17, - `US/IA` ↔ 18, `US/NE` ↔ 19, `US/NH` ↔ 20, `US/NJ` ↔ 21, `US/TN` ↔ 22, - `US/MN` ↔ 23. Dropping to 7–12 would silently lose, e.g., a Texas - (section 16) sale opt-out. The implementation PR cross-checks this - list against the current decoder's section set; versions are those - published at this spec's date; adding a section or version is a change - to this map. + current"), matching the official IAB registry in full:** section 6 ↔ + the **US Privacy string carried as a GPP section** (it maps to the USP + rows of the field table, not to nothing); `US` national ↔ 7 (usnat); + the state sections — `US/CA` ↔ 8, `US/VA` ↔ 9, `US/CO` ↔ 10, + `US/UT` ↔ 11, `US/CT` ↔ 12, `US/FL` ↔ 13, `US/MT` ↔ 14, `US/OR` ↔ 15, + `US/TX` ↔ 16, `US/DE` ↔ 17, `US/IA` ↔ 18, `US/NE` ↔ 19, `US/NH` ↔ 20, + `US/NJ` ↔ 21, `US/TN` ↔ 22, `US/MN` ↔ 23, **`US/MD` ↔ 24, + `US/IN` ↔ 25, `US/KY` ↔ 26, `US/RI` ↔ 27** (an earlier draft wrongly + claimed MD/IN/KY/RI had no section). A truncated map silently loses + opt-outs — a Texas (16) or Maryland (24) sale opt-out must not vanish. + The implementation PR cross-checks this list against both the current + decoder's section set and the official registry; adding a section or + version is a change to this map. 2. **Applicability gates grants only — never opt-outs.** A mapped **opt-out** field (either subclass) is honored from **any** section on **any** request, whatever the regime — this is §4's global-opt-out @@ -544,9 +558,14 @@ nothing. national section only. A configured privacy state with no state-specific section (e.g. MD, IN, KY, RI today) uses the national section alone. -3. **State-over-national, per field:** where an applicable state section - carries a field, it governs that field; the national section fills only - fields the state section lacks. +3. **State-over-national, per field — for grants only:** where an + applicable state section carries a field, its value governs that + field's **grant** derivation; the national section fills only fields + the state section lacks. This precedence **never suppresses an + opt-out**: a national-section opt-out stands even where the state + section's same field says not-opted-out — step 2's global rule wins, + or a state string could erase a globally authoritative national + opt-out. 4. **Aggregate across what remains applicable:** an opt-out (of either subclass) in any applicable field beats a grant from another — restrictive aggregation. @@ -677,21 +696,21 @@ Consumers of the resolved set in this epic: inventory, normative per path (one test per row; a denylist check proves no ungated egress exists): - | Path | Required permissions | Notes | - | ---------------------------------------------------------------- | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | OpenRTB `user.id` | `store-on-device` ∧ `select-personalised-ads` | Raw EC is identity in the bidstream — gated exactly as EIDs. PR #838 gated only EIDs, leaving `user.id` reachable with Purpose 4 refused | - | EC-derived auction request IDs | both purposes | Derived values are identity | - | Page-bids path | both purposes | | - | Bidstream EIDs | both purposes | The one gate PR #838 had | - | Proxy / click / Testlight forwarding of the EC cookie or headers | both purposes | **New hardening, declared change** — these paths extract the raw cookie/header without today's jurisdiction gate (migration spec §2 row 11b) | - | Identify endpoint (partner-facing) | both purposes | Partner identity exchange, not a first-party lookup — decided here | - | Pull sync (browser-request-scoped partner exchange) | both purposes, from the **live** request resolution | Pull sync is created from a browser request and checks the live `EcContext` today — it keeps using the live P1 ∧ P4 decision plus the family revocation state (§4.3); stored provenance is never a substitute for available live evidence | - | Batch sync (context-free S2S partner exchange) | both purposes, from **stored provenance** | The only truly signal-less path; authority rules below. Today's handler only authenticates and checks row state, so this gate is **declared hardening** (migration spec §2) | - | Request-scoped graph reads/writes (non-revocation) | `store-on-device` | | - | Revocation paths (tombstones, withdrawal reads) | **exempt** | Must work when permissions are unset | - | Stored consent-state lookup (§4.4) | **exempt**, narrowly scoped | Determining `store-on-device` cannot require `store-on-device` | - | Integration persistent response cookies (hook spec §3) | `store-on-device` | Applied at mutation time from the request's resolved permissions; session cookies are the declared exemption (sign-off items 9–10) | - | Suppression-record writes (§4.3) | **exempt** | Clearing authority is protective, like revocation | + | Path | Required permissions | Notes | + | ---------------------------------------------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | OpenRTB `user.id` | `store-on-device` ∧ `select-personalised-ads` | Raw EC is identity in the bidstream — gated exactly as EIDs. PR #838 gated only EIDs, leaving `user.id` reachable with Purpose 4 refused | + | EC-derived auction request IDs | both purposes | Derived values are identity | + | Page-bids path | both purposes | | + | Bidstream EIDs | both purposes | The one gate PR #838 had | + | Proxy / click / Testlight forwarding of the EC cookie or headers | both purposes | **New hardening, declared change** — these paths extract the raw cookie/header without today's jurisdiction gate (migration spec §2 row 11b) | + | Identify endpoint (partner-facing) | both purposes | Partner identity exchange, not a first-party lookup — decided here | + | Pull sync (browser-request-scoped partner exchange) | both purposes, from the **live** request resolution | Pull sync is created from a browser request and checks the live `EcContext` today — it keeps using the live P1 ∧ P4 decision plus the family revocation state (§4.3); stored provenance is never a substitute for available live evidence | + | Batch sync (context-free S2S partner exchange) | both purposes, from **stored provenance** | The only truly signal-less path; authority rules below. Today's handler only authenticates and checks row state, so this gate is **declared hardening** (migration spec §2) | + | Request-scoped graph reads/writes (non-revocation) | `store-on-device` | | + | Revocation paths (tombstones, withdrawal reads) | **exempt** | Must work when permissions are unset | + | Stored consent-state lookup (§4.4) | **exempt**, narrowly scoped | Determining `store-on-device` cannot require `store-on-device` | + | Integration persistent response cookies | `store-on-device` (+ P4 where the cookie is an advertising identifier) | **Deferred with the hook's cookie surface** — the write-side gate alone was insufficient (read/use/forward/withdrawal unmodeled), so cookie operations ship only with the full model; this row and the client-cycle **page leg** (module injection gated on the provider's full declaration) join the inventory when their features do, and the §5.3 no-geo guard's consumer list grows with them | + | Suppression-record writes (§4.3) | **exempt** | Clearing authority is protective, like revocation | With **no EC provider configured**, identity use fails closed: a cookie value present on the request never egresses anywhere — never vacuously @@ -739,12 +758,12 @@ Consumers of the resolved set in this epic: 4. **Server-side auction dispatch** — gated on the policy `regime` class, normatively: - | Regime | Dispatch rule | Preserves | - | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | - | `gdpr` | Dispatch only with a decodable, unexpired TCF record consenting to Purpose 1. Malformed, expired, or absent record → **no bid request leaves** (no-bid response). | Today's GDPR/unknown arm | - | `us-privacy` | Dispatch proceeds in every signal state, including opt-out — the opt-out strips identity (rows above) but the contextual auction runs. | Today's US-state arm | - | `none` | Dispatch proceeds. | Today's non-regulated arm | - | **Any regime, raw TCF signal present** — a TC string on the request or a GPP section-2 hint, detected **before decoding** | The `gdpr` row applies: dispatch requires the _effective_ record to be decodable, unexpired, and consenting to Purpose 1. A **malformed or expired** raw signal therefore blocks dispatch — today a malformed raw TCF blocks, and gating this arm on decodability would have silently relaxed that. A US or non-regulated request carrying a Purpose 1 refusal is likewise blocked. | Today's raw-signal arm — **must not regress** | + | Regime | Dispatch rule | Preserves | + | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | + | `gdpr` | Dispatch only with a decodable, unexpired TCF record consenting to Purpose 1. Malformed, expired, or absent record → **no bid request leaves** (no-bid response). | Today's GDPR/unknown arm | + | `us-privacy` | Dispatch proceeds in every signal state, including opt-out — the opt-out strips identity (rows above) but the contextual auction runs. | Today's US-state arm | + | `none` | Dispatch proceeds. | Today's non-regulated arm | + | **Any regime, TCF-sourced effective record** — a raw TC string on the request, a GPP section-2 hint (both detected **before decoding**), or a persisted-KV fallback record of TCF origin (§4.4) | The `gdpr` row applies: dispatch requires the _effective_ record to be decodable, unexpired, and consenting to Purpose 1. A **malformed or expired** raw signal therefore blocks dispatch — today a malformed raw TCF blocks, and gating this arm on decodability would have silently relaxed that. A US or non-regulated request carrying a Purpose 1 refusal is likewise blocked. | Today's raw-signal arm — **must not regress** | The **compiled-in fallback policy has `regime = "gdpr"`** (§3.1) — the no-policy posture must be the most protective for dispatch too, and a diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index d34382907..c92253a3d 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -142,6 +142,16 @@ Three global rules sit above every provider: a prefix-listing case. For `hmac`, the equivalence fixtures pin: uppercase/lowercase hex-prefix variants are equivalent; suffix case is preserved and significant. +- **Cluster size means live identity rows.** Prefix counting lists + identity-row keys; family, suppression, reservation, and transaction + records live in other namespaces and never inflate a count. Member + tombstones share the identity key but carry the short cleanup TTL, so + their inflation is transient and biases conservative (an over-count + trips the trust threshold toward denial, never toward extra writes); + the listing filters by the value's `kind`/liveness within the existing + list limit where the backend returns values, and the residual + over-count where it cannot is declared. Aliases are reserved-future + (§6.1) and excluded by `kind` when they exist. - **No-cluster behavior is still defined.** A provider without cluster support deduplicates pull-sync by canonical graph key and redacts logs with a fixed-length hash of the graph key; `cluster_fallback` (§6.1) @@ -241,6 +251,20 @@ header emission; the identity exists durably from that moment. A graph-commit failure means the mint never happened: no cookie, no egress, error logged, the next request retries. +**Pre-existing cookies without rows are adopted, not orphaned.** Current +graphless deployments have minted cookies with no row; under +no-active-until-commit those identities could never be used again and — +without care — never withdrawn. The contract: a recognized legacy cookie +with no reachable row triggers a **permission-gated, race-safe adoption** +on a live request — gated exactly like minting (`store-on-device`), +implemented as create-if-absent on the verbatim key (concurrent adopters +converge: same key, same deterministic family ID), provenance from the +live resolution. Until adoption succeeds the cookie **never egresses**; +**withdrawal works without adoption** — the deterministic family ID +(permission model spec §4.3) needs no row, so a first post-upgrade +request that is an opt-out revokes and expires the cookie with zero +migrated state. Migration matrix row 13 declares this path. + **Egress is typed, not policed.** The inventory-and-denylist test (permission model spec §7) is a backstop, but conventions do not survive new code — the ungated proxy/click/Testlight paths happened precisely @@ -254,7 +278,13 @@ bids, sync, identify, forwarding) accept `AuthorizedIdentity` and nothing weaker — an unparameterized wrapper would let a P1-only identity flow into an ORTB request. A future bypass then requires deliberately reconstructing the raw string — visible in review — -rather than passing along what was already in hand. +rather than passing along what was already in hand. The same boundary +applies **request-side**: integration-facing request views (proxy +interfaces, filter inputs, forwarded header/cookie maps) receive +**identity-redacted** views — the EC cookie and identity headers are +stripped unless the path holds `AuthorizedIdentity` — +because the ungated forwarding paths of PR #838 were exactly integrations +reading the raw request. The gate applies to EC providers **only**. Geo and device are ungated for two _different_ reasons, stated separately because only one of them is @@ -300,16 +330,17 @@ one. This spec resolves that by **not having** the method on those traits All validation happens at **settings construction** — a misconfiguration is a startup error, never a request-time error and never a silent behavior change. -| Configuration state | Behavior | -| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `provider` names an unknown key | Startup error listing valid keys. | -| `provider` set, its `[ec.providers.]` block missing | Startup error. | -| `[ec.providers.]` block present, `provider` unset | **Startup error.** (In PR #838 this silently ran stateless — the half-migrated config becomes a production identity outage detected by revenue drop. Rejecting it is the fix.) An operator who genuinely wants stateless deletes the block. | -| `provider` set to an implementation the running adapter cannot satisfy (e.g. a provider requiring host TLS fingerprints on an adapter that has none) | Startup error at adapter wiring time. Adapters declare their host capabilities to the composition root; the root checks the selected provider's needs against them **once**, at startup — not per request. | -| No `provider`, no providers block | Valid: the neutral default for that concern. | -| `rewrite_legacy = true` with a **client-resolve** active writer | **Startup error.** An organic request carries no signed client payload to mint from, and a later resolve POST meeting a different existing identity is a `409` by the client-cycle spec — the combination is incoherent until an authenticated linking/migration flow is specified. | -| `provider = "none"` (explicit stateless) | Valid, and the only way to combine statelessness with `legacy_providers`: minting stops, legacy readers keep existing identities resolvable and **withdrawable** (§6.1). Without this state, `hmac` → stateless would strand every live row in revoke-proof limbo. | -| A minting provider (or any `legacy_providers`) configured, but no identity-graph store configured or openable | **Startup error.** The lifecycle contract assumes graph persistence (§5); discovering its absence at first mint would be a request-time config failure, which this table exists to forbid. | +| Configuration state | Behavior | +| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `provider` names an unknown key | Startup error listing valid keys. | +| `provider` set, its `[ec.providers.]` block missing | Startup error. | +| `[ec.providers.]` block present, `provider` unset | **Startup error.** (In PR #838 this silently ran stateless — the half-migrated config becomes a production identity outage detected by revenue drop. Rejecting it is the fix.) An operator who genuinely wants stateless deletes the block. | +| `provider` set to an implementation the running adapter cannot satisfy (e.g. a provider requiring host TLS fingerprints on an adapter that has none) | Startup error at adapter wiring time. Adapters declare their host capabilities to the composition root; the root checks the selected provider's needs against them **once**, at startup — not per request. | +| A `[ec.providers.]` block referenced by neither `provider`, `legacy_providers`, nor a `versions`/`mint_version` chain | **Startup error.** An unreferenced block is almost always a dropped `legacy_providers` entry — accepted silently, it strands every identity that provider minted: unresolvable and, worse, non-withdrawable. | +| No `provider`, no providers block | Valid: the neutral default for that concern. | +| `rewrite_legacy` present at all (deferred out of the epic, §6.1) | **Startup error** — unknown key; transparent re-mint returns only with its own spec. | +| `provider = "none"` (explicit stateless) | Valid, and the only way to combine statelessness with `legacy_providers`: minting stops, legacy readers keep existing identities resolvable and **withdrawable** (§6.1). Without this state, `hmac` → stateless would strand every live row in revoke-proof limbo. | +| A minting provider (or any `legacy_providers`) configured, but no identity-graph store configured or openable | **Startup error.** The lifecycle contract assumes graph persistence (§5); discovering its absence at first mint would be a request-time config failure, which this table exists to forbid. | Unknown fields inside every provider config block are rejected (`deny_unknown_fields` on all new settings structs — the pre-existing `Ec` @@ -357,56 +388,22 @@ The contract: unavailable (a cookie with no reachable row). Removing a version entry is a retirement subject to the same evidence rules as retiring a legacy reader (migration spec §6). -- A cookie recognized by a legacy reader is a live identity for - read/withdrawal purposes; whether it is transparently re-minted under the - active writer is a per-deployment choice - (`[ec] rewrite_legacy = true|false`), and re-minting is subject to the - full minting gate of §5. -- **Rewrite is a persistent fenced transaction aliasing to one canonical - row — no dual-write window, no duplicate targets, no lost updates.** - The steps, each resumable because the transaction record (its own - linearizable record class, §7 matrix) is written **first** and pins the - chosen target key and fencing epoch: - 1. **Transaction record** commits: source key, target key, epoch, - state. A crashed rewrite retried later reads it and resumes with - the **same** target — a fresh random target (and an orphaned first - one) cannot exist, and any target row without a committed transaction - pointing at it is garbage-collectable by that absence. - 2. **Canonical row** commits under the pinned target key, sharing the - old row's revocation family ID (permission model spec §4.3); - provenance is the **current live resolution** (copying old consent - evidence would rejuvenate stale authority); partner mappings copy - with their **original per-field timestamps and expiry**, and the - copy point is recorded in the transaction. - 3. **A per-key CAS on the source identity key replaces the row value - with the alias** (same address, `kind` discriminator — §6.3). This - is why rewrite requires the row store itself to offer CAS with - read-your-writes (§7 matrix): the participants must share - transactional primitives, or a source update landing through an - eventual replica after the copy could be silently dropped. If the - CAS loses to a concurrent pull/batch/identify update, the rewrite - **re-runs a reconciliation pass** under its epoch — re-reading the - source with the store's strongest read, merging updates newer than - the recorded copy point into the canonical — and retries; an update - that won the old row is therefore never lost. - 4. The new cookie is emitted; the transaction marks complete. - - From step 3 on, every read or update through either cookie chases the - alias to the single canonical row. Chains are handled by **bounded - traversal with path compression**, not an inbound-alias index (an - index would need its own fenced schema, bounds, and concurrency rules - that nothing defined): traversal follows at most **4** hops with - visited-set cycle detection (deeper or cyclic → treated as row-read - failure, fail closed per §6.2); whenever a traversal crosses more than - one hop, it opportunistically CASes the first alias to point at the - final canonical, so chains converge to one hop without coordination. The server - cannot observe `Set-Cookie` acceptance, so the alias stays live until a - later request **presents the new cookie**, and in any case until a - **finite retirement deadline** no shorter than the old cookie's maximum - lifetime plus rollout skew. An interrupted rewrite at any step leaves - the old cookie resolving — no state in which neither identity works. - **Withdrawal through either cookie revokes the shared family.** - +- **`rewrite_legacy` is deferred out of the epic.** Transparent re-mint + under the active writer required primitives no production adapter has + (row-store CAS with read-your-writes plus a linearizable transaction + class — §7 matrix), and successive reviews kept surfacing open protocol + problems: retention lineage (a rewritten 364-day-old row either + rejuvenates the identity or leaves a year-long cookie pointing at an + expiring row — a lineage expiry must be pinned across canonical row, + alias, family record, and emitted cookie), alias visibility under + eventual stores, chain stranding after repeated migrations, and + cluster-count inflation by alias keys. Those are recorded here as the + entry bar for a future `rewrite_legacy` spec. Within the epic, provider + switching is served by **legacy readers alone**: old identities keep + resolving and stay withdrawable; they are never transparently + re-minted. The `rewrite_legacy` key is rejected at startup as unknown, + and the alias record class exists in the key grammar (§6.3) only as + reserved-for-future — nothing in the epic writes one. - Retiring a legacy reader is the explicit end of those identities: the migration guide documents the cleanup procedure (migration spec §6). - Tests: switch active provider → request with old cookie → identity still @@ -478,7 +475,17 @@ Grammars are pairwise non-intersecting by their literal prefixes (plus the reserved hmac grammar), and every record value carries a `kind` discriminator alongside its schema version — so a reader always knows what it fetched, including where two classes deliberately share an -address (row vs. alias). +address (row vs. alias). The `/` shown in key sketches is **notation, +not the wire byte**: the physical segment delimiter is a +**backend-safe character validated per adapter** — Fastly permits `/` in +keys but not in prefix _queries_, so a slash-delimited `id//…` +key could never be cluster-listed there despite the matrix marking +prefix listing supported. The reference delimiter is `:`; each adapter's +capability declaration includes which delimiter its prefix queries +accept, and cluster-capability eligibility for a provider requires its +physical prefix to be queryable on that backend — checked at startup, +not discovered at the first cluster count. (hmac verbatim keys contain +no delimiter before the 64-hex prefix and are unaffected.) **Wire schemas** (JSON, like identity rows; every class carries a schema version): the **alias record** holds target key, created-at, retirement @@ -550,7 +557,7 @@ Requirements: | Family revocation records | **Strongly consistent (read-after-write) primitive required.** Cloudflare Workers KV is **not eligible** — its documentation says propagation may take "60 seconds or more", an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | | Family suppression records | Same strong class as family revocation — negative authority must not lose races to stale replicas | | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | - | Alias installs | Row-store **per-key CAS with read-your-writes** — the alias lives at the source identity key (§6.3), so the _row store itself_ must supply the CAS; a purely eventual row store cannot host rewrite. `rewrite_legacy = true` is rejected at startup unless both this and the transaction class are available (§6) | + | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | | Identity rows | Eventual consistency acceptable — rows are accretive, and the family-record check (strong, per above) governs use | Each adapter's declaration is part of its wiring, drives the §6 @@ -566,14 +573,14 @@ Requirements: them is how a "yes" cell hides an unusable feature. Feature eligibility requires wired, not merely available: - | Capability | Fastly | Axum (dev) | Cloudflare | Spin | - | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | - | Graph persistence (eventual OK) | KV Store: available + wired | Local store: available + wired (dev-grade) | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | - | Prefix listing (cluster) | Yes (used today) | Yes | Yes (eventual) | _verify_ | - | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | Yes (in-process) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | - | Linearizable fenced CAS (reservations, alias/rewrite) | **Not currently available** — client-cycle and `rewrite_legacy` unselectable until a primitive exists | Yes (in-process) | Durable Objects: possible, not wired | **No** | - | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | - | Device host evidence (JA4/H2) | Yes | No | No | No | + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | ----------------------------------------------------- | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | + | Graph persistence (eventual OK) | KV Store: available + wired | Local store: available + wired (dev-grade) | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | + | Prefix listing (cluster) | Yes (used today) | Yes | Yes (eventual) | _verify_ | + | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | Yes (in-process) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | + | Linearizable fenced CAS (reservations, alias/rewrite) | **Not currently available** — the client-cycle feature (deferred) would need it | Yes — in-process only: linearizable but **non-durable**, dev-eligibility only, not a production persistence claim | Durable Objects: possible, not wired | **No** | + | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | + | Device host evidence (JA4/H2) | Yes | No | No | No | - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index ccd48ac0b..4be41a00a 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -55,6 +55,7 @@ discoverable only because a deleted test had pinned the old behavior. | 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | | 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | | 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | +| 13 | Cookies minted by graphless deployments have no graph row | Recognized rowless legacy cookies are adopted via a permission-gated, race-safe create-if-absent on a live request (providers spec §5); never egress before adoption; withdrawal works without adoption via the derived family ID | **Declared** — identity use of pre-existing cookies pauses until adopted | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -124,10 +125,19 @@ Requirements: Preserving unknown JSON does not chase aliases, consult family revocations, honor suppression records, or fail closed on provenance; an N+1 that merely preserved would, after rollback, - treat aliased rows as missing and revoked identities as live. - Rollback tests therefore run the alias, family-revocation, - suppression, and provenance paths **on N+1** against N+2-written - data. **Rollback is binaries-first too, in the other direction** — + treat revoked identities as live (aliases are reserved-future with + the rewrite deferral, providers spec §6.1). N+1 must also **write** + the safety-critical record kinds — family revocation and + suppression — not only read them: a withdrawal arriving on a + rolled-back N+1 fleet must still revoke; only provenance _writing_ + is deferred to N+2, which is what makes the boundary safe in both + directions. N+1 further **accepts an N+2-only provider in the + `legacy_providers` position** (parse/withdraw need no provenance + encoding) while rejecting it as active writer — otherwise the + rollback rule "retain the new provider as a legacy reader" would be + unsatisfiable on the very release it targets. Rollback tests + therefore run the family-revocation, suppression, and provenance + paths — read **and write** — on N+1 against N+2-written data. **Rollback is binaries-first too, in the other direction** — N+2 → N+1 binaries roll back keeping the new config (N+1 reads it fully; reverting config first would hand the old shape to N+2 binaries that reject it) — **with one precondition**: if an @@ -183,9 +193,12 @@ Requirements: required nor achievable through a structured serializer — and a genuinely pre-N+1 worker cannot preserve at all, which is exactly why the floor exists); after the **fleet-convergence gate**, **N+2 - activates the writer** and begins emitting the new fields. **Rollback - below N+1 is prohibited once any new-format row exists** — a pre-floor - binary would silently strip the new fields from every row it touches. + activates the writer** and begins emitting the new fields. **The rollback floor is crossed at N+2 writer activation itself** — an + observable deploy event, recorded as a durable schema-floor marker in + the config store before writes enable — not at "any new-format row + exists", which no operator can disprove. Below-floor rollback is + prohibited from that marker on; a pre-floor binary would silently + strip the new fields from every row it touches. Rows carry the existing `v` schema discriminator; backfill is lazy via live requests (the same pass that backfills legacy provenance, permission spec §7) — and, critically, **withdrawal never depends on @@ -221,10 +234,16 @@ Requirements: into the silent-stateless state. 9. Every misconfiguration in the providers spec §6 table fails at **startup**. Request-time failure for a configuration error is a defect. -10. Config-store payload validation (`ts config push`) applies the same - rules — including `[permissions]` policy validation (permission spec - §3.3) — so a bad config is rejected at push time, before any instance - restarts into it. +10. Validation is split into two named layers, because "the same + validation at push and startup" is not implementable: **structural + validation** (schema, types, `[permissions]` policy — permission + spec §3.3) runs at `ts config push` and again at startup; + **deployment validation** (adapter capabilities, store bindings, + store openability — a structurally valid selection can still be one + an adapter must reject) runs at startup, where those facts exist. + Push may additionally pre-check deployment facts when given a + **machine-readable adapter capability profile** (the providers §7 + matrix, serialized), but startup remains the authority. ## 5. Minimal-divergence migration recipe (operator-facing) @@ -373,19 +392,20 @@ made differently). **Implementation is blocked while any row is `open`**; each row needs an owner, a status, and a link to its decision record — an unratified row reverts to open, not to silently implemented. -| # | Decision | Where | Owner | Status | -| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | --------------------- | ------ | -| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | -| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | -| 3 | Sharing / targeted-advertising opt-outs remove P4 but retain the stored identity | permission §4.5 | maintainers + legal | open | -| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | -| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | -| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | -| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | -| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | -| 9 | Integration persistent cookies inside the permission model (P1-gated, declared registration) | hook §3; permission §7 | maintainers | open | -| 10 | Session cookies exempt from `store-on-device` even when they carry a stable identifier | hook §3 | maintainers + legal | open | -| 11 | A single failed destructive withdrawal may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | -| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | -| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | +| # | Decision | Where | Owner | Status | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | +| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | +| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | +| 3 | Sharing / targeted-advertising opt-outs remove P4 but retain the stored identity | permission §4.5 | maintainers + legal | open | +| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | +| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | +| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | +| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | +| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | +| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | +| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | +| 11 | A single failed destructive withdrawal may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | +| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | +| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | +| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md new file mode 100644 index 000000000..fcfcd400f --- /dev/null +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -0,0 +1,132 @@ +# PR #986 review-finding ledger + +Disposition of every review finding against the provider/permission spec +set, by round. Statuses: **fixed** (commit noted) · **reapplied** (fix was +lost to a failed edit batch and re-landed — the round-4 script loss is +called out where it happened) · **partial → refixed** (a later round showed +the fix incomplete; both commits noted) · **superseded** (descope or a +later design change removed the surface) · **deferred** (moves with a +deferred feature; recorded as its entry bar) · **open** (sign-off table, +migration spec §8). + +Commits: R1 `a35f2ca78` · R2 `9886091e5` · R3 `5c8c2e893` · R4 `2b4d776b6` +· R5 `de70ca931` · R6 `c8b4b849e` · R7 (this commit). + +## Round 1 — adversarial self-review (22 findings) + +All 22 fixed in R1, three later shown partial and refixed: geo/device +gating circularity (refixed R3 — device half was wrong again), identity +stability vectors (refixed R3 — random suffix), §5.3 citations (fixed R1). +Policy moved YAML → TOML in R1 (maintainer decision). No open remnants. + +## Round 2 — first maintainer review (15 blocking + 1 + 4) + +| Finding | Status | +| -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | +| B1 raw-EC egress ungated | fixed R2; egress table concrete R3; typed R5; scoped types R6 | +| B2 recipe grants everywhere | fixed R2; fixture-not-delta R4; per-adapter R5; minimal-divergence R6 | +| B3 blanket gate blocks withdrawal | fixed R2 (split gate, spy test) | +| B4 provider switch strands identities | fixed R2 (legacy readers); rewrite portion superseded R7 (descope) | +| B5 graph-key/prefix incomplete | fixed R2; literal-prefix R3; namespace R3; core-constructed R5→R6; delimiter R7 | +| B6 device gating not circular | fixed R2; reasoning corrected R3; qualifier R5 (reapplied R6 after batch loss) | +| B7 withdrawal trigger contradiction | fixed R2 (requires_signal ∨ denied) | +| B8 withdrawal storage failure | fixed R2; family record R3; idempotent families R4; unbounded residual honesty R7 (sign-off 11) | +| B9 signal normalization missing | fixed R2 (subjects); outcomes R4; preserved semantics R5; state machine R6 | +| B10 auction class inference lossy | fixed R2 (regime); dispatch matrix R4; raw arm R5; persisted-TCF arm R7 | +| B11 no-geo guard too narrow | fixed R2 (all jurisdiction consumers); cookie consumer deferred R7 | +| B12 validation incomplete | fixed R2; rules.default/dupes R3; region assigned R5 | +| B13 Sec-Fetch-Site insufficient | fixed R2 (exact Origin / CSRF); deferred with client spec R7 | +| B14 replay unmitigated | fixed R2; session binding required R4; ownerless mode removed R7 | +| B15 unbounded inputs | fixed R2; exact limits R5; media-type matching R6 — all deferred with client spec R7 | +| ❓ issue contradictions | fixed R2 (divergence tables per spec) | +| Hook &mut HeaderMap + framing headers | fixed R2 (structured ops, reserved list) | +| 4 non-blocking (geo residual, eligibility matrix, cluster capability, deterministic entropy) | all fixed R2 | + +## Round 3 — second maintainer review (15 blocking + hook + gaps) + +| Finding | Status | +| --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| Permission algebra can't preserve US | fixed R3 (grant class); regime-scoped R4; field-scoped R5→R6 | +| Auction dispatch undefined | fixed R3 (regime matrix + fallback regime) | +| Normalization delegated | fixed R3→R5 (in-spec outcomes, preserved semantics R6) | +| Egress inventory incomplete/wrong | fixed R3 (path table, 11a/11b split); pull/batch split R5 | +| Batch sync no authority source | fixed R3 (stored provenance); full recompute R5; live-refusal rule R7 | +| Hook ordering vs cache protection | fixed R3 (invariant pass); snapshot monotonic R5; sticky axes R6; must-understand R7 | +| Universal case equivalence | fixed R3 (provider-declared fixtures) | +| Namespacing vs HMAC stability | fixed R3 (reserved grammar); all-versions-verbatim R7 | +| Legacy-reader gaps | fixed R3→R5 (namespaces, governing permissions, provenance, provider=none); rewrite parts superseded R7 | +| Graph prerequisites/runtime failures | fixed R3 (startup requirement, runtime matrix, active-after-commit); adoption path R7 | +| Withdrawal atomicity | fixed R3 (idempotent families); family record R4; suppression R6→R7 | +| No dual-compatible config | fixed R3 (dual-read N+1); ordering corrected R5; both-direction rollback R6→R7 | +| Source-agnostic sources dropped | fixed R3 (explicit deferral, divergence row) | +| Client resolve replacement/replay | fixed R3→R5; deferred with client spec R7 | +| Device contract / region default | fixed R3 (stale wording, US/CA default) | +| Hook API/eligibility | fixed R3 (ops API, eligibility matrix) | +| Completeness gaps (EcId bounds, non-cluster dedupe, mutator limits, per-row tests, runtime behavior, metrics, fail-closed labels) | all fixed R3→R5 | + +## Round 4 — architecture review (1 P0, 12 P1 groups, 9 P2, 3 P3) + +| Finding | Status | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | +| P0 legacy family-ID protocol hole | fixed R4 (deterministic derivation) | +| US GPP/USP not permission-scoped | fixed R4 (§4.5 table); regime scope R5; global aggregation R7 | +| Malformed state machine | fixed R4; six-state machine + matrix column R6 | +| TCF conflict nondeterminism | fixed R4; **wrongly "preserved" — refixed R6** (conjunction algorithm) | +| Raw-TCF arm excludes malformed | fixed R4 (raw-presence trigger); persisted fallback R7 | +| Stale authority renewal | fixed R4 (valid_until, snapshot replace); evidence classes R6; digest scope R6 | +| Degraded mode cross-instance | fixed R4 (state machine); **local-only honesty R7** (sign-off 11) | +| Workers KV "bounded" lag | **partial R4 → refixed R6/R7** (strong primitive required; single normative home) | +| Physical keys/schemas contradictory | fixed R4→R6; alias-at-source-key + versionless keys R7 | +| HMAC version via parse | fixed R4 (**lost in failed batch, reapplied R6**; provenance-resolved) | +| AuthorizedIdentity unscoped | fixed R6 (GraphOps/PartnerEgress); request-side redaction R7 | +| Client contract missing | fixed R5 (Acquisition modes); ClientResolveContext R7; deferred R7 | +| Revocation-wins impossible | fixed R6 (family epoch CAS); cross-key atomicity recorded open, deferred R7 | +| Rewrite storage/transactionality | fixed R5→R6; **superseded R7 — rewrite_legacy cut from epic** | +| Release/rollback inconsistent | fixed R5 (dual-read); reader-first R5; preconditions R6; floor marker + N+1 duties R7 | +| Graphless deployments | fixed R6 (readiness step); adoption path R7 (matrix row 13) | +| No concrete adapter matrix | fixed R5→R6; availability-vs-wiring split + Axum honesty R7 | +| Hook cookies bypass model | fixed R5 (write gate); **shown insufficient → cookie ops deferred R7** | +| Cache lattice invalid | fixed R5; **ordered lattice wrong → sticky axes R6**; full Vary R6 | +| P2/P3 (mint ordering, health machine, client limits, fixtures graph config, thresholds, N/A, KV pipeline, transitions, protective labels, FR wording, device qualifier, illustrative label) | all fixed R4–R6 (device qualifier reapplied R6 after batch loss) | + +## Round 5 — package re-review + +All 15 P1 and 4 P2 dispositioned above where they refined earlier rows; +net-new: sections map (fixed R5; **registry-corrected R7**: +6, +24–27), +live-vs-stored refusal (fixed R7), suppression record (fixed R6; +completeness R7), fixture invalidity (stateless fixtures R7). + +## Round 6 — re-review at de70ca9 + +All 18 P1 and 4 P2 fixed in R6 except where R7 shows partials (tracked in +the R7 table below). Sign-off table with owners/status introduced R6. + +## Round 7 — current + +| Finding | Status | +| ------------------------------------------------- | ---------------------------------------------------------------------------------- | +| Client page leg pre-gate | fixed (page-leg gating; deferred with client spec) | +| Cookie read/use/withdraw unmodeled | **cookie ops deferred out of v1 hook** (entry bar recorded; sign-off 9/10 updated) | +| Ownerless mode reintroduces fixation | fixed — ownerless mode removed outright | +| Graphless cookie adoption | fixed (adoption transaction, matrix row 13) | +| Rewrite retention lineage | superseded — rewrite_legacy cut; finding recorded as entry bar | +| Unreferenced provider blocks | fixed (startup error) | +| Fastly prefix-query delimiter | fixed (backend-safe delimiter, per-adapter query validation) | +| P2 alias/tombstone cluster counting | fixed (liveness/kind filtering; aliases reserved-future) | +| P2 push-vs-deploy validation | fixed (two named layers, capability profile) | +| P2 rollback floor unobservable | fixed (floor = writer activation, durable marker) | +| P2 cookie ownership uniqueness | integration-ID uniqueness kept; cookie ownership deferred with cookie ops | +| Still-open: GPP 6/24–27 | fixed (registry-complete map) | +| Still-open: state-over-national opt-out erasure | fixed (grants-only precedence) | +| Still-open: suppression completeness | fixed (monotonic ordering, re-consent clear, write-failure semantics) | +| Still-open: family-epoch cross-key CAS | recorded as client-spec open question 0; deferred | +| Still-open: eventual rows vs alias guarantees | superseded (rewrite cut) | +| Still-open: 4-hop stranding | superseded (rewrite cut) | +| Still-open: N+1 enforce vs N+2 write boundary | fixed (N+1 writes safety-critical records) | +| Still-open: N+2-only legacy reader on N+1 | fixed (accepted in legacy position) | +| Still-open: no-geo guard cookie consumers | deferred with cookie ops (inventory row updated) | +| Still-open: persisted TCF in raw arm | fixed (TCF-sourced effective record triggers arm) | +| Still-open: request-side raw identity | fixed (identity-redacted integration request views) | +| Still-open: RequestFilterEffects.response_headers | fixed (folded into hook, done-when item) | +| Still-open: must-understand | fixed (sticky set extended) | +| Still-open: Axum persistence overstated | fixed (in-process, non-durable, dev-only cell) | From ff1e113e689ce16dae6ad77ff59e0d07db9b047d Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Sat, 1 Aug 2026 12:58:27 -0700 Subject: [PATCH 10/24] Address eighth review: suppression realizability, N+1 interim semantics, and descope propagation P1 fixes: - Suppression is realizable: writes require linearizable per-key CAS with the record's own version counter ordering transitions (strong reads alone let an older clear overwrite a newer suppress); coverage is every positive-to-unset delta regardless of cause (the refusal/opt-out-only list left malformed-present and applicable absence leaking stale authority to batch sync); timestamp-less GPP/USP sources get sticky opt-out - only an authoritatively timestamped newer grant clears, since opt-out/consent/opt-out(same value) is indistinguishable from replay (sign-off 16); and a failed suppression write is an unbounded residual for a never-returning visitor, sharing sign-off 11 - not 'transient'. - N/A has one meaning everywhere: explicit Not Applicable is grant-class (preserving pinned USP tests and GPP NotApplicable handling), absent grants nothing; the P4-authorizing consequence is sign-off 17. - Adoption authenticates: ServerMint providers gain verify(id, evidence); rowless cookies failing verification are expired, not adopted (including the declared roaming false-negative); adoption gates on the provider's complete required_permissions, needs an atomic create-if-absent capability (Workers KV ineligible), distinguishes read errors from not-found, and bounds adopted-row TTL by a migration cutoff instead of granting a fresh year (sign-off 21). - N+1 has a valid write behavior: it mints v1 rows with today's semantics, and old-shape config runs the pre-epic consent gate unchanged (dual-read = dual-behavior), so neither the active-after-commit contract nor the compiled protective fallback fires mid-convergence; the new contracts activate with N+2/new-shape config (sign-off 20). - Providers ship compiled-in dormant one release before selectability - there is no dynamic provider ABI, so 'N+2-only provider readable by N+1' was impossible as written; new providers get their own reader-first rollout. - The hook's cookie deferral is contradiction-free: the operation list is headers-only, the reserved-surface and generic-op remnants are swept, and the cache test uses a core-owned queued cookie. - The RequestFilterEffects.response_headers channel is NOT folded in - that would break DataDome's challenge/deny flows (headers + cookies on 200/301/302/401/403/429, response classes the hook never runs on). It stays a distinct core-owned security channel with core-mediated security cookies, adopting the shared validation and cache-invariant layers. - Age, Date, and Expires are reserved: replacing Age:59 with Age:0 or extending Expires re-extends freshness in exactly the way the monotonic merge forbids. - Client-cycle types (Acquisition/ClientResolve/reservations) are out of the normative trait surface per the spec's own minimalism rule; the epic's only acquisition mode is server mint. - Request-side redaction is a specified boundary: typed RedactedRequestView with an enumerated strip set, same-PR migration of the raw filter/proxy inputs, and denied/withdrawn tests. P2/P3: rewrite residue swept (tests, runtime row, metrics, retirement gate; alias schema marked reserved); physical keys become one portable delimiter-free fixed-width grammar (class tag + 4-char registry provider code - Fastly rejects both / and : in prefix queries, and per-adapter delimiters would fork physical keys across adapters); the Axum matrix cell reflects UnavailableKvStore; stored cluster sizes cannot outlive their inputs; GPP applicability leftovers reconciled (MD/IN/KY/RI sentence removed, section-6 grants defined, regime-none row aligned); mixed-revision divergence explicitly accepted (sign-off 19); the schema floor lives in write-once/CAS deployment metadata that config rollback cannot erase; sign-off rows 16-21 added and rows 3/11 amended; duplicate integration IDs rejected at startup; GPP versions enumerated with unknown-version-as-malformed; custom geo region vocabularies require a canonical ISO mapping; and the review ledger's overstated R7 dispositions are corrected (suppression, delimiter, redaction, Axum, header-channel) with a full R8 section. --- ...integration-response-header-hook-design.md | 50 ++++-- .../2026-07-30-permission-model-design.md | 117 ++++++++----- .../2026-07-30-pluggable-providers-design.md | 156 +++++++++++------- ...07-30-provider-migration-rollout-design.md | 110 +++++++----- docs/superpowers/specs/pr986-review-ledger.md | 84 +++++++--- 5 files changed, 331 insertions(+), 186 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index a8a7ef7cb..b16350ac2 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -31,8 +31,9 @@ mutators to the outbound response for HTML document responses it processed. registered mutators in registration order. - **The mutator API is structured operations, not header-map access.** A mutator returns (or is handed a recorder for) typed operations — - `append(name, value)`, `replace(name, value)`, - `append_set_cookie(cookie)` — which **core validates and applies**, + `append(name, value)` and `replace(name, value)` (v1 is headers-only; + the cookie operation arrives with the deferred cookie surface, §3) — + which **core validates and applies**, attributing each to its integration id. PR #838's shape handed the integration an unrestricted `&mut HeaderMap`, which makes §3's collision policy unenforceable by construction: core cannot validate or attribute @@ -88,9 +89,14 @@ mutators to the outbound response for HTML document responses it processed. granularities because `Set-Cookie` is multi-valued: (a) reserved header _names_ — HTTP framing and hop-by-hop headers (`Content-Length`, `Transfer-Encoding`, `Connection`, `Trailer`, `Upgrade`, `TE`, - `Keep-Alive`), **representation headers coupled to body bytes the hook - cannot see** (`Content-Encoding`, `Content-Range`, `Content-Type`, - `ETag`, `Last-Modified`, `Accept-Ranges`, and digest headers — + `Keep-Alive`), **freshness metadata** (`Age`, `Date`, `Expires` — replacing `Age: 59` + with `Age: 0` on a cached `max-age=60` response, or pushing `Expires` + into the future, extends downstream freshness in exactly the way the + monotonic merge forbids for `max-age`, so these are reserved outright + rather than merged), **representation headers coupled to body bytes + the hook cannot see** (`Content-Encoding`, `Content-Range`, + `Content-Type`, `ETag`, `Last-Modified`, `Accept-Ranges`, and digest + headers — relabeling uncompressed bytes as Brotli, or advertising a validator or digest for bytes the hook never saw, corrupts responses or poisons caches), the `x-ts-*` namespace, and the @@ -108,9 +114,14 @@ mutators to the outbound response for HTML document responses it processed. permissions, a typed authorized request-side view, stripping from unauthorized integration/proxy inputs, mandatory expiry on destructive P1 withdrawal, and startup-unique (name, domain, path) ownership. - Integration IDs are startup-unique regardless. Until then the + Integration IDs are **startup-unique, enforced**: registry + construction rejects a duplicate ID (current code silently coalesces, + which corrupts attribution and budgets), with a duplicate-ID test in + the done-when. Until then the operation set is headers-only, and `Set-Cookie` is fully reserved. - reserved cookie name. Violations are rejected at the operation layer (§2) and + reserved cookie name — in v1 that is every cookie name, since + `Set-Cookie` is fully reserved (§3 deferral). Violations are rejected + at the operation layer (§2) and logged at `warn` with the integration id. The reserved lists are single constants next to the definitions they protect, not duplicated in the hook. @@ -125,7 +136,8 @@ mutators to the outbound response for HTML document responses it processed. which is deterministic). - **Operation-layer hygiene:** generic `append`/`replace` reject the `Set-Cookie` header name outright — cookies go only through - `append_set_cookie`, so its validation cannot be bypassed by spelling + the deferred cookie builder (when it exists), so cookie validation + cannot be bypassed by spelling the header name in a generic op. Per-integration limits bound total operations (≤ 32), added headers (≤ 16), and added bytes (≤ 8 KiB), and a **cumulative final-response budget** (≤ 128 headers / ≤ 32 KiB total, @@ -174,11 +186,20 @@ processed documents (§6). ## 4. Done-when (from #782, sharpened) 1. Trait + builder + registry application, each public item documented. -2. **The pre-existing `RequestFilterEffects.response_headers` channel is - folded into the hook in the same PR** — its outputs become hook - operations subject to the same validation, reserved surface, budgets, - and invariant pass, or the channel is removed; a second, unvalidated - header path bypassing the hook defeats every rule above. +2. **The pre-existing `RequestFilterEffects.response_headers` channel + remains a distinct, core-owned security channel — not folded in, and + not left unvalidated.** Folding it into this hook would break its one + real consumer: DataDome sets headers **and cookies** on 200, 301/302, + 401, 403, and 429 responses — challenge and deny flows on exactly the + response classes (§3a) this hook never runs on, and with cookie + emission v1 reserves. Instead, the channel keeps its own eligibility + (security-integration responses of any status), its cookies are + **core-mediated security cookies** (explicitly outside the deferred + integration-cookie surface, migrated deliberately when that surface + lands), and it adopts the **shared validation layers**: the + structured-operation checks, reserved header names, budgets, and the + final cache/privacy invariant pass. One invariant enforcer, two + eligibility domains. 3. **At least one real consumer ships in the same PR** — an existing integration registering a mutator for a real need (or, failing a real need, the feature waits; scaffolding with only self-referential tests is @@ -192,7 +213,8 @@ processed documents (§6). cache-hit, pass-through, redirect, error, and 304 each proven to run or not run the hook — not merely one positive header test per adapter. 8. Cache/privacy invariant tests, one per restriction source and shape: - cookie appended + public `Cache-Control` replacement → private/no-store, + a **core-owned** cookie already queued before the hook + an + integration's public `Cache-Control` replacement → private/no-store, surrogate stripped; **core-private cookieless** processed HTML + public replacement → restriction preserved; **origin-private cookieless** processed HTML that retained the origin's cache restrictions + public replacement → restriction diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 4a3feb629..21889eb54 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -187,7 +187,7 @@ Validation rejects: - rule keys whose country part is not in the embedded **assigned** ISO 3166-1 alpha-2 list (not merely `[A-Z]{2}` — an unassigned code is almost certainly a typo silently diverting a country to the fallback); - the region part must be an assigned ISO 3166-2 subdivision of that country (not merely a shape check — `US/ZZ` would parse but can never match a request), unless the selected geo provider declares its own region vocabulary, in which case validation uses that declaration. The `US/CA` slash form is the + the region part must be an assigned ISO 3166-2 subdivision of that country (not merely a shape check — `US/ZZ` would parse but can never match a request), unless the selected geo provider declares its own region vocabulary **together with a canonical mapping to ISO subdivisions** — §4.5 applicability and policy rule keys operate on canonical `US/CA`-form keys, so a provider emitting anything else must declare the translation, validated at startup. The `US/CA` slash form is the house rule-key format corresponding to ISO 3166-2 `US-CA`; - references to permissions outside the enforced vocabulary (§2); - references to undefined groups, and groups missing the `regime` class; @@ -273,11 +273,11 @@ blocked but an **explicit non-opt-out** value grants: permission-scoped** — grant signals are NOT interchangeable across regimes: - | Regime of the resolved rule | Evidence accepted as a grant for a `requires_signal` permission | - | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | - | `gdpr` | **Only** a TCF record consenting to that specific purpose | - | `us-privacy` | TCF consent for the purpose, or GPP/USP evidence **per the §4.5 field mapping** — a field grants only the permissions it maps to | - | `none` | Any grant-class signal | + | Regime of the resolved rule | Evidence accepted as a grant for a `requires_signal` permission | + | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | `gdpr` | **Only** a TCF record consenting to that specific purpose | + | `us-privacy` | TCF consent for the purpose, or GPP/USP evidence **per the §4.5 field mapping** — a field grants only the permissions it maps to | + | `none` | TCF consent; GPP/USP grants only where §4.5 applicability yields one (the national section applies under `us-privacy` regimes, so in practice `none` grants via TCF — moot in the shipped policy, whose `none` baseline is `granted`) | Without this scoping, a US-style `sale_opt_out = false` would satisfy a French `requires_signal` rule — no TCF, both purposes granted, EC minted, @@ -423,23 +423,48 @@ and the fail-closed marker: which the refusal just unset, and identity rows may be eventually consistent, so a stale replica could resurrect a P4 grant after a targeted-advertising opt-out. The fix is a **suppression record** in - the strongly consistent class (providers spec §6.3: `sup/`). - Its semantics are complete, not sketched: entries are **per permission** - with the triggering state (refusal or non-destructive opt-out — the - full negative-state coverage; absence writes nothing) and an - authoritative timestamp; ordering is **monotonic per permission** — - a write with an older timestamp than the stored entry is a no-op, so - replays cannot regress the state; **re-consent clears**: a live - resolution carrying an accepted grant with a newer authoritative - timestamp than the suppression entry supersedes it (recorded as a - clear entry in the same record — still an exempt write, since it only - ever reflects the live resolution); and **write failure fails closed - for the live request** (the refusal's effect stands for this response) - while S2S may transiently honor prior authority until the retry lands — - logged, metered, and covered by the degraded-mode rule. Every S2S - recompute and partner-egress check consults the record: a suppressed - permission is unset whatever the row's provenance says, so no - eventual-consistency edge can restore it. + the strongly consistent class, and — because monotonicity is a + read-modify-write property, not a read property — suppression writes + require **linearizable per-key CAS** (providers spec §6.3 + `sup/`, §7 matrix): with plain read-after-write, two + writers can both read the record and an older clear can overwrite a + newer suppress. The record carries its own **version counter, + incremented through the CAS**, so transitions are ordered by + serialization, not by comparing wall-clock timestamps. + + Coverage is **every authority-clearing delta, not an enumerated cause + list**: after each live resolution, any permission whose new state is + unset while its stored provenance is positive gets a suppression + entry — refusal, non-destructive opt-out, malformed-present, and + applicable absence alike. (The earlier refusal/opt-out-only list left + a hole: a malformed record unsets P1, the P1-gated row update is + thereby forbidden, no suppression is written, and batch sync later + honors the stale grant.) + + **Timestamp-less sources get sticky opt-out.** GPP/USP values carry no + intrinsic timestamp, and opt-out → consent → opt-out(same value) is + information-theoretically indistinguishable from a replay of the first + opt-out. Latest-observation semantics would let a replayed old consent + string clear a newer opt-out, so: a suppression from a timestamp-less + source is cleared **only** by a grant carrying an authoritative + timestamp newer than the suppression's observation (TCF + `LastUpdated`) — a timestamp-less re-consent alone does not clear it. + The consequence (a genuine GPP-only re-consent does not restore + authority until a timestamped source or policy provides it) is + declared and is sign-off item 16. Fixtures: suppress-vs-clear race + under concurrent writers; repeated-value opt-out/consent/opt-out. + + **Write failure fails closed for the live request** (the refusal's + effect stands for this response), but the S2S residual is **unbounded + for a never-returning visitor** — exactly like a failed destructive + revocation, not "transient": other instances continue honoring old + provenance, the breaker is per-instance, and no durable retry exists. + This shares sign-off item 11 (extended to cover suppression) and gets + its own fault test. Every S2S recompute and partner-egress check + consults the record: a suppressed permission is unset whatever the + row's provenance says, so no eventual-consistency edge can restore + it. + - **The cookie expires only after the family record commits.** - **If the family-record write itself fails, nothing durable exists** — the cookie stays and the durable client-side signal (GPC, CMP-stored @@ -498,14 +523,19 @@ an expired record before clearing both sources; expiry-first is a | Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | | Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**, then flows through the full normalization pipeline (syntax, expiry, conflict) like any live record; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | **Changed (declared)**: current code returns immediately after the KV load, bypassing expiry and conflict normalization | | Proxy/mirror mode | **Minimal opt-out extraction still runs; full semantic decoding is skipped.** Because opt-outs are globally authoritative (§4), proxy mode must not suppress them: the §4.5-mapped opt-out fields (GPP US sections) and the US Privacy string are decoded — nothing else — alongside syntax validation, so a valid SaleOptOut or USP opt-out revokes and withdraws exactly as outside proxy mode. No grants are ever derived from records in proxy mode; a present record otherwise blocks grants (fail-closed); absent → baseline. GPC needs no decoding | **Changed (declared)**: today proxy mode skips decoding entirely — fail-open under permissive baselines and, worse, opt-out-blind | -| GPP / US Privacy fields | Per the normative field mapping of §4.5 — fields are not interchangeable signals, and absent/N-A fields grant nothing | Decided here (§4.5) | +| GPP / US Privacy fields | Per the normative field mapping of §4.5 — fields are not interchangeable signals; **explicit N/A is grant-class (not-opted-out), absent grants nothing** — one meaning, everywhere | Decided here (§4.5) | | Malformed-but-present record, no valid record of that family | **Blocks grants** (fail-closed acquisition — it does not degrade to "absent", which under a `granted` baseline would turn garbage into a grant, the fail-open path in both #838 and the first draft of this spec). Never triggers withdrawal — destruction requires an affirmative, decodable signal (§4.2) | Changed (declared) | ### 4.5 US signal field mapping — normative GPP and US Privacy fields map to specific permissions with specific effects; they are never interchangeable, a field's absence or N/A value -contributes nothing, and only the fields marked destructive trigger +behaves per its table row — **explicit _Not Applicable_ is grant-class +(not-opted-out), preserving current USP tests and GPP `NotApplicable` +handling; only a genuinely absent field contributes nothing** (this is +the single normative statement; an earlier "N/A contributes nothing" +rule is dead, and the P4-authorizing consequence is sign-off item 17) — +and only the fields marked destructive trigger withdrawal. Section IDs and versions are those of the IAB GPP specification current at implementation time; adding a section or field is a change to this table. @@ -523,19 +553,17 @@ a change to this table. | Any field | explicitly _Not Applicable_ | as the field's not-opted-out row above | as the field's not-opted-out row above | — | | Any field | absent | — | — | — | -**N/A vs absent:** a field explicitly set to _Not Applicable_ is treated -as not-opted-out (grant-class) — pinned by today's USP tests and matching -current GPP `NotApplicable` handling, and declared as such since an -earlier draft said N/A contributes nothing. A field **absent** from an -applicable section, or any field of a non-applicable section, contributes -nothing. +**N/A vs absent (restating the single rule):** explicit _Not +Applicable_ = grant-class; absent = nothing; a non-applicable section's +fields grant nothing (their opt-outs still count, per step 2). **Applicability and aggregation — ordered algorithm:** 1. **Section map (normative, pinned here — not "whatever GPP is current"), matching the official IAB registry in full:** section 6 ↔ - the **US Privacy string carried as a GPP section** (it maps to the USP - rows of the field table, not to nothing); `US` national ↔ 7 (usnat); + the **US Privacy string carried as a GPP section** — it maps to the USP + rows of the field table in full, opt-outs _and_ grant-class values, + under the same applicability rules as the national section; `US` national ↔ 7 (usnat); the state sections — `US/CA` ↔ 8, `US/VA` ↔ 9, `US/CO` ↔ 10, `US/UT` ↔ 11, `US/CT` ↔ 12, `US/FL` ↔ 13, `US/MT` ↔ 14, `US/OR` ↔ 15, `US/TX` ↔ 16, `US/DE` ↔ 17, `US/IA` ↔ 18, `US/NE` ↔ 19, `US/NH` ↔ 20, @@ -544,8 +572,11 @@ nothing. claimed MD/IN/KY/RI had no section). A truncated map silently loses opt-outs — a Texas (16) or Maryland (24) sale opt-out must not vanish. The implementation PR cross-checks this list against both the current - decoder's section set and the official registry; adding a section or - version is a change to this map. + decoder's section set and the official registry, and **enumerates the + accepted version per section**; a mapped section carrying an unknown + version is treated as malformed-present (blocks grants, never + withdraws — §4.4), not as absent. Adding a section or version is a + change to this map. 2. **Applicability gates grants only — never opt-outs.** A mapped **opt-out** field (either subclass) is honored from **any** section on **any** request, whatever the regime — this is §4's global-opt-out @@ -556,8 +587,7 @@ nothing. to the resolved `US/`; foreign-state sections and all sections on non-`us-privacy` requests grant nothing. Regionless US traffic: national section only. A configured privacy state with no - state-specific section (e.g. MD, IN, KY, RI today) uses the national - section alone. + state-specific section uses the national section alone. 3. **State-over-national, per field — for grants only:** where an applicable state section carries a field, its value governs that field's **grant** derivation; the national section fills only fields @@ -647,9 +677,16 @@ policy** (§4.2 trigger 3) — the one revision-sensitive destructive case (trigger 2 under a now-`denied` baseline) requires an affirmative user refusal at the evaluating instance, which is safe under either revision. S2S recomputation always evaluates against the instance's current -revision and records it. Rolling a policy revision back restores -acquisition rules but **cannot resurrect tombstoned identities**; the -migration guide says so where operators will read it. +revision and records it. One divergence is explicitly accepted rather +than fenced: during convergence, a live refusal under a +`granted`-revision instance suppresses while the same refusal under a +tightened-revision instance destroys (trigger 2) — the destructive +outcome is the target revision's intended behavior arriving early on +part of the fleet, coordinated activation fencing is not worth its +machinery, and the acceptance is sign-off item 19. Rolling a policy +revision back restores acquisition rules but **cannot resurrect +tombstoned identities**; the migration guide says so where operators +will read it. ## 6. Failure-mode matrix — normative diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index c92253a3d..f231601e9 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -150,7 +150,12 @@ Three global rules sit above every provider: trips the trust threshold toward denial, never toward extra writes); the listing filters by the value's `kind`/liveness within the existing list limit where the backend returns values, and the residual - over-count where it cannot is declared. Aliases are reserved-future + over-count where it cannot is declared. A computed cluster size is + **not persisted beyond its inputs' lifetime**: today's code stores the + calculated `cluster_size` in the row and reuses it for the row's full + TTL, which would freeze a tombstone-inflated count for up to a year — + stored values carry a short validity (within the tombstone-TTL + horizon) or are recomputed on use. Aliases are reserved-future (§6.1) and excluded by `kind` when they exist. - **No-cluster behavior is still defined.** A provider without cluster support deduplicates pull-sync by canonical graph key and redacts logs @@ -201,26 +206,20 @@ pub trait EdgeCookieProvider { /// key, shared across identifiers minted from the same client /// evidence. None when the provider lacks IP-cluster semantics (§3). fn cluster_prefix(&self, id: &EcId) -> Option; - /// Acquisition mode — exactly one: - fn acquisition(&self) -> Acquisition<'_>; + /// Cryptographic verification of a parsed identifier against request + /// evidence — recognition (`parse`) is not authentication; adoption + /// (§5) and any rowless acceptance require this. + fn verify(&self, id: &EcId, input: &IdentityInput<'_>) -> bool; } - -/// How a provider's identifiers come into being. Server-mint providers -/// generate from request evidence; client-resolve providers verify a -/// browser-posted payload (client-cycle spec) and declare the JS module -/// their page leg needs. One provider implements exactly one mode. -pub enum Acquisition<'a> { - ServerMint(&'a dyn ServerMint), // fn generate(&IdentityInput) -> EcId - ClientResolve(&'a dyn ClientResolve), -} -// ClientResolve::resolve_from_client(&ClientResolveContext) -> Result -// ctx is core-built: canonical publisher audience, verified session -// owner hash, clock, and the bounded payload — a bare payload could -// not verify audience binding, session binding, or expiry. -// VerifiedIdentity carries the identifier, reservation id, and expiry. -// ClientResolve::js_module_id() -> &str ``` +The acquisition-mode enum (`ServerMint` / `ClientResolve`), the +`ClientResolveContext` contract, replay-reservation schemas, and the +reservation capability rows are **not part of this normative surface** — +they live in the deferred client-cycle document and return with that +feature, per this spec's own minimalism rule (§4): the epic's only +acquisition mode is server mint, expressed directly as `generate`. + (Names indicative; the shape is normative. `required_permissions` joins the trait at step 5 of §11, together with its enforcement point.) @@ -251,19 +250,40 @@ header emission; the identity exists durably from that moment. A graph-commit failure means the mint never happened: no cookie, no egress, error logged, the next request retries. -**Pre-existing cookies without rows are adopted, not orphaned.** Current -graphless deployments have minted cookies with no row; under -no-active-until-commit those identities could never be used again and — -without care — never withdrawn. The contract: a recognized legacy cookie -with no reachable row triggers a **permission-gated, race-safe adoption** -on a live request — gated exactly like minting (`store-on-device`), -implemented as create-if-absent on the verbatim key (concurrent adopters -converge: same key, same deterministic family ID), provenance from the -live resolution. Until adoption succeeds the cookie **never egresses**; -**withdrawal works without adoption** — the deterministic family ID -(permission model spec §4.3) needs no row, so a first post-upgrade -request that is an opt-out revokes and expires the cookie with zero -migrated state. Migration matrix row 13 declares this path. +**Pre-existing cookies without rows are verified, then adopted — parse +is recognition, not authentication.** A syntactically valid +`{64hex}.{6alnum}` string is constructible by anyone; adopting it on +shape alone would let an attacker mint durable rows. The contract: + +- ServerMint providers implement `verify(id, &IdentityInput) -> bool` — + cryptographic verification against **request evidence** (for `hmac`: + recompute over the request's evidence with each configured version's + passphrase and compare to the 64-hex prefix). A recognized rowless + cookie that fails verification is **expired, not adopted** — including + the honest false-negative: a legitimate cookie presented from a + changed network no longer verifies and is expired; the affected + population is graphless deployments only, declared in migration row 13. +- Adoption is gated on the provider's **complete + `required_permissions()`** — exactly like minting, not a hard-coded + `store-on-device`. +- The row write is **atomic create-if-absent** — a distinct capability + row in the §7 matrix (strong class; Workers KV's concurrent same-key + writes can overwrite each other, so it is ineligible, consistent with + its revocation ineligibility). +- **Read errors are not "not found"**: adoption proceeds only on an + authoritative not-found; a failed graph read means no adoption this + request, fail closed. +- Adopted rows do **not** get a fresh full TTL — a nearly expired legacy + identity must not gain a year (the exact rejuvenation problem that + deferred rewrite). Expiry is `min(adopted_at + standard TTL, +migration_cutoff + grace)` with the cutoff configured; sign-off + item 21. + +Until adoption succeeds the cookie **never egresses**; **withdrawal +works without adoption** — the deterministic family ID (permission model +spec §4.3) needs no row, so a first post-upgrade opt-out revokes and +expires the cookie with zero migrated state. Migration matrix row 13 +declares this path. **Egress is typed, not policed.** The inventory-and-denylist test (permission model spec §7) is a backstop, but conventions do not survive @@ -279,12 +299,20 @@ and nothing weaker — an unparameterized wrapper would let a P1-only identity flow into an ORTB request. A future bypass then requires deliberately reconstructing the raw string — visible in review — rather than passing along what was already in hand. The same boundary -applies **request-side**: integration-facing request views (proxy -interfaces, filter inputs, forwarded header/cookie maps) receive -**identity-redacted** views — the EC cookie and identity headers are -stripped unless the path holds `AuthorizedIdentity` — -because the ungated forwarding paths of PR #838 were exactly integrations -reading the raw request. +applies **request-side, as a concrete API transition, not an +assertion**: the current filter/proxy inputs expose the raw request +(cookies included), so a filter can read `ts-ec`, copy it into +`X-Vendor-Identity`, and return it through response effects — response +snapshot redaction cannot undo that. The contract: integration-facing +request access moves to a typed **`RedactedRequestView`** whose stripped +set is enumerated — the `ts-ec` cookie and every `ts-*` cookie, `x-ts-*` +identity/consent headers, and the EIDs header — with identity reachable +only through a scoped `AuthorizedIdentity` parameter; the raw-request +filter/proxy interfaces are migrated in the **same PR** as the typed +egress boundary (they are the same boundary), and the tests are +enumerated: a denied/withdrawn request through a filter, a proxy, and a +forwarding path, each asserting no identity value is readable or +emittable. The gate applies to EC providers **only**. Geo and device are ungated for two _different_ reasons, stated separately because only one of them is @@ -406,12 +434,11 @@ The contract: reserved-for-future — nothing in the epic writes one. - Retiring a legacy reader is the explicit end of those identities: the migration guide documents the cleanup procedure (migration spec §6). -- Tests: switch active provider → request with old cookie → identity still - resolves and a withdrawal tombstones it (both linked rows when - rewritten); old cookie with no matching legacy reader → treated as - absent and **never egresses**; interrupted rewrite → old cookie still - live, retry completes; `provider = "none"` + legacy reader → no mints, - withdrawal still works. +- Tests: switch active provider → request with old cookie → identity + still resolves and a withdrawal tombstones it; old cookie with no + matching legacy reader → treated as absent and **never egresses**; + `provider = "none"` + legacy reader → no mints, withdrawal still + works. (Rewrite-specific tests left with the rewrite deferral.) Cluster degradation config (referenced from §3): when the active writer lacks the cluster capability, `[ec] cluster_fallback = "allow" | "deny"` @@ -431,7 +458,6 @@ when a healthy configuration meets an unhealthy runtime. Every row logs at | Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | | Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | | Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | -| Legacy rewrite fails mid-flight | Old cookie remains live; rewrite retries (§6.1) | | Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | | Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | @@ -476,19 +502,23 @@ the reserved hmac grammar), and every record value carries a `kind` discriminator alongside its schema version — so a reader always knows what it fetched, including where two classes deliberately share an address (row vs. alias). The `/` shown in key sketches is **notation, -not the wire byte**: the physical segment delimiter is a -**backend-safe character validated per adapter** — Fastly permits `/` in -keys but not in prefix _queries_, so a slash-delimited `id//…` -key could never be cluster-listed there despite the matrix marking -prefix listing supported. The reference delimiter is `:`; each adapter's -capability declaration includes which delimiter its prefix queries -accept, and cluster-capability eligibility for a provider requires its -physical prefix to be queryable on that backend — checked at startup, -not discovered at the first cluster count. (hmac verbatim keys contain -no delimiter before the 64-hex prefix and are unaffected.) +not the wire byte** — and the wire form is **one portable grammar, not +per-adapter delimiters** (per-adapter delimiters would give the same +logical identity different physical keys on different adapters, breaking +migration, shared storage, and parity; and Fastly's prefix queries +reject both `/` and `:`, so no delimiter character is safely portable). +Physical keys are **delimiter-free with fixed-width segments**: a +1-character class tag (`i` row, `f` family, `s` suppression, `x` +transaction), a **4-character registry-assigned provider code** +(zero-padded, `[a-z0-9]`), then the suffix — segment boundaries are +positional, so no segment can contain or escape a delimiter, prefix +queries are plain string prefixes on every backend, and cluster +eligibility needs no per-adapter delimiter negotiation. (hmac verbatim +keys remain the reserved exception, with the 64-hex cluster prefix at +position zero.) **Wire schemas** (JSON, like identity rows; every class carries a schema -version): the **alias record** holds target key, created-at, retirement +version): the **alias record** (reserved-future, with rewrite) holds target key, created-at, retirement deadline, and fencing epoch; the **family revocation record** holds the family ID, revoked-at, triggering signal class (§4.5 destructive column), and a **family epoch** bumped on every revocation-state change (the @@ -573,14 +603,14 @@ Requirements: them is how a "yes" cell hides an unusable feature. Feature eligibility requires wired, not merely available: - | Capability | Fastly | Axum (dev) | Cloudflare | Spin | - | ----------------------------------------------------- | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | - | Graph persistence (eventual OK) | KV Store: available + wired | Local store: available + wired (dev-grade) | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | - | Prefix listing (cluster) | Yes (used today) | Yes | Yes (eventual) | _verify_ | - | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | Yes (in-process) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | - | Linearizable fenced CAS (reservations, alias/rewrite) | **Not currently available** — the client-cycle feature (deferred) would need it | Yes — in-process only: linearizable but **non-durable**, dev-eligibility only, not a production persistence claim | Durable Objects: possible, not wired | **No** | - | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | - | Device host evidence (JA4/H2) | Yes | No | No | No | + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | ----------------------------------------------------- | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | + | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | + | Prefix listing (cluster) | Yes (used today) | Yes | Yes (eventual) | _verify_ | + | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | Yes (in-process) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | + | Linearizable fenced CAS (reservations, alias/rewrite) | **Not currently available** — the client-cycle feature (deferred) would need it | Yes — in-process only: linearizable but **non-durable**, dev-eligibility only, not a production persistence claim | Durable Objects: possible, not wired | **No** | + | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | + | Device host evidence (JA4/H2) | Yes | No | No | No | - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 4be41a00a..6299d5448 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -129,34 +129,52 @@ Requirements: the rewrite deferral, providers spec §6.1). N+1 must also **write** the safety-critical record kinds — family revocation and suppression — not only read them: a withdrawal arriving on a - rolled-back N+1 fleet must still revoke; only provenance _writing_ - is deferred to N+2, which is what makes the boundary safe in both - directions. N+1 further **accepts an N+2-only provider in the - `legacy_providers` position** (parse/withdraw need no provenance - encoding) while rejecting it as active writer — otherwise the - rollback rule "retain the new provider as a legacy reader" would be - unsatisfiable on the very release it targets. Rollback tests - therefore run the family-revocation, suppression, and provenance - paths — read **and write** — on N+1 against N+2-written data. **Rollback is binaries-first too, in the other direction** — + rolled-back N+1 fleet must still revoke. + + **N+1's identity-write behavior is v1, explicitly** — this resolves + what was an impossible trilemma (write rows without provenance, + violating active-after-commit; write provenance, violating the + N+2-only writer boundary; or stop minting, an undeclared outage): + N+1 **keeps minting v1 rows with today's semantics**, and the new + active-after-commit/provenance contract activates **with the N+2 + writer**, not before. Likewise the permission model itself: + **old-shape config on N+1 runs the pre-epic consent gate + unchanged** — dual-read means dual-behavior — so the compiled + protective fallback cannot flip behavior mid-convergence before the + operator pushes the new-shape policy; the new model engages only + with new-shape config. The interim (N+1 minting v1 rows, S2S + running today's checks) is declared as sign-off item 20, not + discovered. + + Rollback tests therefore run the family-revocation and suppression + paths — read **and write** — plus v1-minting behavior, on N+1 + against N+2-written data. **Rollback is binaries-first too, in the other direction** — N+2 → N+1 binaries roll back keeping the new config (N+1 reads it fully; reverting config first would hand the old shape to N+2 - binaries that reject it) — **with one precondition**: if an - N+2-only provider or version has been adopted, the fleet must first - converge on an N+1-compatible new-shape config (deselecting what - N+1 rejects, retaining the new provider's secrets as a **legacy - reader** so its minted identities keep resolving and stay - withdrawable until they expire — never "revert to the previous - config", which would strand them); only then do binaries roll back. - N+1 additionally **rejects provider or version selections whose - provenance it cannot yet encode** — new-provider adoption waits for - N+2, so no row is minted that N+2 would misclassify. Every new + binaries that reject it) — **with one structural rule that makes it possible at all**: + providers are compiled into the composition root — there is no + dynamic provider ABI — so an N+1 binary can only read what it + shipped with. Therefore **every provider selectable in release R + must ship compiled-in (dormant: registered, parseable, + configurable, not selectable as writer) in R−1**; adopting a + genuinely new provider gets its own reader-first rollout, exactly + like the epic itself. With that rule, rolling N+2 → N+1 keeps the + new provider's identities resolvable and withdrawable through the + dormant registration ("retain as legacy reader" is now satisfiable + because N+1 physically contains the code); the fleet still first + converges on a config selecting only what N+1 accepts as writer. + N+1 additionally **rejects writer selections whose provenance it + cannot yet encode** — new-writer adoption waits for N+2, so no row + is minted that N+2 would misclassify. Every new config section introduced by the epic follows this same compatibility rule, not only `[ec]`. + - **Release N+2:** rejects `[ec] passphrase` at startup with a message naming the new location — not a generic unknown-field error (implementation note: producing the actionable message means keeping a deprecated `passphrase` field whose presence triggers the custom error). + 2. **Revocation-eligible storage is a per-adapter gate, and ungated adapters migrate stateless.** Identity features require the adapter's strong-consistency rows in the capability matrix (providers spec §7) @@ -194,8 +212,11 @@ Requirements: genuinely pre-N+1 worker cannot preserve at all, which is exactly why the floor exists); after the **fleet-convergence gate**, **N+2 activates the writer** and begins emitting the new fields. **The rollback floor is crossed at N+2 writer activation itself** — an - observable deploy event, recorded as a durable schema-floor marker in - the config store before writes enable — not at "any new-format row + observable deploy event, recorded as a durable schema-floor marker + **in write-once/CAS deployment metadata that ordinary config rollback + cannot touch** — floor-in-rollbackable-config would let "restore the + previous config version" erase the floor after new-format rows exist, + which is exactly the state it guards — not at "any new-format row exists", which no operator can disprove. Below-floor rollback is prohibited from that marker on; a pre-floor binary would silently strip the new fields from every row it touches. @@ -323,14 +344,13 @@ global honoring of opt-out signals is unconditional. silent grant to everyone), not an error rate. The full metric set, each with a stated healthy range: geo lookup-failure/fallback rate (permission spec §5.2), raw-egress denials by path, tombstone family retries, - legacy-reader hit rate, rewrite failures, and cluster-fallback - engagements. Two of these carry thresholds, not just ranges: + legacy-reader hit rate, and cluster-fallback engagements. Two of these carry thresholds, not just ranges: legacy-reader hits at zero for a **quiet period no shorter than the maximum cookie/row lifetime plus rollout skew** — or provable rewrite/backfill completion — is the **retirement-readiness** bar for a legacy provider ("trending to ~zero" is not evidence; a yearly visitor - is not churn), and a nonzero rewrite-failure rate blocks retirement - outright. The telemetry set also includes: graph read/commit failures, + is not churn), (rewrite-based backfill and its metrics left with the rewrite + deferral). The telemetry set also includes: graph read/commit failures, stored-provenance denials, schema-migration failures, and replay-reservation recoveries. **Each rollout-gate metric ships with a threshold, an evaluation window, and a named action** (pause rollout / @@ -392,20 +412,26 @@ made differently). **Implementation is blocked while any row is `open`**; each row needs an owner, a status, and a link to its decision record — an unratified row reverts to open, not to silently implemented. -| # | Decision | Where | Owner | Status | -| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | -| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | -| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | -| 3 | Sharing / targeted-advertising opt-outs remove P4 but retain the stored identity | permission §4.5 | maintainers + legal | open | -| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | -| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | -| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | -| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | -| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | -| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | -| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | -| 11 | A single failed destructive withdrawal may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | -| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | -| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | -| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | +| # | Decision | Where | Owner | Status | +| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | +| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | +| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | +| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | maintainers + legal | open | +| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | +| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | +| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | +| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | +| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | +| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | +| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | +| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | +| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | +| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | +| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | +| 16 | Sticky opt-out for timestamp-less sources: a GPP/USP-only re-consent does not restore authority without a timestamped grant | permission §4.3 | maintainers + legal | open | +| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | maintainers + legal | open | +| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | maintainers + legal | open | +| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | maintainers + product | open | +| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | maintainers + product | open | +| 21 | Adopted legacy rows are bounded by a migration cutoff, not a fresh full TTL | providers §5 | maintainers + product | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index fcfcd400f..fa5a68756 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -103,30 +103,60 @@ the R7 table below). Sign-off table with owners/status introduced R6. ## Round 7 — current -| Finding | Status | -| ------------------------------------------------- | ---------------------------------------------------------------------------------- | -| Client page leg pre-gate | fixed (page-leg gating; deferred with client spec) | -| Cookie read/use/withdraw unmodeled | **cookie ops deferred out of v1 hook** (entry bar recorded; sign-off 9/10 updated) | -| Ownerless mode reintroduces fixation | fixed — ownerless mode removed outright | -| Graphless cookie adoption | fixed (adoption transaction, matrix row 13) | -| Rewrite retention lineage | superseded — rewrite_legacy cut; finding recorded as entry bar | -| Unreferenced provider blocks | fixed (startup error) | -| Fastly prefix-query delimiter | fixed (backend-safe delimiter, per-adapter query validation) | -| P2 alias/tombstone cluster counting | fixed (liveness/kind filtering; aliases reserved-future) | -| P2 push-vs-deploy validation | fixed (two named layers, capability profile) | -| P2 rollback floor unobservable | fixed (floor = writer activation, durable marker) | -| P2 cookie ownership uniqueness | integration-ID uniqueness kept; cookie ownership deferred with cookie ops | -| Still-open: GPP 6/24–27 | fixed (registry-complete map) | -| Still-open: state-over-national opt-out erasure | fixed (grants-only precedence) | -| Still-open: suppression completeness | fixed (monotonic ordering, re-consent clear, write-failure semantics) | -| Still-open: family-epoch cross-key CAS | recorded as client-spec open question 0; deferred | -| Still-open: eventual rows vs alias guarantees | superseded (rewrite cut) | -| Still-open: 4-hop stranding | superseded (rewrite cut) | -| Still-open: N+1 enforce vs N+2 write boundary | fixed (N+1 writes safety-critical records) | -| Still-open: N+2-only legacy reader on N+1 | fixed (accepted in legacy position) | -| Still-open: no-geo guard cookie consumers | deferred with cookie ops (inventory row updated) | -| Still-open: persisted TCF in raw arm | fixed (TCF-sourced effective record triggers arm) | -| Still-open: request-side raw identity | fixed (identity-redacted integration request views) | -| Still-open: RequestFilterEffects.response_headers | fixed (folded into hook, done-when item) | -| Still-open: must-understand | fixed (sticky set extended) | -| Still-open: Axum persistence overstated | fixed (in-process, non-durable, dev-only cell) | +| Finding | Status | +| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Client page leg pre-gate | fixed (page-leg gating; deferred with client spec) | +| Cookie read/use/withdraw unmodeled | **cookie ops deferred out of v1 hook** (entry bar recorded; sign-off 9/10 updated) | +| Ownerless mode reintroduces fixation | fixed — ownerless mode removed outright | +| Graphless cookie adoption | fixed (adoption transaction, matrix row 13) | +| Rewrite retention lineage | superseded — rewrite_legacy cut; finding recorded as entry bar | +| Unreferenced provider blocks | fixed (startup error) | +| Fastly prefix-query delimiter | R7's per-adapter `:` delimiter was itself Fastly-rejected and non-portable → **refixed R8**: delimiter-free fixed-width grammar (class tag + registry provider code) | +| P2 alias/tombstone cluster counting | fixed (liveness/kind filtering; aliases reserved-future) | +| P2 push-vs-deploy validation | fixed (two named layers, capability profile) | +| P2 rollback floor unobservable | fixed (floor = writer activation, durable marker) | +| P2 cookie ownership uniqueness | integration-ID uniqueness kept; cookie ownership deferred with cookie ops | +| Still-open: GPP 6/24–27 | fixed (registry-complete map) | +| Still-open: state-over-national opt-out erasure | fixed (grants-only precedence) | +| Still-open: suppression completeness | partial R7 (ordering claimed without CAS; cause-list coverage; timestamp-less unrealizable) → **refixed R8** (CAS + version counter, delta coverage, sticky opt-out) | +| Still-open: family-epoch cross-key CAS | recorded as client-spec open question 0; deferred | +| Still-open: eventual rows vs alias guarantees | superseded (rewrite cut) | +| Still-open: 4-hop stranding | superseded (rewrite cut) | +| Still-open: N+1 enforce vs N+2 write boundary | fixed (N+1 writes safety-critical records) | +| Still-open: N+2-only legacy reader on N+1 | fixed (accepted in legacy position) | +| Still-open: no-geo guard cookie consumers | deferred with cookie ops (inventory row updated) | +| Still-open: persisted TCF in raw arm | fixed (TCF-sourced effective record triggers arm) | +| Still-open: request-side raw identity | asserted R7 without API/tests → **specified R8** (`RedactedRequestView`, enumerated strip set, same-PR migration, denied/withdrawn tests) | +| Still-open: RequestFilterEffects.response_headers | R7 fold-in would have **broken DataDome** (302/401/403/429 + cookies) → **refixed R8**: distinct core-owned security channel sharing validation + invariant layers | +| Still-open: must-understand | fixed (sticky set extended) | +| Still-open: Axum persistence overstated | partial R7 (still marked wired; head installs `UnavailableKvStore`) → **refixed R8** (not wired) | + +## Round 8 — re-review at 09e54e96 + +| Finding | Status | +| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| P1.1 suppression monotonicity unprovidable | fixed (linearizable CAS + record version counter; sticky opt-out for timestamp-less sources — sign-off 16; race fixtures) | +| P1.2 malformed/absent leave stale authority | fixed (suppression on every positive→unset delta, cause-agnostic) | +| P1.3 suppression-write failure "transient" | fixed (unbounded residual, shares sign-off 11, fault test) | +| P1.4 N/A double meaning | fixed (single rule: explicit N/A = grant-class, absent = nothing; sign-off 17) | +| P1.5 adoption = syntax-as-authentication | fixed (`verify` against request evidence; expire on failure; full required_permissions; atomic create-if-absent capability; read-error ≠ not-found) | +| P1.6 N+1 impossible write behavior | fixed (N+1 mints v1 with today's semantics; old-shape config runs pre-epic gate; new contracts activate at N+2/new-shape — sign-off 20) | +| P1.7 N+2-only provider readable by N+1 | fixed (providers ship compiled-in dormant one release early; reader-first per provider) | +| P1.8 cookie deferral contradictions | fixed (ops list, reserved remnant, generic-op wording, core-owned-cookie test) | +| P1.9 DataDome fold-in breakage | fixed (distinct security channel, shared validation/invariant layers, core-mediated security cookies) | +| P1.10 freshness metadata weakening | fixed (`Age`/`Date`/`Expires` reserved, rationale in-spec) | +| P1.11 client-cycle in normative core | fixed (Acquisition enum/ClientResolve/reservations removed from trait surface; `verify` added; deferred doc holds the rest) | +| P1.12 redaction unspecified | fixed (see corrected R7 row above) | +| P2.1 rewrite residue | fixed (tests, runtime row, metrics/retirement swept; alias schema marked reserved) | +| P2.2 delimiter portability | fixed (see corrected R7 row above) | +| P2.3 Axum matrix | fixed (see corrected R7 row above) | +| P2.4 adoption rejuvenation | fixed (migration-cutoff-bounded TTL; sign-off 21) | +| P2.5 stored cluster overcount | fixed (no persistence beyond inputs' lifetime) | +| P2.6 GPP applicability leftovers | fixed (MD/IN/KY/RI sentence removed; section-6 grants defined; regime-`none` row reconciled with applicability) | +| P2.7 mixed-revision divergence | accepted explicitly (sign-off 19) | +| P2.8 floor marker rollbackable | fixed (write-once/CAS deployment metadata) | +| P2.9 sign-off gaps | fixed (rows 16–21 added; rows 3 and 11 amended) | +| P2.10 duplicate integration IDs | fixed (startup rejection + test; current silent coalescing named) | +| P3 GPP version pinning | fixed (accepted versions enumerated; unknown version = malformed-present) | +| P3 geo region vocabulary | fixed (ISO output or declared canonical mapping) | +| P3 stale fragments + ledger overstatement | fixed (this ledger corrected; hook remnants swept) | From 43422b5fc5a3f2c09edddc8a31c1c74caea6ea7a Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Sat, 1 Aug 2026 14:22:24 -0700 Subject: [PATCH 11/24] Address ninth and tenth reviews: authentication-sound rowless handling, complete suppression contract, closed security channel Both review passes against ff1e113e, dispositioned together. Trait and rowless identity: - generate is restored to the normative trait (lost in an R8 editing accident, making the required mint sequence unimplementable); verify returns VerifiedIdentity carrying the matched configuration version. - Rowless legacy cookies are expired and re-minted, never adopted: prefix-only HMAC verification cannot authenticate the random suffix, so adoption would let H.aaaaab, H.aaaaac, ... each mint a durable row/family. The rowless family ID derives from the authenticated 64-hex prefix only, collapsing all suffix variants into one withdrawable family; matrix row 13 and sign-off 21 updated; the migration cutoff disappears with adoption. Suppression, completed: - Creation is cause-aware and read-free for signal causes (refusal, opt-out, malformed) - conditioning on observing positive provenance through an eventual row loses the stale-replica race; absence uses a narrow permission-exempt suppression-decision read (the P1-gated read circularity); policy-only tightening writes nothing. - CAS fences writes; authoritative evidence recency decides semantics, with a per-cause transition table: sticky clearing only for opt-out causes, malformed/absence clear on any newer valid grant (sign-off 24), delayed older grants never clear, policy never clears. - Anti-replay: beyond-skew future timestamps are malformed; a digest's first normalized timestamp is pinned and never advanced by re-presentation; digests cover the canonical per-permission semantic result, so equivalent encodings cannot renew authority. - Suppression is inside both AuthorizedIdentity constructors, reads fail closed, retention outlives masked authority, and clearing is fenced on visibility of the matching provenance generation. Storage and capabilities: - New capability rows with per-adapter values: linearizable per-key CAS (suppression), generation-CAS row mutation (rows are heavily mutable, not accretive - Fastly generation markers eligible, Workers KV last-write-wins ineligible), atomic create-if-absent, write-once deployment metadata; revocation reads require globally observable strong consistency, not writer-session read-your-writes; per-class durability and maximum-retention proofs at startup. - Rows carry an absolute expires_at pinned at mint (updates write remaining lifetime - cluster refresh can no longer immortalize an identity) and an immutable mint tag split from the replaceable evidence snapshot; network evidence split from refreshable derived cluster state; cluster_trust_threshold validated against listing caps. - The key grammar's class tags are all non-hex (family tag f -> r) so legacy-grammar disjointness is provable, and provider codes come from a checked-in append-only never-reused registry; the identifier bound is numeric (256 bytes) in the normative contract. Migration: - Once new-shape config is active, N+1 batch sync fails closed on provenance-less rows - the fail-closed rule cannot activate later than the model it protects. - The schema floor has a protocol: dedicated deployment-metadata primitive, create-or-CAS with read-back before enabling writes, startup enforcement, fail-closed on unreadable. - Provider rollback is config-first (writer back, retain as legacy reader, then binaries), distinct from binaries-first schema rollback. - Irreversible artifacts enumerated (revocation, floor, sticky suppression) with recovery/administrative procedures; fixtures branch on capability eligibility; sign-off rows 22-24 added. Hook and security channel: - CDN cache fields reserved outright; stale-* durations shrink-only; the last Set-Cookie contradiction removed; append/replace legality from a core-owned field registry with unknown-fields-reject-append; operations are attributed batches, validated and budgeted atomically (a security 302 can never keep its cookie but lose Location). - Section 4a closes the security channel: typed owned-name cookie operation (ts-* rejected, sign-off 23 for the identifier lifecycle), direction-scoped request-header allowlists applied to a scoped upstream overlay (no credential/identity/routing injection), decision-scoped representation (a challenge owns its body; Continue cannot touch publisher bytes), one global order with the invariant pass unconditionally last. - Eligibility matrix gains HEAD (header parity with GET mandatory) and explicit 1xx/204/205/206 rows. Deferred drafts: the client-cycle page leg gains a pre-vendor-contact live permission check with BFCache abort (TOCTOU); its stale references to the removed acquisition API and old key grammar are marked for renormalization. Ledger: the five overstated R8 dispositions are reopened and corrected, and a Rounds 9-10 section records every finding above. --- ...26-07-30-client-cycle-ec-resolve-design.md | 19 +- ...integration-response-header-hook-design.md | 115 +++++++--- .../2026-07-30-permission-model-design.md | 142 +++++++----- .../2026-07-30-pluggable-providers-design.md | 207 ++++++++++-------- ...07-30-provider-migration-rollout-design.md | 155 +++++++------ docs/superpowers/specs/pr986-review-ledger.md | 85 ++++--- 6 files changed, 451 insertions(+), 272 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index c1ea840ed..eda8268eb 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -66,10 +66,13 @@ Everything in this spec follows from that. enough to set identity. Requests with no `Origin` and no valid token are rejected. 2. **Verify the payload cryptographically per provider — including against - replay.** `resolve_from_client` is the client-resolve acquisition mode - of the provider contract (providers spec §4 - `Acquisition::ClientResolve`, which also carries the JS module the - page leg needs) — no longer an undeclared method this spec invents. It + replay.** `resolve_from_client` belongs to the client-resolve acquisition + surface that was **removed from the normative provider contract with + this feature's deferral** (providers spec §4) — it, the acquisition + enum, and the reservation schemas live only in this informative + draft, to be renormalized (against the current trait and the + delimiter-free key grammar, which superseded the `resv/…` sketch + here) when the feature gets its issue. It accepts only payloads that are signed by an expected party, **audience-bound** to this publisher, and **expiring**. Audience binding and expiry alone do not @@ -195,7 +198,13 @@ with only the later POST refused. The module is injected/activated only when the request's resolved permissions already satisfy the provider's complete `required_permissions()`, and the page leg is a listed row in the permission spec's §7 enforcement inventory (deferred alongside this -feature). +feature). Injection-time gating alone has a TOCTOU gap: consent can be +withdrawn between document delivery and asynchronous vendor contact, or +the document can be restored from BFCache long after its permissions +were resolved — so the module must additionally perform a **live +CMP/permission check immediately before vendor contact** and abort on +permission change or BFCache restoration; endpoint rejection is too late +to undo browser-side identity derivation or vendor egress. - The re-post guard must not depend on reading an HttpOnly cookie. Either the server injects a "resolved" marker the script _can_ read (a diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index b16350ac2..cee462fe4 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -73,9 +73,13 @@ mutators to the outbound response for HTML document responses it processed. mutation ⇒ present in the final response, independently; `public` is dropped whenever any restriction is present; `max-age`/`s-maxage` may only shrink relative to the snapshot; `stale-while-revalidate`/ - `stale-if-error` may appear only if the snapshot had them; every - CDN/surrogate directive (`Surrogate-Control`, `CDN-Cache-Control`, - host equivalents) is stripped from any restricted response; and the + `stale-if-error` may appear only if the snapshot had them **and their + durations may only shrink** (present-at-1s must not become + present-at-1y); CDN-specific cache fields (`Surrogate-Control`, + `CDN-Cache-Control`, host equivalents) are **reserved outright** — + merging them per-directive on unrestricted responses was a hole (an + unrestricted `CDN-Cache-Control: max-age=60` could become a year), and + they are additionally stripped from any restricted response; and the final `Vary` is the **union of the complete snapshot `Vary` set** — origin-supplied members included, not only core-required ones — and the mutation. Middle-stage placement also keeps @@ -126,11 +130,13 @@ mutators to the outbound response for HTML document responses it processed. constants next to the definitions they protect, not duplicated in the hook. - For non-reserved headers, the mutator API distinguishes **append** from - **replace** explicitly; **append is valid only for genuinely - list-valued headers** (a singleton header accepts only replace — two - values of a singleton header by append is a malformed response, not a - merge); the default is append where legal (for `Set-Cookie`, append is - the only non-reserved operation — replace is not offered). Replacing a + **replace** explicitly; append/replace legality comes from a **core-owned field registry**, + not adapter judgment: each known field is classified + append-legal (genuinely list-valued: `Link`, CSP report groups, …), + replace-only (singletons: `Content-Language`, …), or rejected; + **unknown extension fields reject append by default** (replace only) — + "genuinely list-valued" is not a decision four adapters can make + independently and identically (`Set-Cookie` is fully reserved in v1 — neither append nor replace). Replacing a header the origin set is a deliberate act, visible in the mutator's code. - Later registrations see earlier mutations (order = registration order, which is deterministic). @@ -155,8 +161,16 @@ mutators to the outbound response for HTML document responses it processed. **excludes every `Set-Cookie` value and every reserved identity, consent, and privacy header value** (names may be listed as present; values are withheld). - Exceeding a limit rejects the excess operations (logged, attributed), - never the response. A mutator that returns an error is skipped in full — its + Operations arrive as **attributed batches bound to a registration + ID** — one batch per integration per response, ordered by + registration, with the security channel's batch (§4a) ordered before + response mutators; the current flat effects vector satisfies neither + attribution nor budgets and is restructured accordingly. Validation + and budgeting are **atomic per batch**: a batch that exceeds its + budget is rejected whole (logged, attributed), never partially + applied — item-by-item rejection could apply a security 302's + `Set-Cookie` while dropping its `Location`. The response itself is + never rejected. A mutator that returns an error is skipped in full — its operations are all-or-nothing — and the response proceeds without it. **Panics are forbidden and fatal, not recoverable**: the primary target (`wasm32-wasip1`) builds with `panic = "abort"`, so there is no unwind @@ -170,36 +184,73 @@ mutators to the outbound response for HTML document responses it processed. Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| --------------------------------------------- | ------------------------------------------------------------ | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Yes, applied at serve time (mutations are not cached) | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` | No | +| Response | Hook runs? | +| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Yes, applied at serve time (mutations are not cached) | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` | No | +| `HEAD` of a processed document | **Yes — header parity with GET is mandatory** (a cache may refresh stored GET metadata from HEAD, and divergent CSP/privacy metadata between the two is a leak) | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). +## 4a. The security channel — normative closed boundary + +The security channel (today: DataDome) is not a general exception; every +degree of freedom is closed: + +- **Typed security-cookie operation, not header strings.** The channel + emits cookies only through a typed operation whose cookie **names come + from the integration's registered ownership list** (for DataDome, its + documented cookie); every `ts-*` name is rejected; domain/path scope, + attributes, size, and lifetime are constrained by the registration. + Read/vendor-egress/withdrawal semantics of the resulting identifier + are a ratified security-purpose carve-out — **sign-off item 23** — + because the tag-injection → cookie/ClientID read → vendor-send + lifecycle otherwise hands a permission-denied visitor a stable, + exported identifier. No other request filter inherits the cookie + capability. +- **Request-header pointers are direction-scoped allowlists.** Values a + security response names for copying into the request (DataDome's + header-pointer mechanism) are accepted only from a **documented + enrichment-header allowlist**; authentication, `Cookie`, + `Forwarded`/`X-Forwarded-*`, identity, consent, and routing-authority + fields are rejected by name and by class — a compromised endpoint must + not replace origin credentials, inject `ts-ec`, or spoof client + location — and accepted values apply to a **narrowly scoped upstream + overlay**, never the shared request that later integrations read. +- **Representation rules are decision-scoped.** A _Respond_ decision + (challenge/deny) **owns its body** and may set representation headers + (`Content-Type`, encoding, validators) for it — the hook's + representation reservation exists because ordinary mutators do not own + the body, and this one does. A _Continue_ decision may not touch + representation metadata of publisher bytes. +- **One global order, no "wins" exception:** core finalization → + hook/security effects → **final cache/privacy invariant pass, + unconditionally last**. The prior DataDome contract's "applies last + and wins" holds only _within_ the effects layer; nothing outranks the + invariant pass, or a challenge could combine `Set-Cookie` with public + caching. +- The channel adopts the shared layers: structured attributed batches + (§3, atomic per batch — a 302 must never lose `Location` to a budget + while keeping its cookie; on rejection the channel follows DataDome's + specified fail-open), reserved header names, budgets, and the + invariant pass. + ## 4. Done-when (from #782, sharpened) 1. Trait + builder + registry application, each public item documented. 2. **The pre-existing `RequestFilterEffects.response_headers` channel - remains a distinct, core-owned security channel — not folded in, and - not left unvalidated.** Folding it into this hook would break its one - real consumer: DataDome sets headers **and cookies** on 200, 301/302, - 401, 403, and 429 responses — challenge and deny flows on exactly the - response classes (§3a) this hook never runs on, and with cookie - emission v1 reserves. Instead, the channel keeps its own eligibility - (security-integration responses of any status), its cookies are - **core-mediated security cookies** (explicitly outside the deferred - integration-cookie surface, migrated deliberately when that surface - lands), and it adopts the **shared validation layers**: the - structured-operation checks, reserved header names, budgets, and the - final cache/privacy invariant pass. One invariant enforcer, two - eligibility domains. + remains a distinct, core-owned security channel — §4a defines its + closed boundary.** Folding it into this hook would break its one real + consumer: DataDome sets headers **and cookies** on 200, 301/302, 401, + 403, and 429 responses — response classes (§3a) this hook never runs + on, with cookie emission v1 reserves. 3. **At least one real consumer ships in the same PR** — an existing integration registering a mutator for a real need (or, failing a real need, the feature waits; scaffolding with only self-referential tests is diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 21889eb54..360a3f2c1 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -417,53 +417,76 @@ and the fail-closed marker: discoverable from every member, and the record survives member-tombstone replacement (which today discards the original row's identity and metadata, making sibling discovery impossible). -- **Negative authority has its own permission-exempt record.** A live - refusal or non-destructive opt-out must clear prior positive - provenance — but the row write that would do it requires `store-on-device`, - which the refusal just unset, and identity rows may be eventually - consistent, so a stale replica could resurrect a P4 grant after a - targeted-advertising opt-out. The fix is a **suppression record** in - the strongly consistent class, and — because monotonicity is a - read-modify-write property, not a read property — suppression writes - require **linearizable per-key CAS** (providers spec §6.3 - `sup/`, §7 matrix): with plain read-after-write, two - writers can both read the record and an older clear can overwrite a - newer suppress. The record carries its own **version counter, - incremented through the CAS**, so transitions are ordered by - serialization, not by comparing wall-clock timestamps. - - Coverage is **every authority-clearing delta, not an enumerated cause - list**: after each live resolution, any permission whose new state is - unset while its stored provenance is positive gets a suppression - entry — refusal, non-destructive opt-out, malformed-present, and - applicable absence alike. (The earlier refusal/opt-out-only list left - a hole: a malformed record unsets P1, the P1-gated row update is - thereby forbidden, no suppression is written, and batch sync later - honors the stale grant.) - - **Timestamp-less sources get sticky opt-out.** GPP/USP values carry no - intrinsic timestamp, and opt-out → consent → opt-out(same value) is - information-theoretically indistinguishable from a replay of the first - opt-out. Latest-observation semantics would let a replayed old consent - string clear a newer opt-out, so: a suppression from a timestamp-less - source is cleared **only** by a grant carrying an authoritative - timestamp newer than the suppression's observation (TCF - `LastUpdated`) — a timestamp-less re-consent alone does not clear it. - The consequence (a genuine GPP-only re-consent does not restore - authority until a timestamped source or policy provides it) is - declared and is sign-off item 16. Fixtures: suppress-vs-clear race - under concurrent writers; repeated-value opt-out/consent/opt-out. - - **Write failure fails closed for the live request** (the refusal's - effect stands for this response), but the S2S residual is **unbounded - for a never-returning visitor** — exactly like a failed destructive - revocation, not "transient": other instances continue honoring old - provenance, the breaker is per-instance, and no durable retry exists. - This shares sign-off item 11 (extended to cover suppression) and gets - its own fault test. Every S2S recompute and partner-egress check - consults the record: a suppressed permission is unset whatever the - row's provenance says, so no eventual-consistency edge can restore - it. +- **Negative authority has its own permission-exempt record, with a + complete transition contract.** A live refusal or opt-out must clear + prior positive provenance, but the row write that would do it requires + `store-on-device` — which the refusal just unset — and identity rows + may be eventually consistent. The **suppression record** + (`s`-class key per family, providers spec §6.3) resolves this. Its + contract: + + **Creation is cause-aware and mostly read-free.** A live resolution + whose outcome for a permission is unset writes suppression when the + cause is a **signal state** — refusal, non-destructive opt-out, + malformed-present — **unconditionally**, with no row read: conditioning + on observing positive provenance through an eventually consistent row + loses the race where a stale replica hides a just-committed grant. The + one cause that inherently needs prior state — applicable **absence** + clearing a previously positive permission — uses a narrow + **permission-exempt suppression-decision read** exposing only the + family ID and authority metadata (an undeclared exempt read was the + alternative, and skipping it leaves stale S2S authority). + **Policy-only tightening writes nothing**: a policy edit is not a user + signal (§4.2 trigger 3), and a signal-less request after + granted→denied must not create sticky user suppression that a policy + rollback cannot undo. + + **Writes are CAS-fenced; evidence recency decides semantics.** The + record requires linearizable per-key CAS (providers spec §7): CAS + serialization prevents lost updates, but arrival order does **not** + decide outcomes — each per-permission entry stores its cause, source + class, and authoritative evidence timestamp (first-seen normalization + for timestamp-less sources), and an incoming transition applies only + when its evidence timestamp is **newer than or equal to** the stored + entry's; ties resolve to the more restrictive state. So a delayed + grant with `LastUpdated = 100` never clears a suppression whose + refusal carried `200`, while a genuine re-consent at `300` does. The + transition table, by stored cause: **opt-out from a timestamp-less + source** — cleared only by a grant with an authoritative timestamp + newer than the suppression's observation (sticky opt-out, sign-off + 16); **TCF refusal** — cleared by any regime-accepted grant with newer + authoritative evidence; **malformed-present / absence** — cleared by + any regime-accepted valid grant with newer evidence, including a + timestamp-less grant whose first-seen is newer (these causes are not + user opt-outs, so stickiness does not apply — without this rule, one + truncated request would permanently deny a GPP-only user). Policy + changes never clear user-signal suppressions. + + **Anti-replay for timestamps.** A future-dated record is rejected as + malformed beyond the skew window; within it, the record's digest is + stored with its **first normalized timestamp, which re-presentation + never advances** — otherwise a future-dated TCF string replayed after + an opt-out would keep re-normalizing to "now" and clear it. Equality + digests are computed over the **canonical per-permission semantic + result** of §4.5 aggregation, not the raw encoding — two encodings + (or `N` vs explicit N/A) with the same meaning are the same evidence + and keep the original first-seen, so alternating equivalent values + cannot renew authority. + + **Boundary, retention, ordering.** Suppression is checked by **every + `AuthorizedIdentity` constructor** (both scopes), by pull sync's live + path, and by every S2S recompute — not only "partner egress" prose; a + suppression read failure **fails closed** like a revocation read + failure; retention must outlive the positive authority it masks + (providers spec durability/retention capability); and **clearing is + fenced on provenance visibility**: a clear entry records the + provenance generation it reflects, and S2S honors the clear only when + it can read that generation or newer — clearing first would expose the + _older_ positive snapshot through an eventual read. **Write failure + fails closed for the live request**, and the S2S residual is unbounded + for a never-returning visitor (sign-off 11), with fault tests for + suppress-vs-clear races, repeated-value sequences, and the + stale-provenance-read case. - **The cookie expires only after the family record commits.** - **If the family-record write itself fails, nothing durable exists** — @@ -572,10 +595,13 @@ fields grant nothing (their opt-outs still count, per step 2). claimed MD/IN/KY/RI had no section). A truncated map silently loses opt-outs — a Texas (16) or Maryland (24) sale opt-out must not vanish. The implementation PR cross-checks this list against both the current - decoder's section set and the official registry, and **enumerates the - accepted version per section**; a mapped section carrying an unknown - version is treated as malformed-present (blocks grants, never - withdraws — §4.4), not as absent. Adding a section or version is a + decoder's section set and the official registry, and the accepted version per + section is **pinned normatively to the named IAB GPP registry + revision current at this spec's date (2026-08-01)** — "enumerated by + the implementation PR" was two-implementations-diverge territory; a + mapped section carrying a version outside the pinned revision is + treated as malformed-present (blocks grants, never withdraws — §4.4), + not as absent. Adding a section or version is a change to this map. 2. **Applicability gates grants only — never opt-outs.** A mapped **opt-out** field (either subclass) is honored from **any** section on @@ -671,9 +697,9 @@ A policy edit propagates through the config store, so a fleet briefly mixes revisions. The contract: instances stamp every resolution and every provenance write with the policy revision they used (already required by §7); the mixing window is bounded by config propagation and observable via -the config-version metric; and mixed revisions cannot cause irreversible -harm, because **destructive withdrawal triggers are user signals, never -policy** (§4.2 trigger 3) — the one revision-sensitive destructive case +the config-version metric; and mixed-revision irreversibility is bounded and **accepted, not +denied** (sign-off 19): destructive withdrawal triggers are user +signals, never policy (§4.2 trigger 3) — the one revision-sensitive destructive case (trigger 2 under a now-`denied` baseline) requires an affirmative user refusal at the evaluating instance, which is safe under either revision. S2S recomputation always evaluates against the instance's current @@ -770,8 +796,12 @@ Consumers of the resolved set in this epic: | GPP / USP values (no intrinsic timestamp) | **First-seen**: when TS first observed this exact normalized value (a **per-permission equality digest computed over only the applicable, aggregated §4.5 fields for that permission** — never the whole GPP record, or a CMP touching an unrelated notice field would mint a new digest and reset first-seen forever) | Re-presenting an identical digest **keeps the original first-seen**; a different value is new evidence with a new first-seen | Consent TTL (same as TCF) | | Policy-baseline grant (`granted` rule, no signal) | The policy revision that granted | Re-derived on every recompute against the current revision — policy is not user evidence and does not age; it changes | n/a | - Timestamps are compared with bounded clock-skew tolerance and - future-dated values are clamped to receipt time. And every live + Timestamps are compared with bounded clock-skew tolerance; + beyond-window future-dated records are **rejected as malformed**, and + within the window a record's first normalized timestamp is pinned to + its digest and never advanced by re-presentation (§4.3's anti-replay + rule — clamping every presentation to "now" would make a future-dated + string perpetually fresh). And every live resolution **atomically replaces the complete per-permission snapshot**, never merges — a refusal, opt-out, malformed or absent state in the fresh resolution clears prior positive authority for its diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index f231601e9..5302a7251 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -119,7 +119,9 @@ Three global rules sit above every provider: - **Identifier bounds.** A minted identifier obeys a global cookie-safe alphabet (valid cookie-octets: no separators, whitespace, or control - characters) and a global maximum length — for the identifier itself, not + characters; normatively `[A-Za-z0-9._~-]`) and a global maximum of + **256 bytes** — stated here, in the normative contract, so dependent + documents reference one number instead of assuming their own — for the identifier itself, not only the graph key — enforced by core at mint and at parse, so no provider can emit a value the cookie layer or logs cannot carry. - **Namespaces are declarative and core-proven.** Disjointness of two @@ -150,7 +152,10 @@ Three global rules sit above every provider: trips the trust threshold toward denial, never toward extra writes); the listing filters by the value's `kind`/liveness within the existing list limit where the backend returns values, and the residual - over-count where it cannot is declared. A computed cluster size is + over-count where it cannot is declared. `cluster_trust_threshold` is validated against the backend's listing + cap at startup — a threshold of 200 against a 100-key listing cap + would make every capped count look trusted; the count must page or + saturate at threshold + 1. A computed cluster size is **not persisted beyond its inputs' lifetime**: today's code stores the calculated `cluster_size` in the row and reuses it for the row's full TTL, which would freeze a tombstone-inflated count for up to a year — @@ -206,11 +211,20 @@ pub trait EdgeCookieProvider { /// key, shared across identifiers minted from the same client /// evidence. None when the provider lacks IP-cluster semantics (§3). fn cluster_prefix(&self, id: &EcId) -> Option; + /// Mint an identifier from request evidence. The one acquisition + /// operation of the epic (server mint); failure means no identity + /// this request (§6.2). Lost from an earlier revision by editing + /// accident — its absence made the required gate → generate → + /// graph-commit sequence unimplementable. + fn generate(&self, input: &IdentityInput<'_>) -> Result>; /// Cryptographic verification of a parsed identifier against request - /// evidence — recognition (`parse`) is not authentication; adoption - /// (§5) and any rowless acceptance require this. - fn verify(&self, id: &EcId, input: &IdentityInput<'_>) -> bool; + /// evidence. Recognition (`parse`) is not authentication; rowless + /// handling (§5) requires this. Returns the matched configuration + /// version — provenance needs it and a bool cannot carry it — or + /// None when nothing verifies. + fn verify(&self, id: &EcId, input: &IdentityInput<'_>) -> Option; } +// VerifiedIdentity { version: ProviderVersion /* … */ } ``` The acquisition-mode enum (`ServerMint` / `ClientResolve`), the @@ -250,40 +264,31 @@ header emission; the identity exists durably from that moment. A graph-commit failure means the mint never happened: no cookie, no egress, error logged, the next request retries. -**Pre-existing cookies without rows are verified, then adopted — parse -is recognition, not authentication.** A syntactically valid -`{64hex}.{6alnum}` string is constructible by anyone; adopting it on -shape alone would let an attacker mint durable rows. The contract: - -- ServerMint providers implement `verify(id, &IdentityInput) -> bool` — - cryptographic verification against **request evidence** (for `hmac`: - recompute over the request's evidence with each configured version's - passphrase and compare to the 64-hex prefix). A recognized rowless - cookie that fails verification is **expired, not adopted** — including - the honest false-negative: a legitimate cookie presented from a - changed network no longer verifies and is expired; the affected - population is graphless deployments only, declared in migration row 13. -- Adoption is gated on the provider's **complete - `required_permissions()`** — exactly like minting, not a hard-coded - `store-on-device`. -- The row write is **atomic create-if-absent** — a distinct capability - row in the §7 matrix (strong class; Workers KV's concurrent same-key - writes can overwrite each other, so it is ineligible, consistent with - its revocation ineligibility). -- **Read errors are not "not found"**: adoption proceeds only on an - authoritative not-found; a failed graph read means no adoption this - request, fail closed. -- Adopted rows do **not** get a fresh full TTL — a nearly expired legacy - identity must not gain a year (the exact rejuvenation problem that - deferred rewrite). Expiry is `min(adopted_at + standard TTL, -migration_cutoff + grace)` with the cutoff configured; sign-off - item 21. - -Until adoption succeeds the cookie **never egresses**; **withdrawal -works without adoption** — the deterministic family ID (permission model -spec §4.3) needs no row, so a first post-upgrade opt-out revokes and -expires the cookie with zero migrated state. Migration matrix row 13 -declares this path. +**Pre-existing rowless cookies are expired and re-minted — never +adopted.** An earlier adoption design failed on an authentication limit: +HMAC verification can authenticate only the 64-hex prefix; the 6-char +suffix is independent randomness, so a client holding `H.aaaaaa` can +present `H.aaaaab`, `H.aaaaac`, … — every variant prefix-verifies, and +an adopt path would mint a **separate durable row and family per +variant**. Therefore: + +- A recognized rowless cookie whose prefix verifies + (`verify → VerifiedIdentity`, carrying the matched version for + provenance) is **expired and replaced by a fresh mint through the + ordinary graph-backed path** (gate → `generate` → commit) when the + request's permissions allow one; continuity with the old identifier is + deliberately not preserved (migration matrix row 13, sign-off 21). A + cookie whose prefix does not verify (including the declared roaming + false-negative) is simply expired. +- **Rowless withdrawal cannot be a row/family-minting oracle**: for + rowless legacy HMAC cookies the derived family ID is a function of the + **authenticated 64-hex prefix only**, so every suffix variant maps to + the _same_ family — one family record withdraws them all, and + attacker-generated variants create nothing new. +- Read errors are still not "not found": a failed graph read means the + cookie is treated as absent this request, fail closed, no expiry + emitted (the row may exist). + declares this path. **Egress is typed, not policed.** The inventory-and-denylist test (permission model spec §7) is a backstop, but conventions do not survive @@ -292,8 +297,10 @@ because raw EC values circulate as ordinary strings. Core therefore introduces a **scope-parameterized `AuthorizedIdentity`**, constructible only by core, only after the checks _for that exact scope_: `AuthorizedIdentity` after parse + `store-on-device` + -family-revocation check; `AuthorizedIdentity` additionally -after `select-personalised-ads`. Outbound serializers (ORTB builder, page +family-revocation **and suppression** checks; +`AuthorizedIdentity` additionally after +`select-personalised-ads` — suppression is part of both constructors, not +a separate prose obligation on S2S callers. Outbound serializers (ORTB builder, page bids, sync, identify, forwarding) accept `AuthorizedIdentity` and nothing weaker — an unparameterized wrapper would let a P1-only identity flow into an ORTB request. A future bypass then @@ -433,7 +440,11 @@ The contract: and the alias record class exists in the key grammar (§6.3) only as reserved-for-future — nothing in the epic writes one. - Retiring a legacy reader is the explicit end of those identities: - the migration guide documents the cleanup procedure (migration spec §6). + the migration guide documents the cleanup procedure (migration spec + §6). **Provenance backfill is not retirement evidence** — a backfilled + row still lives under the legacy cookie namespace and still needs that + provider's parser; only a quiet period spanning the full cookie/row + lifetime justifies removal. - Tests: switch active provider → request with old cookie → identity still resolves and a withdrawal tombstones it; old cookie with no matching legacy reader → treated as absent and **never egresses**; @@ -476,8 +487,10 @@ backend-wide outage degrades every instance through its own observations within one window, but an instance-local family-write failure leaves other instances — which have no record to find, and healthy backends of their own — serving S2S egress until the browser's durable signal retries -successfully. That residual is bounded by the user's return latency, is -counted (failed family writes are a first-class metric), and is accepted +successfully. That residual is **unbounded for a never-returning visitor** +(sign-off item 11 — the permission and migration specs state this and +this spec must not undercut them), is counted (failed family writes are +a first-class metric), and is accepted in place of a deployment-wide shared fail-closed channel, whose own availability and freshness would be a harder problem than the one it solves. @@ -487,15 +500,15 @@ solves. **Physical key grammar.** Core constructs every key; providers supply only the bounded suffix: -| Record class | Key | Notes | -| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ 128 bytes, KV-safe alphabet. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | -| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | -| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | -| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | -| Suppression (negative authority) | `sup/` | Per-permission suppression entries + timestamps; permission-exempt writes; consulted by every S2S recompute and partner-egress check (permission spec §4.3) | -| Rewrite transaction | `rwx/` | One in-flight rewrite per family | -| Replay reservation | `resv////` | Client-cycle spec; payload id ≤ 128 bytes | +| Record class | Key | Notes | +| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ 128 bytes, KV-safe alphabet. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | +| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | +| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | +| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | +| Suppression (negative authority) | `sup/` | Per-permission suppression entries + timestamps; permission-exempt writes; consulted by every S2S recompute and partner-egress check (permission spec §4.3) | +| Rewrite transaction | `rwx/` | One in-flight rewrite per family | +| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | Grammars are pairwise non-intersecting by their literal prefixes (plus the reserved hmac grammar), and every record value carries a `kind` @@ -508,14 +521,19 @@ logical identity different physical keys on different adapters, breaking migration, shared storage, and parity; and Fastly's prefix queries reject both `/` and `:`, so no delimiter character is safely portable). Physical keys are **delimiter-free with fixed-width segments**: a -1-character class tag (`i` row, `f` family, `s` suppression, `x` -transaction), a **4-character registry-assigned provider code** -(zero-padded, `[a-z0-9]`), then the suffix — segment boundaries are -positional, so no segment can contain or escape a delimiter, prefix -queries are plain string prefixes on every backend, and cluster -eligibility needs no per-adapter delimiter negotiation. (hmac verbatim -keys remain the reserved exception, with the 64-hex cluster prefix at -position zero.) +1-character class tag — `i` row, `r` family revocation, `s` suppression, +`x` transaction, every tag chosen **outside the hex alphabet** so no +generated key can begin with 64 hex characters, which is what makes +disjointness from the legacy `{64hex}.{6alnum}` grammar _provable_ +rather than asserted (an earlier `f` tag was itself a hex digit) — then +a **4-character provider code from a checked-in, append-only, +never-reused registry file** (allocation is a reviewed commit; +codes are immutable and never recycled, including for retired +providers), then the suffix. Segment boundaries are positional, so no +segment can contain or escape a delimiter, prefix queries are plain +string prefixes on every backend, and a grammar-disjointness test covers +every class against the legacy grammar. (hmac verbatim keys remain the +reserved exception, with the 64-hex cluster prefix at position zero.) **Wire schemas** (JSON, like identity rows; every class carries a schema version): the **alias record** (reserved-future, with rewrite) holds target key, created-at, retirement @@ -543,23 +561,25 @@ readers round-trip unknown keys **semantically** (values preserved through read-modify-write; byte-identical output is not required and not achievable through a structured serializer). -| Field | Purpose | Source | Gating permission (egress) | TTL / refresh | Rewrite | On revocation | -| ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | -| key (v1: identifier verbatim; v2: core-constructed, §4) | Row identity | Provider/core | — | Row TTL (1 y today) | New canonical row; old key becomes alias | Family record governs; member tombstone as cleanup | -| `v` | Schema discriminator | Core | — | — | Written at current version | Retained | -| `created` | Row age | Core | P1 (first-party ops) | Never refreshed | Preserved (no rejuvenation) | Retained in tombstone | -| `consent.tcf` / `consent.gpp` | Raw signal snapshot for audit; superseded as authority by provenance | Request | Never egressed to partners | Replaced on live resolution (§7 snapshot rule, permission spec) | Fresh live values | Scrubbed | -| `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | -| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision, provider/version) | S2S authority | Live resolution only | Read by S2S recompute | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | -| New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | -| `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | -| `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | -| `pub_properties` (origin/seen domains) | Creation context | Core at mint | P1 | Write-once | Preserved | Scrubbed | -| `device.*` (JA4 class, H2 hash, quality metadata) | **Discontinued for new rows** (§5): fingerprint-derived, buyer-facing — beyond security-classification authorization. v1 rows retain them read-only; they are never egressed post-epic and are dropped at rewrite | Fastly device provider | None grants egress | Write-once (v1) | **Dropped** | Scrubbed | -| New: security classification outcome (boolean) | Bot-gate result | Device provider | — (never egressed) | Written at mint | Fresh | Scrubbed | -| `network.*` | Cluster disambiguation | Platform at mint | P1 | Write-once | Fresh | Scrubbed | -| `ids` (partner → UID map) | Partner identity graph | Pixel/pull/batch sync | P1 ∧ P4 (partner egress) | Per-mapping timestamps; bounded count/length | Copied **with original timestamps/expiry** | Scrubbed | -| New: alias record kind | Rewrite indirection (§6.1) | Core | — | Retirement deadline | Is the mechanism | Family-revoked like any member | +| Field | Purpose | Source | Gating permission (egress) | TTL / refresh | Rewrite | On revocation | +| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | +| key (v1: identifier verbatim; v2: core-constructed, §4) | Row identity | Provider/core | — | Row TTL (1 y today) | New canonical row; old key becomes alias | Family record governs; member tombstone as cleanup | +| `v` | Schema discriminator | Core | — | — | Written at current version | Retained | +| `created` / **`expires_at`** | Row age and the **absolute retention deadline, pinned at mint** — every update writes with the _remaining_ lifetime, never a fresh full TTL (today's full-TTL rewrite lets a frequently visited identity live forever; refreshable derived state must never rejuvenate the identity) | Core | P1 (first-party ops) | Never extended | Preserved (no rejuvenation) | Retained in tombstone | +| `consent.tcf` / `consent.gpp` | Raw signal snapshot for audit; superseded as authority by provenance | Request | Never egressed to partners | Replaced on live resolution (§7 snapshot rule, permission spec) | Fresh live values | Scrubbed | +| `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | +| New: **immutable mint tag** (`mint_provider`, `mint_version`) | Credential retirement and audit — write-once at mint (legacy backfill may populate a missing tag once); **never part of the replaceable snapshot**, or a v1 identity revisited after rotation would be restamped v2 | Mint (or one-time backfill) | — | Immutable | — | Retained | +| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision, provider/version) | S2S authority | Live resolution only | Read by S2S recompute | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | +| New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | +| `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | +| `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | +| `pub_properties` (origin/seen domains) | Creation context | Core at mint | P1 | Write-once | Preserved | Scrubbed | +| `device.*` (JA4 class, H2 hash, quality metadata) | **Discontinued for new rows** (§5): fingerprint-derived, buyer-facing — beyond security-classification authorization. v1 rows retain them read-only; they are never egressed post-epic and are dropped at rewrite | Fastly device provider | None grants egress | Write-once (v1) | **Dropped** | Scrubbed | +| New: security classification outcome (boolean) | Bot-gate result | Device provider | — (never egressed) | Written at mint | Fresh | Scrubbed | +| `network.*` (immutable evidence: ASN etc.) | Cluster disambiguation | Platform at mint | P1 | Write-once | Fresh | Scrubbed | +| Derived cluster state (`cluster_size`, computed-at) | Trust gating | Computed | — | **Refreshable, short validity; generation-CAS update; never touches `expires_at`** | Recomputed | Scrubbed | +| `ids` (partner → UID map) | Partner identity graph | Pixel/pull/batch sync | P1 ∧ P4 (partner egress) | Per-mapping timestamps; bounded count/length | Copied **with original timestamps/expiry** | Scrubbed | +| New: alias record kind | Rewrite indirection (§6.1) | Core | — | Retirement deadline | Is the mechanism | Family-revoked like any member | ## 7. Composition root and adapter parity @@ -581,17 +601,26 @@ Requirements: **per-record-class consistency requirements**, because "has KV" says nothing about whether revocation is observable: - | Record class | Required semantics | - | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | - | Family revocation records | **Strongly consistent (read-after-write) primitive required.** Cloudflare Workers KV is **not eligible** — its documentation says propagation may take "60 seconds or more", an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | - | Family suppression records | Same strong class as family revocation — negative authority must not lose races to stale replicas | - | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | - | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | - | Identity rows | Eventual consistency acceptable — rows are accretive, and the family-record check (strong, per above) governs use | - - Each adapter's declaration is part of its wiring, drives the §6 - capability-mismatch startup error, and every §6.2 runtime-failure row + | Record class | Required semantics | + | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | + | Family revocation records | **Globally observable strong consistency** — every instance's read observes a committed revocation, not merely the writing session's own writes (writer-scoped read-your-writes is insufficient for a fleet). Cloudflare Workers KV is **not eligible** — "60 seconds or more" is an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | + | Family suppression records | **Linearizable per-key CAS** — read-after-write alone cannot provide read-modify-write monotonicity: two writers both read, and an older clear overwrites a newer suppress | + | Identity-row mutation | **Generation CAS** (conditional write on row generation) with reread/recompute on conflict — rows are heavily mutable (snapshots replaced, partner IDs merged, derived state refreshed), so unordered last-writer-wins loses newer evidence and mappings; _visibility_ may stay eventual, unordered _mutation_ may not. Fastly KV offers generation-marker conditional writes; Workers KV's documented concurrent last-write-wins is ineligible for mutation-bearing rows | + | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | + | Deployment metadata (schema floor) | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | + | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | + | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | + | Identity rows | Eventual consistency acceptable — rows are accretive, and the family-record check (strong, per above) governs use | + + Every record class additionally declares **durability and maximum + retention**: a store passing the consistency check but capping TTLs + below the computed revocation/suppression horizon (e.g. a 30-day + maximum against one-year rows) would let identities become usable + again when their revocation expires — startup proves the configured + store meets each class's computed horizon, and persistence across + restart is part of the declaration. Each adapter's declaration is part + of its wiring, drives the §6 capability-mismatch startup error, and every §6.2 runtime-failure row gets fault-injection coverage on every adapter declaring the corresponding capability. The **concrete per-adapter values** — the actual matrix, not the abstract capability list — as known today; a diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 6299d5448..82fe5779b 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -32,30 +32,30 @@ and whether that is a preservation or a declared change. **Silent changes are defects.** PR #838 changed six of these without declaring any; each was discoverable only because a deleted test had pinned the old behavior. -| # | Decision (today) | After epic | Status | -| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | -| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | -| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | -| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | -| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | -| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | -| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | -| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — opt-out effects are more protective; the same fields' not-opted-out values can also **newly grant P4**, which is not (both effects classified in permission spec §4.5) | -| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | -| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | -| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | -| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | -| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | -| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | -| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | -| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | -| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | -| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | -| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | -| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | -| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | -| 13 | Cookies minted by graphless deployments have no graph row | Recognized rowless legacy cookies are adopted via a permission-gated, race-safe create-if-absent on a live request (providers spec §5); never egress before adoption; withdrawal works without adoption via the derived family ID | **Declared** — identity use of pre-existing cookies pauses until adopted | +| # | Decision (today) | After epic | Status | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | +| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | +| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | +| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | +| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | +| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — opt-out effects are more protective; the same fields' not-opted-out values can also **newly grant P4**, which is not (both effects classified in permission spec §4.5) | +| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | +| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | +| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | +| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | +| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | +| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | +| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | +| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | +| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | +| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | +| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | +| 13 | Cookies minted by graphless deployments have no graph row | Recognized rowless cookies are **expired and re-minted** through the ordinary graph-backed path (providers spec §5) — never adopted, since prefix-only verification cannot authenticate suffix variants; identity continuity is deliberately lost; withdrawal works without any row via the prefix-derived family ID | **Declared** — pre-existing identities restart rather than carry over | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -142,9 +142,15 @@ Requirements: unchanged** — dual-read means dual-behavior — so the compiled protective fallback cannot flip behavior mid-convergence before the operator pushes the new-shape policy; the new model engages only - with new-shape config. The interim (N+1 minting v1 rows, S2S - running today's checks) is declared as sign-off item 20, not - discovered. + with new-shape config. The interim is declared as sign-off item 20 — with one + boundary that does **not** wait for N+2: once new-shape config is + active, **context-free partner egress (batch sync) on N+1 fails + closed for rows without provenance**, exactly as the permission + spec's legacy rule requires. Otherwise N+1 would mint a P1-only v1 + row under the new model and then release it through today's + row-state-only batch check — the fail-closed rule cannot activate + later than the model it protects. Live-request paths keep v1 + semantics until N+2. Rollback tests therefore run the family-revocation and suppression paths — read **and write** — plus v1-minting behavior, on N+1 @@ -158,11 +164,15 @@ Requirements: must ship compiled-in (dormant: registered, parseable, configurable, not selectable as writer) in R−1**; adopting a genuinely new provider gets its own reader-first rollout, exactly - like the epic itself. With that rule, rolling N+2 → N+1 keeps the - new provider's identities resolvable and withdrawable through the - dormant registration ("retain as legacy reader" is now satisfiable - because N+1 physically contains the code); the fleet still first - converges on a config selecting only what N+1 accepts as writer. + like the epic itself. With that rule, **schema rollback and provider rollback are + distinct sequences**: schema rollback is binaries-first (above); + **provider rollback is config-first** — a fleet whose config + _selects_ the new provider as writer cannot roll binaries first, + because the older binary rejects that active writer even while + containing its dormant code. The order: switch the current fleet's + writer back to the older provider (retaining the new one in + `legacy_providers`, satisfiable because N+1 physically contains the + code), converge, then roll binaries. N+1 additionally **rejects writer selections whose provenance it cannot yet encode** — new-writer adoption waits for N+2, so no row is minted that N+2 would misclassify. Every new @@ -212,12 +222,20 @@ Requirements: genuinely pre-N+1 worker cannot preserve at all, which is exactly why the floor exists); after the **fleet-convergence gate**, **N+2 activates the writer** and begins emitting the new fields. **The rollback floor is crossed at N+2 writer activation itself** — an - observable deploy event, recorded as a durable schema-floor marker - **in write-once/CAS deployment metadata that ordinary config rollback - cannot touch** — floor-in-rollbackable-config would let "restore the - previous config version" erase the floor after new-format rows exist, - which is exactly the state it guards — not at "any new-format row - exists", which no operator can disprove. Below-floor rollback is + observable deploy event, recorded in the **deployment-metadata + primitive** (providers spec §7 capability row; the existing + config-store interface exposes ordinary put/delete and cannot express + a monotonic floor) with a specified protocol, not an assertion: the + marker lives in a dedicated namespace outside rollbackable config; + the **first N+2 instance to activate creates/advances it via + create-or-CAS** (the creation race resolves to one winner), **reads + it back, and only then enables new-format writes**; every binary + reads the floor at startup and a binary below the floor **fails + startup**; an unreadable floor fails closed (writer stays disabled). + Floor-in-rollbackable-config would let "restore the previous config + version" erase the marker after new-format rows exist — exactly the + state it guards — and "any new-format row exists" is a fact no + operator can disprove. Below-floor rollback is prohibited from that marker on; a pre-floor binary would silently strip the new fields from every row it touches. Rows carry the existing `v` schema discriminator; backfill is lazy via @@ -280,7 +298,10 @@ provenance gate (row 11c). Everything else the recipe preserves. The migration guide (a new `docs/guide/` page, linked from the release notes) gives one copy-pasteable recipe per adapter for the minimal-divergence -posture: +posture — **branching on capability eligibility**: adapters passing the +revocation-storage gate get the HMAC + graph fixture below; ungated +adapters get the explicitly stateless fixture of §4.2, and no universal +HMAC requirement contradicts that: The recipe is a **complete, valid TOML fixture per adapter, committed to the repository** (e.g. `docs/guide/fixtures/migration-preserving-fastly.toml` @@ -381,7 +402,12 @@ global honoring of opt-out signals is unconditional. model) is rejected in the permission spec. 5. Rollback is config-only where possible: reverting to the previous config version restores the previous behavior on the previous binary. The - one irreversible artifact is withdrawal tombstones — which is why the + irreversible artifacts are enumerated — not "one": **family + revocation records and member tombstones** (no recovery; that is + their purpose), the **schema-floor marker** (write-once by design; + no administrative clear), and **sticky timestamp-less suppression** + (administrative clear procedure documented in the guide, requiring + recorded operator intent). Withdrawal tombstones — which is why the withdrawal triggers (permission spec §4.2) are exhaustive, why partial withdrawal failure has an explicit tombstones-first, browser-retries contract (permission spec §4.3), and why §2 rows 6 and 8 call out @@ -412,26 +438,29 @@ made differently). **Implementation is blocked while any row is `open`**; each row needs an owner, a status, and a link to its decision record — an unratified row reverts to open, not to silently implemented. -| # | Decision | Where | Owner | Status | -| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | -| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | -| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | -| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | maintainers + legal | open | -| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | -| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | -| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | -| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | -| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | -| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | -| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | -| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | -| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | -| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | -| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | -| 16 | Sticky opt-out for timestamp-less sources: a GPP/USP-only re-consent does not restore authority without a timestamped grant | permission §4.3 | maintainers + legal | open | -| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | maintainers + legal | open | -| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | maintainers + legal | open | -| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | maintainers + product | open | -| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | maintainers + product | open | -| 21 | Adopted legacy rows are bounded by a migration cutoff, not a fresh full TTL | providers §5 | maintainers + product | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | +| # | Decision | Where | Owner | Status | +| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | +| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | +| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | +| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | maintainers + legal | open | +| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | +| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | +| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | +| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | +| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | +| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | +| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | +| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | +| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | +| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | +| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | +| 16 | Sticky opt-out for timestamp-less sources: a GPP/USP-only re-consent does not restore authority without a timestamped grant | permission §4.3 | maintainers + legal | open | +| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | maintainers + legal | open | +| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | maintainers + legal | open | +| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | maintainers + product | open | +| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | maintainers + product | open | +| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | maintainers + product | open | +| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | maintainers + legal | open | +| 23 | DataDome security exemption: tag injection, cookie/ClientID read, vendor egress, and cross-integration visibility operate outside the permission model as a ratified security-purpose carve-out with owned cookie names, scope, and withdrawal semantics | hook §4a; permission §7 | maintainers + legal | open | +| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky); opt-out stickiness applies only to opt-out causes | permission §4.3 | maintainers + legal | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index fa5a68756..85a35e965 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -133,30 +133,61 @@ the R7 table below). Sign-off table with owners/status introduced R6. ## Round 8 — re-review at 09e54e96 -| Finding | Status | -| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| P1.1 suppression monotonicity unprovidable | fixed (linearizable CAS + record version counter; sticky opt-out for timestamp-less sources — sign-off 16; race fixtures) | -| P1.2 malformed/absent leave stale authority | fixed (suppression on every positive→unset delta, cause-agnostic) | -| P1.3 suppression-write failure "transient" | fixed (unbounded residual, shares sign-off 11, fault test) | -| P1.4 N/A double meaning | fixed (single rule: explicit N/A = grant-class, absent = nothing; sign-off 17) | -| P1.5 adoption = syntax-as-authentication | fixed (`verify` against request evidence; expire on failure; full required_permissions; atomic create-if-absent capability; read-error ≠ not-found) | -| P1.6 N+1 impossible write behavior | fixed (N+1 mints v1 with today's semantics; old-shape config runs pre-epic gate; new contracts activate at N+2/new-shape — sign-off 20) | -| P1.7 N+2-only provider readable by N+1 | fixed (providers ship compiled-in dormant one release early; reader-first per provider) | -| P1.8 cookie deferral contradictions | fixed (ops list, reserved remnant, generic-op wording, core-owned-cookie test) | -| P1.9 DataDome fold-in breakage | fixed (distinct security channel, shared validation/invariant layers, core-mediated security cookies) | -| P1.10 freshness metadata weakening | fixed (`Age`/`Date`/`Expires` reserved, rationale in-spec) | -| P1.11 client-cycle in normative core | fixed (Acquisition enum/ClientResolve/reservations removed from trait surface; `verify` added; deferred doc holds the rest) | -| P1.12 redaction unspecified | fixed (see corrected R7 row above) | -| P2.1 rewrite residue | fixed (tests, runtime row, metrics/retirement swept; alias schema marked reserved) | -| P2.2 delimiter portability | fixed (see corrected R7 row above) | -| P2.3 Axum matrix | fixed (see corrected R7 row above) | -| P2.4 adoption rejuvenation | fixed (migration-cutoff-bounded TTL; sign-off 21) | -| P2.5 stored cluster overcount | fixed (no persistence beyond inputs' lifetime) | -| P2.6 GPP applicability leftovers | fixed (MD/IN/KY/RI sentence removed; section-6 grants defined; regime-`none` row reconciled with applicability) | -| P2.7 mixed-revision divergence | accepted explicitly (sign-off 19) | -| P2.8 floor marker rollbackable | fixed (write-once/CAS deployment metadata) | -| P2.9 sign-off gaps | fixed (rows 16–21 added; rows 3 and 11 amended) | -| P2.10 duplicate integration IDs | fixed (startup rejection + test; current silent coalescing named) | -| P3 GPP version pinning | fixed (accepted versions enumerated; unknown version = malformed-present) | -| P3 geo region vocabulary | fixed (ISO output or declared canonical mapping) | -| P3 stale fragments + ledger overstatement | fixed (this ledger corrected; hook remnants swept) | +| Finding | Status | +| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| P1.1 suppression monotonicity unprovidable | R8 partial (CAS ordered arrival, not evidence recency; creation still row-observation-dependent) → **refixed R9/R10**: evidence-recency transition table, cause-aware read-free creation, exempt decision read, boundary/retention/fenced clearing | +| P1.2 malformed/absent leave stale authority | fixed (suppression on every positive→unset delta, cause-agnostic) | +| P1.3 suppression-write failure "transient" | fixed (unbounded residual, shares sign-off 11, fault test) | +| P1.4 N/A double meaning | fixed (single rule: explicit N/A = grant-class, absent = nothing; sign-off 17) | +| P1.5 adoption = syntax-as-authentication | R8 partial (prefix verify cannot authenticate suffixes — adoption itself was unsound) → **refixed R9/R10**: adoption removed; rowless cookies expire-and-re-mint; prefix-derived family collapses suffix variants; `verify → VerifiedIdentity{version}` | +| P1.6 N+1 impossible write behavior | fixed (N+1 mints v1 with today's semantics; old-shape config runs pre-epic gate; new contracts activate at N+2/new-shape — sign-off 20) | +| P1.7 N+2-only provider readable by N+1 | fixed (providers ship compiled-in dormant one release early; reader-first per provider) | +| P1.8 cookie deferral contradictions | R8 partial (append-only-non-reserved remnant survived) → **swept R9/R10** | +| P1.9 DataDome fold-in breakage | R8 partial (channel kept but boundary open: untyped cookies, header pointers, ordering conflict) → **refixed R9/R10**: §4a closed boundary — typed owned-name cookie op, direction-scoped allowlists, decision-scoped representation, invariant-last ordering, atomic batches; sign-off 23 | +| P1.10 freshness metadata weakening | fixed (`Age`/`Date`/`Expires` reserved, rationale in-spec) | +| P1.11 client-cycle in normative core | fixed (Acquisition enum/ClientResolve/reservations removed from trait surface; `verify` added; deferred doc holds the rest) | +| P1.12 redaction unspecified | fixed (see corrected R7 row above) | +| P2.1 rewrite residue | fixed (tests, runtime row, metrics/retirement swept; alias schema marked reserved) | +| P2.2 delimiter portability | fixed (see corrected R7 row above) | +| P2.3 Axum matrix | fixed (see corrected R7 row above) | +| P2.4 adoption rejuvenation | fixed (migration-cutoff-bounded TTL; sign-off 21) | +| P2.5 stored cluster overcount | fixed (no persistence beyond inputs' lifetime) | +| P2.6 GPP applicability leftovers | fixed (MD/IN/KY/RI sentence removed; section-6 grants defined; regime-`none` row reconciled with applicability) | +| P2.7 mixed-revision divergence | accepted explicitly (sign-off 19) | +| P2.8 floor marker rollbackable | fixed (write-once/CAS deployment metadata) | +| P2.9 sign-off gaps | fixed (rows 16–21 added; rows 3 and 11 amended) | +| P2.10 duplicate integration IDs | fixed (startup rejection + test; current silent coalescing named) | +| P3 GPP version pinning | R8 partial (still implementation-enumerated) → **refixed R9/R10**: pinned to a named registry revision in the normative spec | +| P3 geo region vocabulary | fixed (ISO output or declared canonical mapping) | +| P3 stale fragments + ledger overstatement | fixed (this ledger corrected; hook remnants swept) | + +## Rounds 9–10 — dual review at ff1e113e + +R9 (10 P1, 13 P2, 1 P3) and the same-head re-audit R10 (9 P1, 9 P2, 1 P3) +are dispositioned together; R10's "previously open" list = R9's P1s, +tracked once. + +| Finding | Status | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | +| R9-1 trait cannot mint (`generate` lost in an R8 edit) | fixed — restored with failure semantics | +| R9-2 / R10-open rowless HMAC unauthenticatable suffixes | fixed — expire-and-re-mint, prefix-derived family, `VerifiedIdentity{version}` | +| R9-3 missing capability rows (suppression CAS, create-if-absent, floor metadata) + global visibility | fixed — distinct rows with per-adapter values (Fastly generation markers; Workers KV ineligible), globally observable revocation reads | +| R9-4 CAS orders arrival not recency | fixed — evidence-recency transition table with per-cause clearing | +| R9-5 creation condition fails both directions | fixed — cause-aware read-free creation; policy-only tightening writes nothing; absence uses the exempt decision read | +| R9-6 suppression outside the authorization boundary | fixed — both constructors, fail-closed reads, retention rule, provenance-generation-fenced clearing | +| R9-7 N+1 new-shape S2S unsafe | fixed — context-free partner egress fails closed on provenance-less rows once new-shape config is active | +| R9-8 schema-floor protocol unspecified | fixed — dedicated primitive, create-or-CAS, read-back-then-enable, startup enforcement, fail-closed | +| R9-9 security-cookie exception reopens cookie surface | fixed — §4a typed owned-name operation, ts-\* rejected, sign-off 23 | +| R9-10 DataDome representation/ordering conflicts | fixed — decision-scoped representation; one global order, invariant last | +| R9-P2 batch (residual wording; deferred residue markers; registry + non-hex tags; cluster cap; retirement evidence; matrix/fixture branching; cutoff removed with adoption; irreversible artifacts enumerated; mixed-policy absolute removed; version pin; effect attribution; Set-Cookie remnants; network split) | all fixed | +| R9-P3 / R10-P2.8 eligibility rows (HEAD, 1xx, 204, 205, 206) | fixed — HEAD mirrors GET; others enumerated No | +| R10-1 future-dated TCF replay | fixed — beyond-skew rejected as malformed; digest-pinned first normalization | +| R10-2 mutable rows vs eventual/accretive claim | fixed — generation-CAS mutation, eventual visibility only | +| R10-3 cluster refresh extends retention | fixed — absolute `expires_at`, remaining-lifetime writes | +| R10-4 mint version inside replaceable snapshot | fixed — immutable mint tag split from evidence | +| R10-5 suppression needs forbidden read | fixed — read-free signal causes; narrow exempt decision read for absence | +| R10-6 durability/retention not validated | fixed — per-class durability + max-retention capabilities, startup horizon proof | +| R10-7 CDN-header weakening | fixed — CDN cache fields reserved outright; stale-\* durations shrink-only | +| R10-8 DataDome request-header injection | fixed — direction-scoped allowlist, scoped upstream overlay | +| R10-9 DataDome identifier outside permission model | ratification — sign-off 23 | +| R10-P2 batch (semantic digests; malformed/absence clearing + sign-off 24; atomic security batches; field registry; 256-byte bound in normative spec; device sign-off 22; provider-vs-schema rollback; TOCTOU live-check) | all fixed | From bf684e5d4a56756ed70c0e3188353062a587abf5 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Sat, 1 Aug 2026 22:41:14 -0700 Subject: [PATCH 12/24] Address eleventh review: suppression recovery, authoritative rowless handling, and the closed DataDome contract P1 fixes: - Suppression cannot deadlock its own recovery: AuthorityRefresh is a permission-exempt write path strictly scoped to committing provenance from the current live resolution while suppression stays effective; the clear then references that provenance's application-level monotonic revision (backend generation markers detect change and carry no order - per Fastly's own contract - so revisions are app-level counters). - Rowless classification is safe: it activates only under a deployment-metadata graphless-migration flag with a strongest-read existence check; otherwise (and on any read error) the state is indeterminate - no identity use, no mint, no cookie expiry. Rowless withdrawal writes nothing: there is no server-side state to revoke, and the prefix-derived family record is removed - it let unauthenticated suffix variants mint records and, because the HMAC prefix is per-IP, would have revoked every identity behind one IP. Family derivation is now one rule everywhere (full graph key, row-backed identities only). - The absence decision reads a strong record: the per-family record is now the authority-state record, carrying a per-permission positive-authority summary CAS-updated by every provenance write - never the eventual identity row, whose stale not-found loses the fresh-grant race. - The concrete adapter matrix gains cells for every mandatory capability (suppression CAS, row-mutation CAS, create-if-absent, deployment metadata, durability/retention) across all four adapters; the accretive claim is deleted (eventual visibility only after a generation-CAS mutation); Axum storage-dependent cells read Unavailable until a store exists. - The suppression wire schema is complete (state, cause, source class, evidence/observation timestamp, referenced provenance revision, positive summary, CAS version, schema version). - generate returns GeneratedIdentity { id, mint_version } - core cannot otherwise record the immutable mint tag; provider/version is removed from the mutable snapshot in both specs. - The DataDome contradictions are resolved: X-DataDome-ClientID is positively enumerated (with the documented X-DataDome-* set) and applies to an owner-scoped upstream overlay, never the shared view, egress under sign-off 23; the cookie carve-out has a concrete lifecycle (name datadome, apex scope, Secure/SameSite=Lax, 13-month ceiling, 4 KiB, owner-only read, deletion always, withdrawal semantics = the open half of item 23, which is pending ratification, not ratified); ordering is one global order (core -> ordinary mutators -> security effects -> invariant pass) with the older DataDome doc marked superseded; challenge batches validate and budget before Respond commits, so rejection can still fail open to Continue. - Headers-only v1 is permission-neutral by construction: the field registry admits inert fields only - Link preload, Reporting- Endpoints/NEL, CSP reporting, and Refresh cause vendor contact and are rejected until permission-declared mutation exists; unknown fields are rejected entirely. P2/P3: exempt-read and AuthorityRefresh rows added to the enforcement inventory with field enumeration; TCF digests include LastUpdated while GPP/USP digest semantics only (genuine CMP renewal refreshes, replays do not); malformed/absence causes get observation timestamps with cross-source comparison rules; the 4.1 matrix gains the suppression condition (one malformed request denies later no-signal requests under granted - sign-off 24 expanded); graph-read errors are indeterminate, not absent; the rewrite/backfill retirement alternative is removed (full-lifetime quiet period only); GPP versions pin to a vendored registry snapshot file; 304 for processed representations gets a 304-safe metadata pass; CDN cache fields are enumerated by name (Surrogate-Control, CDN-Cache-Control, Cloudflare-CDN-Cache-Control, Edge-Control); deferred reservation/CAS material is bracketed informative; the verbatim-key comment covers every hmac version; Content-Language replaced by true singletons; docs/superpowers/specs/ decisions/ created as the sign-off table's decision-record home. Ledger: R9-3, R10-2, R10-4, and GPP-pinning dispositions corrected to partial-then-refixed; full Round 11 section added. --- ...integration-response-header-hook-design.md | 128 +++++++++++------- .../2026-07-30-permission-model-design.md | 92 +++++++++---- .../2026-07-30-pluggable-providers-design.md | 93 ++++++++----- ...07-30-provider-migration-rollout-design.md | 66 ++++----- docs/superpowers/specs/decisions/README.md | 6 + docs/superpowers/specs/pr986-review-ledger.md | 67 +++++---- 6 files changed, 285 insertions(+), 167 deletions(-) create mode 100644 docs/superpowers/specs/decisions/README.md diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index cee462fe4..97dd5d484 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -75,8 +75,11 @@ mutators to the outbound response for HTML document responses it processed. only shrink relative to the snapshot; `stale-while-revalidate`/ `stale-if-error` may appear only if the snapshot had them **and their durations may only shrink** (present-at-1s must not become - present-at-1y); CDN-specific cache fields (`Surrogate-Control`, - `CDN-Cache-Control`, host equivalents) are **reserved outright** — + present-at-1y); CDN-specific cache fields are **reserved outright, by enumerated + name in the field registry** — `Surrogate-Control`, + `CDN-Cache-Control`, `Cloudflare-CDN-Cache-Control`, and + `Edge-Control`, each individually tested ("host equivalents" was not + a matching rule four adapters would implement identically) — merging them per-directive on unrestricted responses was a hole (an unrestricted `CDN-Cache-Control: max-age=60` could become a year), and they are additionally stripped from any restricted response; and the @@ -131,12 +134,18 @@ mutators to the outbound response for HTML document responses it processed. hook. - For non-reserved headers, the mutator API distinguishes **append** from **replace** explicitly; append/replace legality comes from a **core-owned field registry**, - not adapter judgment: each known field is classified - append-legal (genuinely list-valued: `Link`, CSP report groups, …), - replace-only (singletons: `Content-Language`, …), or rejected; - **unknown extension fields reject append by default** (replace only) — - "genuinely list-valued" is not a decision four adapters can make - independently and identically (`Set-Cookie` is fully reserved in v1 — neither append nor replace). Replacing a + not adapter judgment — and the v1 registry admits **inert fields + only**: "headers-only" is not automatically permission-neutral, since + `Link` (preload/prefetch), `Reporting-Endpoints`/NEL, CSP report + directives, and `Refresh` cause browser-initiated vendor contact on + requests that granted nothing. Fields with active egress side effects + are **rejected in v1**; a follow-up may admit them behind declared + required permissions gated at mutation time. Within the inert set, + each field is classified append-legal (genuinely list-valued), + replace-only (true singletons — e.g. `Content-Location`, `Retry-After`; + an earlier draft miscited `Content-Language`, which is list-valued), + or rejected; **unknown extension fields are rejected entirely in v1** + (neither append nor replace — their side-effect class is unknowable) (`Set-Cookie` is fully reserved in v1 — neither append nor replace). Replacing a header the origin set is a deliberate act, visible in the mutator's code. - Later registrations see earlier mutations (order = registration order, which is deterministic). @@ -163,8 +172,11 @@ mutators to the outbound response for HTML document responses it processed. values are withheld). Operations arrive as **attributed batches bound to a registration ID** — one batch per integration per response, ordered by - registration, with the security channel's batch (§4a) ordered before - response mutators; the current flat effects vector satisfies neither + registration, with the security channel's batch (§4a) ordered **after** + ordinary response mutators — one global order, core finalization → + ordinary mutators → security effects → invariant pass — so the + security layer's precedence over publisher-facing mutations holds + without a second ordering claim; the current flat effects vector satisfies neither attribution nor budgets and is restructured accordingly. Validation and budgeting are **atomic per batch**: a batch that exceeds its budget is rejected whole (logged, attributed), never partially @@ -184,17 +196,17 @@ mutators to the outbound response for HTML document responses it processed. Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Yes, applied at serve time (mutations are not cached) | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` | No | -| `HEAD` of a processed document | **Yes — header parity with GET is mandatory** (a cache may refresh stored GET metadata from HEAD, and divergent CSP/privacy metadata between the two is a leak) | -| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | +| Response | Hook runs? | +| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Yes, applied at serve time (mutations are not cached) | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` for a processed representation | **304-safe metadata pass**: the hook's header mutations for the corresponding processed 200 are re-applied (a 304 updates stored `Cache-Control`/`Vary` — excluding it while running on HEAD contradicted the cache-metadata rationale); where mutations cannot be reproduced, respond 200 instead | +| `HEAD` of a processed document | **Yes — header parity with GET is mandatory** (a cache may refresh stored GET metadata from HEAD, and divergent CSP/privacy metadata between the two is a leak) | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). @@ -204,43 +216,59 @@ processed documents (§6). The security channel (today: DataDome) is not a general exception; every degree of freedom is closed: -- **Typed security-cookie operation, not header strings.** The channel - emits cookies only through a typed operation whose cookie **names come - from the integration's registered ownership list** (for DataDome, its - documented cookie); every `ts-*` name is rejected; domain/path scope, - attributes, size, and lifetime are constrained by the registration. - Read/vendor-egress/withdrawal semantics of the resulting identifier - are a ratified security-purpose carve-out — **sign-off item 23** — - because the tag-injection → cookie/ClientID read → vendor-send - lifecycle otherwise hands a permission-denied visitor a stable, - exported identifier. No other request filter inherits the cookie - capability. -- **Request-header pointers are direction-scoped allowlists.** Values a - security response names for copying into the request (DataDome's - header-pointer mechanism) are accepted only from a **documented - enrichment-header allowlist**; authentication, `Cookie`, - `Forwarded`/`X-Forwarded-*`, identity, consent, and routing-authority - fields are rejected by name and by class — a compromised endpoint must - not replace origin credentials, inject `ts-ec`, or spoof client - location — and accepted values apply to a **narrowly scoped upstream - overlay**, never the shared request that later integrations read. +- **Typed security-cookie operation with a concrete lifecycle, not + header strings.** The channel emits cookies only through a typed + operation, and the registration is not a placeholder — for DataDome + it pins: cookie name exactly `datadome`; scope the publisher apex, + path `/`; mandatory `Secure` and `SameSite=Lax`; lifetime at most + DataDome's documented maximum (thirteen months ceiling); size ≤ 4 KiB; + a violating operation is rejected whole (the batch rule). Every + `ts-*` name is rejected. **Read is owner-only** — the cookie is + visible to the security channel and stripped from every other + integration's request view; vendor egress goes only to DataDome + endpoints; deletion is always possible; and whether TS's own + destructive withdrawal also expires it is exactly the open half of + **sign-off item 23** — the carve-out is _pending ratification_, not + ratified, and the permission inventory's cookie deferral stands until + it closes. No other request filter inherits the cookie capability. +- **Request-header pointers are a positive, enumerated allowlist.** + "Documented enrichment headers" is not enforceable; the registration + enumerates the exact names — for DataDome today that is + **`X-DataDome-ClientID` and the documented `X-DataDome-*` enrichment + set, listed one by one** — resolving what was a contradiction: + ClientID propagation is required by the existing DataDome contract + and test, and its identity-class nature is precisely why it applies + only to an **owner-scoped publisher-upstream overlay**, never the + shared request that later integrations read, with its vendor egress + ratified under sign-off 23. Everything else — authentication, + `Cookie`, `Forwarded`/`X-Forwarded-*`, other identity, consent, and + routing-authority fields — is rejected by name and by class: a + compromised endpoint must not replace origin credentials, inject + `ts-ec`, or spoof client location. - **Representation rules are decision-scoped.** A _Respond_ decision (challenge/deny) **owns its body** and may set representation headers (`Content-Type`, encoding, validators) for it — the hook's representation reservation exists because ordinary mutators do not own the body, and this one does. A _Continue_ decision may not touch representation metadata of publisher bytes. -- **One global order, no "wins" exception:** core finalization → - hook/security effects → **final cache/privacy invariant pass, - unconditionally last**. The prior DataDome contract's "applies last - and wins" holds only _within_ the effects layer; nothing outranks the - invariant pass, or a challenge could combine `Set-Cookie` with public - caching. +- **One global order:** core finalization → ordinary mutators → + security effects → **final cache/privacy invariant pass, + unconditionally last**. Security precedence over publisher-facing + mutations comes from its position, not a "wins" rule; nothing outranks + the invariant pass, or a challenge could combine `Set-Cookie` with + public caching. The older DataDome spec's "applies last, after + finalization" wording is **superseded by this order** — updating that + document is a done-when item, since as written it would place DataDome + after the invariant pass and reopen the public-cache-plus-cookie bug. - The channel adopts the shared layers: structured attributed batches (§3, atomic per batch — a 302 must never lose `Location` to a budget - while keeping its cookie; on rejection the channel follows DataDome's - specified fail-open), reserved header names, budgets, and the - invariant pass. + while keeping its cookie), reserved header names, budgets, and the + invariant pass — with one sequencing rule fail-open depends on: the + complete challenge batch is **validated and budgeted before the + Respond decision commits**, so a rejection converts to Continue while + the publisher route is still available; discovering the rejection + after Respond has short-circuited routing would leave nothing to fail + open _to_. ## 4. Done-when (from #782, sharpened) diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 360a3f2c1..f049b38c6 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -332,13 +332,13 @@ defined over those states, so no input state is unmapped. For each enforced permission, with baseline _B_ ∈ {granted, requires_signal, denied}: -| Opt-out present | TCF refusal present | Accepted grant present (regime-scoped) | Malformed-present | Result | -| --------------- | ------------------- | -------------------------------------- | ----------------- | ------------------------------------------------ | -| yes | — | — | — | **unset** (and withdrawal semantics apply, §4.2) | -| no | yes | — | — | unset (withdrawal per §4.2, trigger 2) | -| no | no | yes | — | set, unless B = denied | -| no | no | no | yes | **unset** (precedence 5 — blocks baseline grant) | -| no | no | no | no | set iff B = granted | +| Opt-out present | TCF refusal present | Accepted grant present (regime-scoped) | Malformed-present | Result | +| --------------- | ------------------- | -------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| yes | — | — | — | **unset** (and withdrawal semantics apply, §4.2) | +| no | yes | — | — | unset (withdrawal per §4.2, trigger 2) | +| no | no | yes | — | set, unless B = denied | +| no | no | no | yes | **unset** (precedence 5 — blocks baseline grant) | +| no | no | no | no | set iff B = granted **and no suppression entry stands** — an active suppression (§4.3) beats the baseline, so one malformed request denies later no-signal requests under `granted` until newer valid grant evidence clears it; a policy-baseline grant alone does **not** clear non-user suppression (sign-off 24 covers this consequence) | ### 4.2 Withdrawal vs. absence @@ -451,7 +451,12 @@ and the fail-closed marker: entry's; ties resolve to the more restrictive state. So a delayed grant with `LastUpdated = 100` never clears a suppression whose refusal carried `200`, while a genuine re-consent at `300` does. The - transition table, by stored cause: **opt-out from a timestamp-less + transition table (causes without an intrinsic timestamp — malformed + records decode no `LastUpdated`, absence has no source — use their + **observation timestamp**, server receipt on the shared clock basis + within the skew window; cross-source comparison uses the authoritative + timestamp where one exists, else the observation timestamp, ties + restrictive), by stored cause: **opt-out from a timestamp-less source** — cleared only by a grant with an authoritative timestamp newer than the suppression's observation (sticky opt-out, sign-off 16); **TCF refusal** — cleared by any regime-accepted grant with newer @@ -466,22 +471,45 @@ and the fail-closed marker: malformed beyond the skew window; within it, the record's digest is stored with its **first normalized timestamp, which re-presentation never advances** — otherwise a future-dated TCF string replayed after - an opt-out would keep re-normalizing to "now" and clear it. Equality - digests are computed over the **canonical per-permission semantic - result** of §4.5 aggregation, not the raw encoding — two encodings - (or `N` vs explicit N/A) with the same meaning are the same evidence - and keep the original first-seen, so alternating equivalent values - cannot renew authority. - - **Boundary, retention, ordering.** Suppression is checked by **every - `AuthorizedIdentity` constructor** (both scopes), by pull sync's live - path, and by every S2S recompute — not only "partner egress" prose; a - suppression read failure **fails closed** like a revocation read - failure; retention must outlive the positive authority it masks - (providers spec durability/retention capability); and **clearing is - fenced on provenance visibility**: a clear entry records the - provenance generation it reflects, and S2S honors the clear only when - it can read that generation or newer — clearing first would expose the + an opt-out would keep re-normalizing to "now" and clear it. Equality is + **source-specific**: for GPP/USP the digest is the **canonical + per-permission semantic result** of §4.5 aggregation alone — two + encodings (or `N` vs explicit N/A) with the same meaning are the same + evidence and keep the original first-seen, so alternating equivalent + values cannot renew authority; for TCF the digest is the semantic + result **plus the authoritative `LastUpdated`** — a genuine CMP + renewal with unchanged purposes carries a newer `LastUpdated` and + legitimately refreshes authority age, which a semantics-only digest + would wrongly ignore. + + **Boundary, retention, ordering — without deadlocking recovery.** + Suppression is checked by **every `AuthorizedIdentity` constructor** + (both scopes), by pull sync's live path, and by every S2S recompute. + That gate plus clear-after-provenance would deadlock re-consent — + fresh P1 provenance cannot be written while P1 suppression blocks + `GraphOps`, and clearing first is forbidden — so recovery has its own + narrow write path: **`AuthorityRefresh`**, permission-exempt but + strictly scoped to committing provenance from the _current live + resolution_ (nothing else: no partner writes, no egress, no reads + beyond the row being refreshed) while suppression remains effective; + the clear then references that provenance's revision. Revisions are an + **application-level monotonic counter written with the row** — never + backend generation markers, which (per Fastly's own contract) only + detect change and carry no order — and S2S honors a clear only when it + can read that revision or newer; clearing first would expose the + _older_ positive snapshot through an eventual read. + + **The strong record carries positive-authority state too.** The + per-family record doubles as the **authority-state record**: alongside + negative entries it stores a per-permission positive-authority summary + (revision, evidence timestamp), CAS-updated by every provenance write. + The **absence decision reads this strong summary, never the eventual + identity row** — deciding "no prior authority" from an eventual + not-found loses the race where a just-committed grant is invisible on + a stale replica. A suppression/authority read failure **fails closed** + like a revocation read failure; retention must outlive the positive + authority it masks (providers spec durability/retention capability). + _older_ positive snapshot through an eventual read. **Write failure fails closed for the live request**, and the S2S residual is unbounded for a never-returning visitor (sign-off 11), with fault tests for @@ -596,9 +624,13 @@ fields grant nothing (their opt-outs still count, per step 2). opt-outs — a Texas (16) or Maryland (24) sale opt-out must not vanish. The implementation PR cross-checks this list against both the current decoder's section set and the official registry, and the accepted version per - section is **pinned normatively to the named IAB GPP registry - revision current at this spec's date (2026-08-01)** — "enumerated by - the implementation PR" was two-implementations-diverge territory; a + section is **pinned to a registry snapshot vendored into this + repository** — a checked-in file enumerating, per mapped section, the + accepted version(s), taken from the IAB registry at ratification (a + date is not an immutable identifier, and "enumerated by the + implementation PR" was two-implementations-diverge territory; the + vendored file is the single reproducible authority, and updating it + is a reviewed spec change); a mapped section carrying a version outside the pinned revision is treated as malformed-present (blocks grants, never withdraws — §4.4), not as absent. Adding a section or version is a @@ -774,6 +806,8 @@ Consumers of the resolved set in this epic: | Stored consent-state lookup (§4.4) | **exempt**, narrowly scoped | Determining `store-on-device` cannot require `store-on-device` | | Integration persistent response cookies | `store-on-device` (+ P4 where the cookie is an advertising identifier) | **Deferred with the hook's cookie surface** — the write-side gate alone was insufficient (read/use/forward/withdrawal unmodeled), so cookie operations ship only with the full model; this row and the client-cycle **page leg** (module injection gated on the provider's full declaration) join the inventory when their features do, and the §5.3 no-geo guard's consumer list grows with them | | Suppression-record writes (§4.3) | **exempt** | Clearing authority is protective, like revocation | + | Authority-state / suppression-decision read (§4.3) | **exempt**, narrowly scoped | Returns only family ID, per-permission authority summary, and suppression entries — no identity values, no partner data; a test proves nothing else escapes | + | `AuthorityRefresh` provenance write (§4.3) | **exempt**, strictly scoped | Commits current-live-resolution provenance only; enables suppression recovery without reopening `GraphOps` | With **no EC provider configured**, identity use fails closed: a cookie value present on the request never egresses anywhere — never vacuously @@ -786,7 +820,9 @@ Consumers of the resolved set in this epic: written at mint and replaced on later live requests — grant basis (which signal class granted, per permission), the evidence's **authoritative timestamp and `valid_until`** (per evidence class), - resolved jurisdiction, policy revision, and provider/version (providers + resolved jurisdiction, and policy revision — **not** provider/version, + which lives only in the immutable mint tag, or a post-rotation visit + would restamp a v1 identity as v2 (providers spec §6.1). Freshness is a **per-evidence-class contract**, because not every source carries a timestamp: diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index 5302a7251..f8ebc0ddf 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -205,7 +205,8 @@ pub trait EdgeCookieProvider { /// Canonical graph-key SUFFIX (bounded length, KV-safe). Core — not /// the provider — constructs the physical key (§6.3 key grammar), so /// cross-provider and cross-record-kind isolation is structural. - /// Sole exception: hmac v0 keys are the identifier verbatim. + /// Sole exception: hmac keys (every version) are the identifier + /// verbatim — the reserved legacy grammar. fn graph_key_suffix(&self, id: &EcId) -> GraphKeySuffix; /// Cluster capability: a literal byte prefix of the physical graph /// key, shared across identifiers minted from the same client @@ -213,10 +214,12 @@ pub trait EdgeCookieProvider { fn cluster_prefix(&self, id: &EcId) -> Option; /// Mint an identifier from request evidence. The one acquisition /// operation of the epic (server mint); failure means no identity - /// this request (§6.2). Lost from an earlier revision by editing - /// accident — its absence made the required gate → generate → - /// graph-commit sequence unimplementable. - fn generate(&self, input: &IdentityInput<'_>) -> Result>; + /// this request (§6.2). Returns the identifier WITH the active + /// configuration version — the immutable mint tag needs it and core + /// cannot reach into provider-specific configuration to learn it. + fn generate(&self, input: &IdentityInput<'_>) + -> Result>; + // GeneratedIdentity { id: EcId, mint_version: ProviderVersion } /// Cryptographic verification of a parsed identifier against request /// evidence. Recognition (`parse`) is not authentication; rowless /// handling (§5) requires this. Returns the matched configuration @@ -272,23 +275,34 @@ present `H.aaaaab`, `H.aaaaac`, … — every variant prefix-verifies, and an adopt path would mint a **separate durable row and family per variant**. Therefore: -- A recognized rowless cookie whose prefix verifies - (`verify → VerifiedIdentity`, carrying the matched version for - provenance) is **expired and replaced by a fresh mint through the - ordinary graph-backed path** (gate → `generate` → commit) when the - request's permissions allow one; continuity with the old identifier is - deliberately not preserved (migration matrix row 13, sign-off 21). A - cookie whose prefix does not verify (including the declared roaming - false-negative) is simply expired. -- **Rowless withdrawal cannot be a row/family-minting oracle**: for - rowless legacy HMAC cookies the derived family ID is a function of the - **authenticated 64-hex prefix only**, so every suffix variant maps to - the _same_ family — one family record withdraws them all, and - attacker-generated variants create nothing new. -- Read errors are still not "not found": a failed graph read means the - cookie is treated as absent this request, fail closed, no expiry - emitted (the row may exist). - declares this path. +- **"Rowless" requires an authoritative not-found, and only in + migration mode.** Identity-row visibility may be eventual, so a plain + not-found proves nothing — a just-minted row invisible on a stale + replica would classify its own cookie as rowless and expire/re-mint + it, forking the identity. The rowless path therefore activates only + when the deployment-metadata **graphless-migration flag** is set (set + by the §4.2 readiness step for deployments that actually ran + graphless; permanently-graphed deployments never classify anything + rowless), and the existence check uses the backend's strongest read. + Outside migration mode, or on any read error, the state is + **indeterminate**: no identity use, no mint, no cookie expiry — + "treated as absent" was the wrong contract, since absence feeds the + fresh-mint path. +- A verified rowless cookie (`verify → VerifiedIdentity`, carrying the + matched version) is **expired and replaced by a fresh mint through the + ordinary graph-backed path** when permissions allow; continuity is + deliberately not preserved (migration matrix row 13, sign-off 21). An + unverifiable cookie (including the declared roaming false-negative) is + simply expired. +- **Rowless withdrawal writes nothing** — there is no server-side state + to revoke: no row, no partner mappings, no S2S surface. The cookie is + expired, and that is the entire withdrawal. (An earlier prefix-derived + family record was over-engineering with two defects: unauthenticated + suffix variants could mint records, and — because the HMAC prefix is + per-IP — one visitor's withdrawal would have revoked every identity + behind the same IP. Family records exist only for row-backed + identities, derived from the full graph key, one derivation + everywhere.) **Egress is typed, not policed.** The inventory-and-denylist test (permission model spec §7) is a backstop, but conventions do not survive @@ -541,9 +555,15 @@ deadline, and fencing epoch; the **family revocation record** holds the family ID, revoked-at, triggering signal class (§4.5 destructive column), and a **family epoch** bumped on every revocation-state change (the client-cycle commit CAS is conditioned on it) — deliberately no identity -data, so it can outlive its members; the **suppression record** holds -per-permission suppression entries with timestamps (strong class, -permission-exempt writes, permission model spec §4.3); +data, so it can outlive its members; the **authority-state (suppression) record** holds, per permission: +state (`suppressed`/`cleared`), cause, source class, authoritative or +observation evidence timestamp, the **application-level provenance +revision** a clear references, and the positive-authority summary +(revision + evidence timestamp) — plus the record-level CAS version +counter and schema version; unknown-field and range validation apply +like every class (strong class, permission-exempt writes per the +permission spec's inventory; revisions are app-level counters because +backend generation markers detect change without ordering); the **rewrite transaction** holds source key, target key, copy point, state, and epoch; the **reservation** holds state, owner hash, lease epoch, outcome, and created-at (client-cycle spec). Field validation and @@ -611,7 +631,7 @@ Requirements: | Deployment metadata (schema floor) | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | - | Identity rows | Eventual consistency acceptable — rows are accretive, and the family-record check (strong, per above) governs use | + | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | Every record class additionally declares **durability and maximum retention**: a store passing the consistency check but capping TTLs @@ -632,14 +652,19 @@ Requirements: them is how a "yes" cell hides an unusable feature. Feature eligibility requires wired, not merely available: - | Capability | Fastly | Axum (dev) | Cloudflare | Spin | - | ----------------------------------------------------- | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | - | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | - | Prefix listing (cluster) | Yes (used today) | Yes | Yes (eventual) | _verify_ | - | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | Yes (in-process) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | - | Linearizable fenced CAS (reservations, alias/rewrite) | **Not currently available** — the client-cycle feature (deferred) would need it | Yes — in-process only: linearizable but **non-durable**, dev-eligibility only, not a production persistence claim | Durable Objects: possible, not wired | **No** | - | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | - | Device host evidence (JA4/H2) | Yes | No | No | No | + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | ----------------------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | + | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | + | Prefix listing (cluster) | Yes (used today) | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | + | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | + | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | + | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | + | Device host evidence (JA4/H2) | Yes | No | No | No | + | Suppression / authority-state CAS | Generation-marker conditional write: available, **wiring to verify** | **Unavailable** (no store wired) | Workers KV: **ineligible** (last-write-wins); Durable Objects: feasible, not wired | **Unavailable** | + | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Deployment metadata (write-once/CAS) | **Not wired** — needs a primitive distinct from the config store | **Unavailable** | DO: feasible, not wired | **Unavailable** | + | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 82fe5779b..eb0ffe8b5 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -367,8 +367,8 @@ global honoring of opt-out signals is unconditional. spec §5.2), raw-egress denials by path, tombstone family retries, legacy-reader hit rate, and cluster-fallback engagements. Two of these carry thresholds, not just ranges: legacy-reader hits at zero for a **quiet period no shorter than the - maximum cookie/row lifetime plus rollout skew** — or provable - rewrite/backfill completion — is the **retirement-readiness** bar for a + maximum cookie/row lifetime plus rollout skew** is the **only + retirement-readiness** bar for a legacy provider ("trending to ~zero" is not evidence; a yearly visitor is not churn), (rewrite-based backfill and its metrics left with the rewrite deferral). The telemetry set also includes: graph read/commit failures, @@ -435,32 +435,36 @@ global honoring of opt-out signals is unconditional. These are decisions this spec set makes that #838 had not already made (or made differently). **Implementation is blocked while any row is `open`**; -each row needs an owner, a status, and a link to its decision record — -an unratified row reverts to open, not to silently implemented. - -| # | Decision | Where | Owner | Status | -| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | -| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | -| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | -| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | maintainers + legal | open | -| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | -| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | -| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | -| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | -| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | -| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | -| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | -| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | -| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | -| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | -| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | -| 16 | Sticky opt-out for timestamp-less sources: a GPP/USP-only re-consent does not restore authority without a timestamped grant | permission §4.3 | maintainers + legal | open | -| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | maintainers + legal | open | -| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | maintainers + legal | open | -| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | maintainers + product | open | -| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | maintainers + product | open | -| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | maintainers + product | open | -| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | maintainers + legal | open | -| 23 | DataDome security exemption: tag injection, cookie/ClientID read, vendor egress, and cross-integration visibility operate outside the permission model as a ratified security-purpose carve-out with owned cookie names, scope, and withdrawal semantics | hook §4a; permission §7 | maintainers + legal | open | -| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky); opt-out stickiness applies only to opt-out causes | permission §4.3 | maintainers + legal | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | +each row needs an owner, a status, and a link to its decision record. +Decision records live as files under +`docs/superpowers/specs/decisions/` (one per row, `NN-title.md`, +recording the decision, the deciders, and the date) — the table links +them as rows close; an unratified row reverts to open, not to silently +implemented. + +| # | Decision | Where | Owner | Status | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | +| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | +| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | +| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | maintainers + legal | open | +| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | +| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | +| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | +| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | +| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | +| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | +| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | +| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | +| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | +| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | +| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | +| 16 | Sticky opt-out for timestamp-less sources: a GPP/USP-only re-consent does not restore authority without a timestamped grant | permission §4.3 | maintainers + legal | open | +| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | maintainers + legal | open | +| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | maintainers + legal | open | +| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | maintainers + product | open | +| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | maintainers + product | open | +| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | maintainers + product | open | +| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | maintainers + legal | open | +| 23 | DataDome security exemption: tag injection, cookie/ClientID read, vendor egress, and cross-integration visibility operate outside the permission model as a ratified security-purpose carve-out with owned cookie names, scope, and withdrawal semantics | hook §4a; permission §7 | maintainers + legal | open | +| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | maintainers + legal | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | diff --git a/docs/superpowers/specs/decisions/README.md b/docs/superpowers/specs/decisions/README.md new file mode 100644 index 000000000..fd3d0e1e2 --- /dev/null +++ b/docs/superpowers/specs/decisions/README.md @@ -0,0 +1,6 @@ +# PR #986 product-decision records + +One file per open row of the migration spec §8 sign-off table +(`NN-title.md`), recording the decision, the deciders, and the date. The +table links each record as its row closes; a row without a record here is +open, and implementation is blocked while any row is open. diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index 85a35e965..e58443d90 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -167,27 +167,46 @@ R9 (10 P1, 13 P2, 1 P3) and the same-head re-audit R10 (9 P1, 9 P2, 1 P3) are dispositioned together; R10's "previously open" list = R9's P1s, tracked once. -| Finding | Status | -| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | -| R9-1 trait cannot mint (`generate` lost in an R8 edit) | fixed — restored with failure semantics | -| R9-2 / R10-open rowless HMAC unauthenticatable suffixes | fixed — expire-and-re-mint, prefix-derived family, `VerifiedIdentity{version}` | -| R9-3 missing capability rows (suppression CAS, create-if-absent, floor metadata) + global visibility | fixed — distinct rows with per-adapter values (Fastly generation markers; Workers KV ineligible), globally observable revocation reads | -| R9-4 CAS orders arrival not recency | fixed — evidence-recency transition table with per-cause clearing | -| R9-5 creation condition fails both directions | fixed — cause-aware read-free creation; policy-only tightening writes nothing; absence uses the exempt decision read | -| R9-6 suppression outside the authorization boundary | fixed — both constructors, fail-closed reads, retention rule, provenance-generation-fenced clearing | -| R9-7 N+1 new-shape S2S unsafe | fixed — context-free partner egress fails closed on provenance-less rows once new-shape config is active | -| R9-8 schema-floor protocol unspecified | fixed — dedicated primitive, create-or-CAS, read-back-then-enable, startup enforcement, fail-closed | -| R9-9 security-cookie exception reopens cookie surface | fixed — §4a typed owned-name operation, ts-\* rejected, sign-off 23 | -| R9-10 DataDome representation/ordering conflicts | fixed — decision-scoped representation; one global order, invariant last | -| R9-P2 batch (residual wording; deferred residue markers; registry + non-hex tags; cluster cap; retirement evidence; matrix/fixture branching; cutoff removed with adoption; irreversible artifacts enumerated; mixed-policy absolute removed; version pin; effect attribution; Set-Cookie remnants; network split) | all fixed | -| R9-P3 / R10-P2.8 eligibility rows (HEAD, 1xx, 204, 205, 206) | fixed — HEAD mirrors GET; others enumerated No | -| R10-1 future-dated TCF replay | fixed — beyond-skew rejected as malformed; digest-pinned first normalization | -| R10-2 mutable rows vs eventual/accretive claim | fixed — generation-CAS mutation, eventual visibility only | -| R10-3 cluster refresh extends retention | fixed — absolute `expires_at`, remaining-lifetime writes | -| R10-4 mint version inside replaceable snapshot | fixed — immutable mint tag split from evidence | -| R10-5 suppression needs forbidden read | fixed — read-free signal causes; narrow exempt decision read for absence | -| R10-6 durability/retention not validated | fixed — per-class durability + max-retention capabilities, startup horizon proof | -| R10-7 CDN-header weakening | fixed — CDN cache fields reserved outright; stale-\* durations shrink-only | -| R10-8 DataDome request-header injection | fixed — direction-scoped allowlist, scoped upstream overlay | -| R10-9 DataDome identifier outside permission model | ratification — sign-off 23 | -| R10-P2 batch (semantic digests; malformed/absence clearing + sign-off 24; atomic security batches; field registry; 256-byte bound in normative spec; device sign-off 22; provider-vs-schema rollback; TOCTOU live-check) | all fixed | +| Finding | Status | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | +| R9-1 trait cannot mint (`generate` lost in an R8 edit) | fixed — restored with failure semantics | +| R9-2 / R10-open rowless HMAC unauthenticatable suffixes | fixed — expire-and-re-mint, prefix-derived family, `VerifiedIdentity{version}` | +| R9-3 missing capability rows (suppression CAS, create-if-absent, floor metadata) + global visibility | R9/R10 partial (abstract rows only — the concrete matrix had no cells) → **refixed R11**: concrete per-adapter cells for all five primitives | +| R9-4 CAS orders arrival not recency | fixed — evidence-recency transition table with per-cause clearing | +| R9-5 creation condition fails both directions | fixed — cause-aware read-free creation; policy-only tightening writes nothing; absence uses the exempt decision read | +| R9-6 suppression outside the authorization boundary | fixed — both constructors, fail-closed reads, retention rule, provenance-generation-fenced clearing | +| R9-7 N+1 new-shape S2S unsafe | fixed — context-free partner egress fails closed on provenance-less rows once new-shape config is active | +| R9-8 schema-floor protocol unspecified | fixed — dedicated primitive, create-or-CAS, read-back-then-enable, startup enforcement, fail-closed | +| R9-9 security-cookie exception reopens cookie surface | fixed — §4a typed owned-name operation, ts-\* rejected, sign-off 23 | +| R9-10 DataDome representation/ordering conflicts | fixed — decision-scoped representation; one global order, invariant last | +| R9-P2 batch (residual wording; deferred residue markers; registry + non-hex tags; cluster cap; retirement evidence; matrix/fixture branching; cutoff removed with adoption; irreversible artifacts enumerated; mixed-policy absolute removed; version pin; effect attribution; Set-Cookie remnants; network split) | all fixed | +| R9-P3 / R10-P2.8 eligibility rows (HEAD, 1xx, 204, 205, 206) | fixed — HEAD mirrors GET; others enumerated No | +| R10-1 future-dated TCF replay | fixed — beyond-skew rejected as malformed; digest-pinned first normalization | +| R10-2 mutable rows vs eventual/accretive claim | R9/R10 partial (accretive claim survived in the matrix) → **refixed R11** (claim deleted; eventual visibility only after generation-CAS) | +| R10-3 cluster refresh extends retention | fixed — absolute `expires_at`, remaining-lifetime writes | +| R10-4 mint version inside replaceable snapshot | R9/R10 partial (mutable snapshot still listed provider/version) → **refixed R11** (removed from snapshot and from permission §7's field list) | +| R10-5 suppression needs forbidden read | fixed — read-free signal causes; narrow exempt decision read for absence | +| R10-6 durability/retention not validated | fixed — per-class durability + max-retention capabilities, startup horizon proof | +| R10-7 CDN-header weakening | fixed — CDN cache fields reserved outright; stale-\* durations shrink-only | +| R10-8 DataDome request-header injection | fixed — direction-scoped allowlist, scoped upstream overlay | +| R10-9 DataDome identifier outside permission model | ratification — sign-off 23 | +| R10-P2 batch (semantic digests; malformed/absence clearing + sign-off 24; atomic security batches; field registry; 256-byte bound in normative spec; device sign-off 22; provider-vs-schema rollback; TOCTOU live-check) | all fixed | + +## Round 11 — review at 43422b5f + +| Finding | Status | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| P1 suppression recovery deadlock | fixed — `AuthorityRefresh` scoped write path; clear references app-level provenance revision | +| P1 eventual not-found → rowless misclassification | fixed — rowless only under the graphless-migration deployment flag with strongest-read existence check; otherwise indeterminate (no use, no mint, no expiry); rowless withdrawal writes nothing (prefix-family removed — it also would have revoked whole IPs) | +| P1 absence suppression misses fresh grants | fixed — authority-state record carries a strong positive-authority summary; absence decision never reads the eventual row | +| P1 concrete matrix omits mandatory capabilities | fixed — see corrected R9-3 row above | +| P1 mutable vs accretive | fixed — see corrected R10-2 row above | +| P1 suppression wire schema + generation ordering | fixed — full field schema; app-level monotonic revisions (Fastly markers detect change, no order) | +| P1 `generate` cannot carry mint version | fixed — `GeneratedIdentity { id, mint_version }` | +| P1 ClientID required and prohibited | fixed — positively enumerated allowlist including `X-DataDome-ClientID`, owner-scoped upstream overlay, egress under sign-off 23 | +| P1 cookie exception without lifecycle | fixed — concrete registration (name `datadome`, scope, attributes, 13-month ceiling, 4 KiB, owner-only read, deletion always); "ratified" corrected to _pending ratification_ | +| P1 ordering contradiction | fixed — one order: core → mutators → security → invariant; older DataDome doc marked superseded, update in done-when | +| P1 headers-only not permission-neutral | fixed — v1 field registry admits inert fields only; active-egress fields rejected; unknown fields fully rejected | +| P2 batch (inventory rows for the exempt read and AuthorityRefresh; TCF-vs-GPP source-specific digests; observation timestamps; granted-baseline suppression row in §4.1 + sign-off 24 expanded; snapshot field cleanup; indeterminate read errors; Axum cells unavailable; rewrite/backfill retirement alternative removed; GPP vendored snapshot; 304-safe metadata pass; Respond-first validation; CDN names enumerated; deferred residue bracketed) | all fixed | +| P3 batch (verbatim comment covers every hmac version; `Content-Language` example corrected; stale fragments swept; this ledger corrected) | fixed | +| Ratification note | decisions directory created (`docs/superpowers/specs/decisions/`), table text points to it; item 23 wording no longer claims ratification | From ba25ba85c2b97877e2aa423b44ad7e8c443da10f Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Sun, 2 Aug 2026 15:24:12 -0700 Subject: [PATCH 13/24] Address twelfth review: two-record commit protocol, strong-class rowless proof, and bounded DataDome surfaces P1 fixes: - The two-record authority protocol has an explicit commit point: the identity row commits at revision r (generation-CAS), then the authority-state record CAS-updates to r, and r is usable - by S2S, by the absence decision, for egress eligibility - only when the strong record reports it. A crash between the writes is a recoverable intermediate (AuthorityRefresh re-runs step 2), never divergence. Mint eligibility begins at the authority-state commit, not the row commit. - Authority-state requires globally observable strong reads AND linearizable CAS - a stale successful read would authorize egress after a committed suppression, and read-failure-fails-closed does not cover stale successes; matrix and Fastly cells updated (read semantics to verify). - Rowless classification is proven from the strong class: every post-upgrade identity has an authority-state record under its derivable family ID, so rowless = graphless-migration flag AND no such record on a strong read - no eventual storage participates. The flag has a defined wire key, setter, capability requirement, rollback survival, and explicit operator clearing. - Rowless withdrawal is one contract everywhere: an exact-cookie family record (full-graph-key derivation - no per-IP blast) written by prefix-verified cookies only (attackers can spend withdrawal only on their own prefix), then cookie expiry - aligned with the family-record-first rule and migration row 13; the prefix mechanism is gone from every document, and cookie-only best-effort withdrawal (lost response = live cookie) is rejected. - N+1 neither creates nor clears authority-state records: clearing requires the AuthorityRefresh fence over revision-bearing rows a v1 writer cannot produce. N+1 reads fully and fails closed; suppression persists through rollback and recovery waits for roll-forward - a declared protective limitation. N+1 still writes family revocations. - The graph-row table gains the provenance-revision field (init 1, monotonic u64, overflow is an error, CAS'd with row generation) and loses the provider/version leftover from mutable provenance. - The positive-authority summary carries kind (user evidence vs policy baseline), grant basis/source class, policy revision, and valid_until - the absence decision distinguishes vanished user evidence from policy-only change without touching the eventual row. - The DataDome request-header allowlist is a checked-in file (datadome-header-allowlist.md) pinned to X-DataDome-ClientID alone; the cookie strip inventory is exhaustive (origin forwarding, proxy/click/Testlight upstreams, auction serialization, logs - each a tested row), not just integration views. - The 304 pass re-emits the persisted final post-hook header set stored with the cached representation; absent metadata means cache miss - the where-does-the-200-come-from gap is closed. P2/P3: gpp-registry-snapshot.md vendored (sections 6-27, versions, ratification re-verification note); the v1 field registry is enumerated in-spec; the datadome cookie pins PSL-computed registrable Domain and Max-Age <= 34,214,400 s; the clock-skew window is a normative 300 s constant; sign-off 23 is an open question enumerating observers, not 'ratified'; the permission spec repeats the globally-observable revocation wording verbatim instead of paraphrasing; rewrite/reservation wire schemas are bracketed informative and rewrite links leave the migration expansion; the dangling duplicate sentence, the reserved- cookie-name phrasing, and the negative-authority-only key-table description are gone; and the ledger adds Round 12 with a mechanical-anchor rule so closure claims are greppable rather than trusted. --- ...integration-response-header-hook-design.md | 64 ++++--- .../2026-07-30-permission-model-design.md | 49 +++++- .../2026-07-30-pluggable-providers-design.md | 159 ++++++++++-------- ...07-30-provider-migration-rollout-design.md | 120 +++++++------ .../specs/datadome-header-allowlist.md | 10 ++ .../specs/gpp-registry-snapshot.md | 35 ++++ docs/superpowers/specs/pr986-review-ledger.md | 30 ++++ 7 files changed, 309 insertions(+), 158 deletions(-) create mode 100644 docs/superpowers/specs/datadome-header-allowlist.md create mode 100644 docs/superpowers/specs/gpp-registry-snapshot.md diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 97dd5d484..3d5caa061 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -126,8 +126,8 @@ mutators to the outbound response for HTML document responses it processed. which corrupts attribution and budgets), with a duplicate-ID test in the done-when. Until then the operation set is headers-only, and `Set-Cookie` is fully reserved. - reserved cookie name — in v1 that is every cookie name, since - `Set-Cookie` is fully reserved (§3 deferral). Violations are rejected + cookie via any operation — `Set-Cookie` is fully reserved in v1 (§3 + deferral). Violations are rejected at the operation layer (§2) and logged at `warn` with the integration id. The reserved lists are single constants next to the definitions they protect, not duplicated in the @@ -145,7 +145,13 @@ mutators to the outbound response for HTML document responses it processed. replace-only (true singletons — e.g. `Content-Location`, `Retry-After`; an earlier draft miscited `Content-Language`, which is list-valued), or rejected; **unknown extension fields are rejected entirely in v1** - (neither append nor replace — their side-effect class is unknowable) (`Set-Cookie` is fully reserved in v1 — neither append nor replace). Replacing a + (neither append nor replace — their side-effect class is unknowable). + The v1 registry is enumerated here, not delegated: **admitted** — + `Cache-Control` (monotonic merge per this section), `Vary` (union + merge), `Content-Language` (append), `X-Robots-Tag` (append), + `Retry-After` (replace-only), `Content-Location` (replace-only); + everything else known is classified reserved or rejected by the rules + above, and growing the admitted set is a spec change to this list (`Set-Cookie` is fully reserved in v1 — neither append nor replace). Replacing a header the origin set is a deliberate act, visible in the mutator's code. - Later registrations see earlier mutations (order = registration order, which is deterministic). @@ -196,17 +202,17 @@ mutators to the outbound response for HTML document responses it processed. Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Yes, applied at serve time (mutations are not cached) | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` for a processed representation | **304-safe metadata pass**: the hook's header mutations for the corresponding processed 200 are re-applied (a 304 updates stored `Cache-Control`/`Vary` — excluding it while running on HEAD contradicted the cache-metadata rationale); where mutations cannot be reproduced, respond 200 instead | -| `HEAD` of a processed document | **Yes — header parity with GET is mandatory** (a cache may refresh stored GET metadata from HEAD, and divergent CSP/privacy metadata between the two is a leak) | -| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | +| Response | Hook runs? | +| ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Yes, applied at serve time (mutations are not cached) | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation; a 304 re-emits those persisted finals — deterministic, no mutator re-run, no representation reconstruction. If no persisted metadata exists, the conditional request is treated as a **cache miss** (full 200 fetched and processed); the earlier "respond 200 instead" without saying whence was not implementable | +| `HEAD` of a processed document | **Yes — header parity with GET is mandatory** (a cache may refresh stored GET metadata from HEAD, and divergent CSP/privacy metadata between the two is a leak) | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). @@ -219,13 +225,21 @@ degree of freedom is closed: - **Typed security-cookie operation with a concrete lifecycle, not header strings.** The channel emits cookies only through a typed operation, and the registration is not a placeholder — for DataDome - it pins: cookie name exactly `datadome`; scope the publisher apex, - path `/`; mandatory `Secure` and `SameSite=Lax`; lifetime at most - DataDome's documented maximum (thirteen months ceiling); size ≤ 4 KiB; + it pins: cookie name exactly `datadome`; `Domain` set to the + registrable domain computed against the **Mozilla Public Suffix List** + (vendored revision named by the implementation; host-only is not used + because DataDome requires site-wide scope), path `/`; mandatory + `Secure` and `SameSite=Lax`; `Max-Age` at most **34,214,400 seconds** + (396 days — the thirteen-month ceiling, as an exact number); size ≤ + 4 KiB; a violating operation is rejected whole (the batch rule). Every - `ts-*` name is rejected. **Read is owner-only** — the cookie is - visible to the security channel and stripped from every other - integration's request view; vendor egress goes only to DataDome + `ts-*` name is rejected. **Read is owner-only, and the strip inventory is exhaustive, not + integration-scoped** — the browser sends `datadome` in the ordinary + `Cookie` header, so it is removed from **every non-DataDome surface**: + other integrations' request views, publisher-origin proxy forwarding, + proxy/click/Testlight upstreams, auction/page-bids request + serialization, and logs (redaction list) — each surface a tested row + of the inventory; only the security channel itself observes it; vendor egress goes only to DataDome endpoints; deletion is always possible; and whether TS's own destructive withdrawal also expires it is exactly the open half of **sign-off item 23** — the carve-out is _pending ratification_, not @@ -233,9 +247,13 @@ degree of freedom is closed: it closes. No other request filter inherits the cookie capability. - **Request-header pointers are a positive, enumerated allowlist.** "Documented enrichment headers" is not enforceable; the registration - enumerates the exact names — for DataDome today that is - **`X-DataDome-ClientID` and the documented `X-DataDome-*` enrichment - set, listed one by one** — resolving what was a contradiction: + enumerates the exact names from the **checked-in allowlist file + `docs/superpowers/specs/datadome-header-allowlist.md`** — spec-pinned + today to exactly **`X-DataDome-ClientID`**; every other `X-DataDome-*` + field is rejected until a reviewed commit adds it to that file + ("documented enrichment set, listed one by one" without an actual list + was a wildcard whose contents could change outside the spec) — + resolving what was a contradiction: ClientID propagation is required by the existing DataDome contract and test, and its identity-class nature is precisely why it applies only to an **owner-scoped publisher-upstream overlay**, never the diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index f049b38c6..360df42ad 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -502,7 +502,15 @@ and the fail-closed marker: **The strong record carries positive-authority state too.** The per-family record doubles as the **authority-state record**: alongside negative entries it stores a per-permission positive-authority summary - (revision, evidence timestamp), CAS-updated by every provenance write. + — **kind** (user evidence vs. policy-baseline), grant basis / source + class, policy revision, `valid_until`, provenance revision, and + evidence timestamp — CAS-updated by every provenance write. The kind + and policy revision are load-bearing: the absence decision must + distinguish vanished _user_ evidence (suppress) from a + policy-baseline grant that disappeared because the _policy_ changed + (never suppress — trigger 3), and a revision-and-timestamp-only + summary would force exactly the eventual row read this record exists + to eliminate. The **absence decision reads this strong summary, never the eventual identity row** — deciding "no prior authority" from an eventual not-found loses the race where a just-committed grant is invisible on @@ -510,8 +518,23 @@ and the fail-closed marker: like a revocation read failure; retention must outlive the positive authority it masks (providers spec durability/retention capability). - _older_ positive snapshot through an eventual read. **Write failure - fails closed for the live request**, and the S2S residual is unbounded + **The strong record is the commit point — the two-record protocol is + explicit.** Every provenance-bearing write spans the eventual identity + row and the strong authority-state record, in a fixed order with + defined intermediate states: (1) the row commits at revision _r_ + (generation-CAS); (2) the authority-state record CAS-updates its + summary to _r_. **Revision _r_ is committed — usable by S2S, visible + to the absence decision — only when the strong record reports it**; a + row at _r_ whose summary still reads _r−1_ is simply uncommitted + detail, and a crash between the writes leaves a recoverable state (the + next live resolution re-runs step 2 via `AuthorityRefresh`), never a + divergent one. This ordering is why the absence decision can trust the + summary: there is no state in which the row authorizes something the + strong record has never heard of. Minting follows the same rule — + see the providers spec §5 order, where eligibility begins at the + **authority-state commit**, not the row commit. + + **Write failure fails closed for the live request**, and the S2S residual is unbounded for a never-returning visitor (sign-off 11), with fault tests for suppress-vs-clear races, repeated-value sequences, and the stale-provenance-read case. @@ -532,9 +555,12 @@ and the fail-closed marker: (migration spec §8), not a footnote. - **Consistency and retention are backend contracts with a single normative home**: the providers spec consistency matrix (§7). It — not - this spec — states the requirement, and it requires a **strongly - consistent (read-after-write) primitive** for revocation records; no - bounded-lag alternative exists (an earlier draft here permitted one, + this spec — states the requirement, and it requires **globally observable + strong consistency** for revocation records — every instance's read + observes a committed revocation, never merely the writer's own + session (this spec deliberately repeats the provider contract's exact + wording rather than paraphrasing it into the weaker "read-after-write"); + no bounded-lag alternative exists (an earlier draft here permitted one, which contradicted the matrix — an adapter with a two-second lag would have passed one spec and failed the other). A **failed family-record read fails closed** for egress (revoked-unknown ≠ live), and revocation @@ -624,8 +650,8 @@ fields grant nothing (their opt-outs still count, per step 2). opt-outs — a Texas (16) or Maryland (24) sale opt-out must not vanish. The implementation PR cross-checks this list against both the current decoder's section set and the official registry, and the accepted version per - section is **pinned to a registry snapshot vendored into this - repository** — a checked-in file enumerating, per mapped section, the + section is **pinned to the vendored registry snapshot + `docs/superpowers/specs/gpp-registry-snapshot.md`** — a checked-in file enumerating, per mapped section, the accepted version(s), taken from the IAB registry at ratification (a date is not an immutable identifier, and "enumerated by the implementation PR" was two-implementations-diverge territory; the @@ -832,7 +858,12 @@ Consumers of the resolved set in this epic: | GPP / USP values (no intrinsic timestamp) | **First-seen**: when TS first observed this exact normalized value (a **per-permission equality digest computed over only the applicable, aggregated §4.5 fields for that permission** — never the whole GPP record, or a CMP touching an unrelated notice field would mint a new digest and reset first-seen forever) | Re-presenting an identical digest **keeps the original first-seen**; a different value is new evidence with a new first-seen | Consent TTL (same as TCF) | | Policy-baseline grant (`granted` rule, no signal) | The policy revision that granted | Re-derived on every recompute against the current revision — policy is not user evidence and does not age; it changes | n/a | - Timestamps are compared with bounded clock-skew tolerance; + Timestamps are compared with a bounded clock-skew tolerance that is a + **normative constant — 300 seconds** (five minutes, applied + symmetrically; a spec-level value because suppression precedence, + malformed classification, and future-date handling all hinge on it, + and per-deployment values would give the same input different privacy + outcomes); beyond-window future-dated records are **rejected as malformed**, and within the window a record's first normalized timestamp is pinned to its digest and never advanced by re-presentation (§4.3's anti-replay diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index f8ebc0ddf..fbe362fcf 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -259,8 +259,13 @@ cookie write, no egress, no auction use may observe a minted identifier before its graph row (with provenance, §6.1) has committed — PR #838 let a generated EC reach an auction before finalization refused the cookie, producing an identity that existed for one request and nowhere else. The -normative order is: gate → `generate` → graph-row commit → cookie -scheduled → eligible for egress. "Cookie scheduled" means queued onto the +normative order is: gate → `generate` → graph-row commit → +**authority-state commit** (the strong record reporting the row's +revision — the commit point of the two-record protocol, permission model +spec §4.3) → cookie scheduled → eligible for egress. Eligibility begins +at the authority-state commit, not the row commit: a row whose revision +the strong record has not reported is uncommitted detail, which is what +keeps S2S authorization and the absence decision consistent. "Cookie scheduled" means queued onto the final response — `Set-Cookie` is physically emitted after first-request processing, so egress eligibility begins at **graph commit**, not at header emission; the identity exists durably from that moment. A @@ -275,34 +280,47 @@ present `H.aaaaab`, `H.aaaaac`, … — every variant prefix-verifies, and an adopt path would mint a **separate durable row and family per variant**. Therefore: -- **"Rowless" requires an authoritative not-found, and only in - migration mode.** Identity-row visibility may be eventual, so a plain +- **"Rowless" is proven from the strong class, never from eventual + storage.** Identity-row visibility may be eventual, so a plain not-found proves nothing — a just-minted row invisible on a stale - replica would classify its own cookie as rowless and expire/re-mint - it, forking the identity. The rowless path therefore activates only - when the deployment-metadata **graphless-migration flag** is set (set - by the §4.2 readiness step for deployments that actually ran - graphless; permanently-graphed deployments never classify anything - rowless), and the existence check uses the backend's strongest read. - Outside migration mode, or on any read error, the state is - **indeterminate**: no identity use, no mint, no cookie expiry — - "treated as absent" was the wrong contract, since absence feeds the - fresh-mint path. + replica would classify its own cookie as rowless and fork the + identity, and "the backend's strongest read" over eventual storage is + not an authoritative primitive. The proof uses what the protocol + already guarantees: **every post-upgrade identity has an + authority-state record** (the commit point, §5 mint order) under its + derivable family ID, in the globally-strong class — so _rowless_ = + the deployment-metadata **graphless-migration flag** is set AND the + strong read finds **no authority-state record** for the cookie's + derived family ID. Graphless-era cookies never had one; no eventual + read participates. The flag itself is specified: a named + deployment-metadata key (write-once/CAS class), set by the §4.2 + readiness step only on deployments that actually ran graphless + (requires the deployment-metadata capability), surviving binary + rollback, and **cleared by an explicit operator action** once the + migration window closes (quiet-period criterion in the guide) — + clearing ends rowless classification permanently. Outside the flag, + or on any read error, the state is **indeterminate**: no identity + use, no mint, no cookie expiry — "treated as absent" was the wrong + contract, since absence feeds the fresh-mint path. - A verified rowless cookie (`verify → VerifiedIdentity`, carrying the matched version) is **expired and replaced by a fresh mint through the ordinary graph-backed path** when permissions allow; continuity is deliberately not preserved (migration matrix row 13, sign-off 21). An unverifiable cookie (including the declared roaming false-negative) is simply expired. -- **Rowless withdrawal writes nothing** — there is no server-side state - to revoke: no row, no partner mappings, no S2S surface. The cookie is - expired, and that is the entire withdrawal. (An earlier prefix-derived - family record was over-engineering with two defects: unauthenticated - suffix variants could mint records, and — because the HMAC prefix is - per-IP — one visitor's withdrawal would have revoked every identity - behind the same IP. Family records exist only for row-backed - identities, derived from the full graph key, one derivation - everywhere.) +- **Rowless withdrawal writes an exact-cookie family record, then + expires the cookie** — one contract, aligned with the family-first + rule of the permission spec (cookie-only expiry would be best-effort: + a lost response leaves the "withdrawn" cookie usable on its next + presentation). The family ID uses the **same full-graph-key derivation + as row-backed identities** — one derivation everywhere — so the record + revokes exactly the presented cookie value: no per-IP blast (the + earlier prefix derivation would have revoked every identity behind one + IP), and bounded minting, because only **prefix-verified** cookies may + write one — an attacker can fabricate suffix variants only for their + own evidence's prefix, spending their own withdrawal on themselves. + A re-presented withdrawn variant finds its family record and stays + dead. **Egress is typed, not policed.** The inventory-and-denylist test (permission model spec §7) is a backstop, but conventions do not survive @@ -520,7 +538,7 @@ the bounded suffix: | Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | | Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | | Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | -| Suppression (negative authority) | `sup/` | Per-permission suppression entries + timestamps; permission-exempt writes; consulted by every S2S recompute and partner-egress check (permission spec §4.3) | +| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | | Rewrite transaction | `rwx/` | One in-flight rewrite per family | | Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | @@ -564,9 +582,9 @@ counter and schema version; unknown-field and range validation apply like every class (strong class, permission-exempt writes per the permission spec's inventory; revisions are app-level counters because backend generation markers detect change without ordering); -the **rewrite transaction** holds source key, target key, copy point, -state, and epoch; the **reservation** holds state, owner hash, lease -epoch, outcome, and created-at (client-cycle spec). Field validation and +the **rewrite transaction** _(informative — deferred with rewrite)_ +holds source key, target key, copy point, state, and epoch; the **reservation** _(informative — deferred with client-cycle)_ holds +state, owner hash, lease epoch, outcome, and created-at. Field validation and TTLs: aliases live to their retirement deadline; family records to the §7 retention rule (beyond every member, cookie, rewrite, and retry lifetime); transactions to completion plus an audit window; reservations @@ -581,25 +599,26 @@ readers round-trip unknown keys **semantically** (values preserved through read-modify-write; byte-identical output is not required and not achievable through a structured serializer). -| Field | Purpose | Source | Gating permission (egress) | TTL / refresh | Rewrite | On revocation | -| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | -| key (v1: identifier verbatim; v2: core-constructed, §4) | Row identity | Provider/core | — | Row TTL (1 y today) | New canonical row; old key becomes alias | Family record governs; member tombstone as cleanup | -| `v` | Schema discriminator | Core | — | — | Written at current version | Retained | -| `created` / **`expires_at`** | Row age and the **absolute retention deadline, pinned at mint** — every update writes with the _remaining_ lifetime, never a fresh full TTL (today's full-TTL rewrite lets a frequently visited identity live forever; refreshable derived state must never rejuvenate the identity) | Core | P1 (first-party ops) | Never extended | Preserved (no rejuvenation) | Retained in tombstone | -| `consent.tcf` / `consent.gpp` | Raw signal snapshot for audit; superseded as authority by provenance | Request | Never egressed to partners | Replaced on live resolution (§7 snapshot rule, permission spec) | Fresh live values | Scrubbed | -| `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | -| New: **immutable mint tag** (`mint_provider`, `mint_version`) | Credential retirement and audit — write-once at mint (legacy backfill may populate a missing tag once); **never part of the replaceable snapshot**, or a v1 identity revisited after rotation would be restamped v2 | Mint (or one-time backfill) | — | Immutable | — | Retained | -| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision, provider/version) | S2S authority | Live resolution only | Read by S2S recompute | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | -| New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | -| `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | -| `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | -| `pub_properties` (origin/seen domains) | Creation context | Core at mint | P1 | Write-once | Preserved | Scrubbed | -| `device.*` (JA4 class, H2 hash, quality metadata) | **Discontinued for new rows** (§5): fingerprint-derived, buyer-facing — beyond security-classification authorization. v1 rows retain them read-only; they are never egressed post-epic and are dropped at rewrite | Fastly device provider | None grants egress | Write-once (v1) | **Dropped** | Scrubbed | -| New: security classification outcome (boolean) | Bot-gate result | Device provider | — (never egressed) | Written at mint | Fresh | Scrubbed | -| `network.*` (immutable evidence: ASN etc.) | Cluster disambiguation | Platform at mint | P1 | Write-once | Fresh | Scrubbed | -| Derived cluster state (`cluster_size`, computed-at) | Trust gating | Computed | — | **Refreshable, short validity; generation-CAS update; never touches `expires_at`** | Recomputed | Scrubbed | -| `ids` (partner → UID map) | Partner identity graph | Pixel/pull/batch sync | P1 ∧ P4 (partner egress) | Per-mapping timestamps; bounded count/length | Copied **with original timestamps/expiry** | Scrubbed | -| New: alias record kind | Rewrite indirection (§6.1) | Core | — | Retirement deadline | Is the mechanism | Family-revoked like any member | +| Field | Purpose | Source | Gating permission (egress) | TTL / refresh | Rewrite | On revocation | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | +| key (v1: identifier verbatim; v2: core-constructed, §4) | Row identity | Provider/core | — | Row TTL (1 y today) | New canonical row; old key becomes alias | Family record governs; member tombstone as cleanup | +| `v` | Schema discriminator | Core | — | — | Written at current version | Retained | +| `created` / **`expires_at`** | Row age and the **absolute retention deadline, pinned at mint** — every update writes with the _remaining_ lifetime, never a fresh full TTL (today's full-TTL rewrite lets a frequently visited identity live forever; refreshable derived state must never rejuvenate the identity) | Core | P1 (first-party ops) | Never extended | Preserved (no rejuvenation) | Retained in tombstone | +| `consent.tcf` / `consent.gpp` | Raw signal snapshot for audit; superseded as authority by provenance | Request | Never egressed to partners | Replaced on live resolution (§7 snapshot rule, permission spec) | Fresh live values | Scrubbed | +| `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | +| New: **immutable mint tag** (`mint_provider`, `mint_version`) | Credential retirement and audit — write-once at mint (legacy backfill may populate a missing tag once); **never part of the replaceable snapshot**, or a v1 identity revisited after rotation would be restamped v2 | Mint (or one-time backfill) | — | Immutable | — | Retained | +| New: **provenance revision** (application-level monotonic counter, u64, initialized at 1, incremented by every provenance-bearing write, CAS'd with the row generation, serialized as an integer; overflow is a hard error, not a wrap) | Orders clears vs. snapshots (permission spec §4.3) | Core | Read by S2S/clears | Monotonic | — | Retained | +| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision, ) | S2S authority | Live resolution only | Read by S2S recompute | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | +| New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | +| `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | +| `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | +| `pub_properties` (origin/seen domains) | Creation context | Core at mint | P1 | Write-once | Preserved | Scrubbed | +| `device.*` (JA4 class, H2 hash, quality metadata) | **Discontinued for new rows** (§5): fingerprint-derived, buyer-facing — beyond security-classification authorization. v1 rows retain them read-only; they are never egressed post-epic and are dropped at rewrite | Fastly device provider | None grants egress | Write-once (v1) | **Dropped** | Scrubbed | +| New: security classification outcome (boolean) | Bot-gate result | Device provider | — (never egressed) | Written at mint | Fresh | Scrubbed | +| `network.*` (immutable evidence: ASN etc.) | Cluster disambiguation | Platform at mint | P1 | Write-once | Fresh | Scrubbed | +| Derived cluster state (`cluster_size`, computed-at) | Trust gating | Computed | — | **Refreshable, short validity; generation-CAS update; never touches `expires_at`** | Recomputed | Scrubbed | +| `ids` (partner → UID map) | Partner identity graph | Pixel/pull/batch sync | P1 ∧ P4 (partner egress) | Per-mapping timestamps; bounded count/length | Copied **with original timestamps/expiry** | Scrubbed | +| New: alias record kind | Rewrite indirection (§6.1) | Core | — | Retirement deadline | Is the mechanism | Family-revoked like any member | ## 7. Composition root and adapter parity @@ -621,17 +640,17 @@ Requirements: **per-record-class consistency requirements**, because "has KV" says nothing about whether revocation is observable: - | Record class | Required semantics | - | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | - | Family revocation records | **Globally observable strong consistency** — every instance's read observes a committed revocation, not merely the writing session's own writes (writer-scoped read-your-writes is insufficient for a fleet). Cloudflare Workers KV is **not eligible** — "60 seconds or more" is an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | - | Family suppression records | **Linearizable per-key CAS** — read-after-write alone cannot provide read-modify-write monotonicity: two writers both read, and an older clear overwrites a newer suppress | - | Identity-row mutation | **Generation CAS** (conditional write on row generation) with reread/recompute on conflict — rows are heavily mutable (snapshots replaced, partner IDs merged, derived state refreshed), so unordered last-writer-wins loses newer evidence and mappings; _visibility_ may stay eventual, unordered _mutation_ may not. Fastly KV offers generation-marker conditional writes; Workers KV's documented concurrent last-write-wins is ineligible for mutation-bearing rows | - | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | - | Deployment metadata (schema floor) | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | - | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | - | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | - | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | + | Record class | Required semantics | + | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | + | Family revocation records | **Globally observable strong consistency** — every instance's read observes a committed revocation, not merely the writing session's own writes (writer-scoped read-your-writes is insufficient for a fleet). Cloudflare Workers KV is **not eligible** — "60 seconds or more" is an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | + | Authority-state records (suppression + positive summary) | **Globally observable strong reads AND linearizable per-key CAS** — CAS alone orders writes, but a stale successful _read_ on another instance would authorize egress after a committed suppression ("read failures fail closed" does not cover stale successes); both properties are adapter eligibility gates | + | Identity-row mutation | **Generation CAS** (conditional write on row generation) with reread/recompute on conflict — rows are heavily mutable (snapshots replaced, partner IDs merged, derived state refreshed), so unordered last-writer-wins loses newer evidence and mappings; _visibility_ may stay eventual, unordered _mutation_ may not. Fastly KV offers generation-marker conditional writes; Workers KV's documented concurrent last-write-wins is ineligible for mutation-bearing rows | + | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | + | Deployment metadata (schema floor) | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | + | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | + | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | + | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | Every record class additionally declares **durability and maximum retention**: a store passing the consistency check but capping TTLs @@ -652,19 +671,19 @@ Requirements: them is how a "yes" cell hides an unusable feature. Feature eligibility requires wired, not merely available: - | Capability | Fastly | Axum (dev) | Cloudflare | Spin | - | ----------------------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | - | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | - | Prefix listing (cluster) | Yes (used today) | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | - | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | - | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | - | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | - | Device host evidence (JA4/H2) | Yes | No | No | No | - | Suppression / authority-state CAS | Generation-marker conditional write: available, **wiring to verify** | **Unavailable** (no store wired) | Workers KV: **ineligible** (last-write-wins); Durable Objects: feasible, not wired | **Unavailable** | - | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | - | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | - | Deployment metadata (write-once/CAS) | **Not wired** — needs a primitive distinct from the config store | **Unavailable** | DO: feasible, not wired | **Unavailable** | - | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | + | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | + | Prefix listing (cluster) | Yes (used today) | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | + | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | + | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | + | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | + | Device host evidence (JA4/H2) | Yes | No | No | No | + | Authority-state: global strong reads + CAS | Conditional writes available (generation marker); **globally current read semantics to verify** — both required, wiring to verify | **Unavailable** (no store wired) | Workers KV: **ineligible**; Durable Objects: feasible, not wired | **Unavailable** | + | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Deployment metadata (write-once/CAS) | **Not wired** — needs a primitive distinct from the config store | **Unavailable** | DO: feasible, not wired | **Unavailable** | + | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index eb0ffe8b5..474275616 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -32,30 +32,30 @@ and whether that is a preservation or a declared change. **Silent changes are defects.** PR #838 changed six of these without declaring any; each was discoverable only because a deleted test had pinned the old behavior. -| # | Decision (today) | After epic | Status | -| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | -| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | -| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | -| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | -| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | -| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | -| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | -| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — opt-out effects are more protective; the same fields' not-opted-out values can also **newly grant P4**, which is not (both effects classified in permission spec §4.5) | -| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | -| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | -| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | -| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | -| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | -| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | -| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | -| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | -| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | -| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | -| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | -| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | -| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | -| 13 | Cookies minted by graphless deployments have no graph row | Recognized rowless cookies are **expired and re-minted** through the ordinary graph-backed path (providers spec §5) — never adopted, since prefix-only verification cannot authenticate suffix variants; identity continuity is deliberately lost; withdrawal works without any row via the prefix-derived family ID | **Declared** — pre-existing identities restart rather than carry over | +| # | Decision (today) | After epic | Status | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | +| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | +| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | +| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | +| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | +| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — opt-out effects are more protective; the same fields' not-opted-out values can also **newly grant P4**, which is not (both effects classified in permission spec §4.5) | +| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | +| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | +| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | +| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | +| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | +| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | +| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | +| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | +| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | +| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | +| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | +| 13 | Cookies minted by graphless deployments have no graph row | Rowless proof via the strong authority-state class under the graphless-migration flag (providers spec §5); verified cookies are expired and re-minted without continuity; **rowless withdrawal writes an exact-cookie family record, then expires** — full-key derivation, no prefix mechanism | **Declared** — pre-existing identities restart rather than carry over | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -126,10 +126,19 @@ Requirements: revocations, honor suppression records, or fail closed on provenance; an N+1 that merely preserved would, after rollback, treat revoked identities as live (aliases are reserved-future with - the rewrite deferral, providers spec §6.1). N+1 must also **write** - the safety-critical record kinds — family revocation and - suppression — not only read them: a withdrawal arriving on a - rolled-back N+1 fleet must still revoke. + the rewrite deferral, providers spec §6.1). N+1 must also **write + family revocation records** — a withdrawal arriving on a + rolled-back N+1 fleet must still revoke. **Authority-state + (suppression) is different: N+1 neither creates nor clears it.** + Creating would be safe, but clearing now requires the + `AuthorityRefresh` provenance protocol over revision-bearing rows + that N+1 (a v1 writer) cannot produce — so an N+1 clearing without + the fence would expose stale positive snapshots, and one clearing + with it would need the whole N+2 write model. Instead: N+1 + **reads** authority-state fully and fails closed on suppressed + permissions; suppression created by N+2 stays in force during a + rollback, and **recovery (clearing) waits for roll-forward** — a + protective, declared limitation, not an undefined one. **N+1's identity-write behavior is v1, explicitly** — this resolves what was an impossible trilemma (write rows without provenance, @@ -210,8 +219,7 @@ Requirements: (matrix row 12), not a side effect discovered at boot. 4. **The graph schema change is expand-contract, in lockstep with the binary sequence.** New rows carry fields v1 rows never had — provider/ - version, per-permission grant evidence, policy revision, family ID, - rewrite links — and two failure modes must be engineered away: a naive + version, per-permission grant evidence, policy revision, family ID — and two failure modes must be engineered away: a naive schema-version bump makes old readers fail closed on new rows, and an old worker that reads, modifies, and reserializes a row **silently drops** fields it does not model. The sequence shares the config @@ -442,29 +450,29 @@ recording the decision, the deciders, and the date) — the table links them as rows close; an unratified row reverts to open, not to silently implemented. -| # | Decision | Where | Owner | Status | -| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | -| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | -| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | -| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | maintainers + legal | open | -| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | -| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | -| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | -| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | -| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | -| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | -| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | -| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | -| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | -| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | -| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | -| 16 | Sticky opt-out for timestamp-less sources: a GPP/USP-only re-consent does not restore authority without a timestamped grant | permission §4.3 | maintainers + legal | open | -| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | maintainers + legal | open | -| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | maintainers + legal | open | -| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | maintainers + product | open | -| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | maintainers + product | open | -| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | maintainers + product | open | -| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | maintainers + legal | open | -| 23 | DataDome security exemption: tag injection, cookie/ClientID read, vendor egress, and cross-integration visibility operate outside the permission model as a ratified security-purpose carve-out with owned cookie names, scope, and withdrawal semantics | hook §4a; permission §7 | maintainers + legal | open | -| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | maintainers + legal | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | +| # | Decision | Where | Owner | Status | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | +| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | +| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | +| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | maintainers + legal | open | +| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | +| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | +| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | +| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | +| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | +| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | +| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | +| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | +| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | +| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | +| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | +| 16 | Sticky opt-out for timestamp-less sources: a GPP/USP-only re-consent does not restore authority without a timestamped grant | permission §4.3 | maintainers + legal | open | +| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | maintainers + legal | open | +| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | maintainers + legal | open | +| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | maintainers + product | open | +| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | maintainers + product | open | +| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | maintainers + product | open | +| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | maintainers + legal | open | +| 23 | **Open question, not ratified**: may DataDome's security identifier (tag injection, `datadome` cookie, `X-DataDome-ClientID` read and vendor egress) operate outside the permission model? The decision must enumerate exactly which consumers may observe the cookie/ClientID — today's spec allows only the security channel itself and strips every other surface (hook §4a) — plus retention and whether TS withdrawal expires it | hook §4a; permission §7 | maintainers + legal | open | +| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | maintainers + legal | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | diff --git a/docs/superpowers/specs/datadome-header-allowlist.md b/docs/superpowers/specs/datadome-header-allowlist.md new file mode 100644 index 000000000..9acb8c225 --- /dev/null +++ b/docs/superpowers/specs/datadome-header-allowlist.md @@ -0,0 +1,10 @@ +# DataDome request-header allowlist (normative, checked-in) + +The complete set of response-named header pointers the security channel +(hook spec §4a) may copy into the owner-scoped publisher-upstream +overlay. Every `X-DataDome-*` name not listed here is rejected. Adding a +name is a reviewed commit to this file and a spec change. + +| Header | Direction | Scope | +| --------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------- | +| `X-DataDome-ClientID` | response → upstream overlay | Owner-scoped overlay only; never the shared request view; vendor egress governed by sign-off item 23 | diff --git a/docs/superpowers/specs/gpp-registry-snapshot.md b/docs/superpowers/specs/gpp-registry-snapshot.md new file mode 100644 index 000000000..103e8ca78 --- /dev/null +++ b/docs/superpowers/specs/gpp-registry-snapshot.md @@ -0,0 +1,35 @@ +# GPP registry snapshot (normative, vendored) + +The pinned per-section accepted versions for the permission spec's §4.5 +map. This file is the single reproducible authority; updating it is a +reviewed spec change. A mapped section presenting a version not listed +here is treated as malformed-present (permission spec §4.4). + +| GPP section ID | Section | Accepted version(s) | +| -------------- | --------------------------------------------------- | ------------------- | +| 6 | US Privacy string (uspv1, carried as a GPP section) | 1 | +| 7 | usnat | 1 | +| 8 | usca | 1 | +| 9 | usva | 1 | +| 10 | usco | 1 | +| 11 | usut | 1 | +| 12 | usct | 1 | +| 13 | usfl | 1 | +| 14 | usmt | 1 | +| 15 | usor | 1 | +| 16 | ustx | 1 | +| 17 | usde | 1 | +| 18 | usia | 1 | +| 19 | usne | 1 | +| 20 | usnh | 1 | +| 21 | usnj | 1 | +| 22 | ustn | 1 | +| 23 | usmn | 1 | +| 24 | usmd | 1 | +| 25 | usin | 1 | +| 26 | usky | 1 | +| 27 | usri | 1 | + +Version values were captured from the IAB registry at the time of +writing and are re-verified against the official registry as part of +ratification review; any correction is a change to this file. diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index e58443d90..f0c1291af 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -210,3 +210,33 @@ tracked once. | P2 batch (inventory rows for the exempt read and AuthorityRefresh; TCF-vs-GPP source-specific digests; observation timestamps; granted-baseline suppression row in §4.1 + sign-off 24 expanded; snapshot field cleanup; indeterminate read errors; Axum cells unavailable; rewrite/backfill retirement alternative removed; GPP vendored snapshot; 304-safe metadata pass; Respond-first validation; CDN names enumerated; deferred residue bracketed) | all fixed | | P3 batch (verbatim comment covers every hmac version; `Content-Language` example corrected; stale fragments swept; this ledger corrected) | fixed | | Ratification note | decisions directory created (`docs/superpowers/specs/decisions/`), table text points to it; item 23 wording no longer claims ratification | + +## Round 12 — review at bf684e5 + +The R11 rows this round showed as overstated (provider/version leftover +in the graph table, no GPP snapshot file, sign-off 23 wording, dangling +fragments) are hereby corrected below — and from this round on, ledger +"fixed" claims are **mechanically greppable**: each row's parenthetical +names an anchor phrase present in the tree, so `grep` can audit closure +instead of trusting prose. + +| Finding | Status | +| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| P1 two-record atomicity | fixed ("The strong record is the commit point": row commits at r, then authority-state CAS to r; S2S/absence use r only when the strong record reports it; mint eligibility begins at authority-state commit) | +| P1 suppression reads not globally current | fixed ("Globally observable strong reads AND linearizable per-key CAS" — matrix row and Fastly cell updated with reads-to-verify) | +| P1 rowless authoritative not-found | fixed ("proven from the strong class": no authority-state record under the derived family ID + graphless-migration flag; flag wire/lifecycle defined; no eventual read participates) | +| P1 rowless withdrawal three ways | fixed (one contract: "exact-cookie family record, then expires" — full-key derivation, verified-cookies-only writes; migration row 13 aligned; prefix mechanism gone from every spec) | +| P1 N+1 cannot run suppression recovery | fixed ("N+1 neither creates nor clears" authority-state; reads fail closed; clears wait for roll-forward — declared protective limitation) | +| P1 row schema missing revision / stale provider-version | fixed (provenance-revision field row added with init/overflow/CAS rules; provider/version stripped from the mutable provenance row) | +| P1 summary insufficient for policy-only rule | fixed (summary carries kind, grant basis/source class, policy revision, valid_until — absence decision reproducible from the strong record) | +| P1 DataDome allowlist not enumerated | fixed (checked-in `datadome-header-allowlist.md`, spec-pinned to `X-DataDome-ClientID` alone; other `X-DataDome-*` rejected) | +| P1 cookie confinement misses upstream/log surfaces | fixed (exhaustive strip inventory: origin forwarding, proxy/click/Testlight upstreams, auction serialization, logs — each a tested row) | +| P1 304 not implementable | fixed (persisted final post-hook header set re-emitted; absent metadata → cache miss) | +| P2 GPP snapshot missing | fixed (`gpp-registry-snapshot.md` vendored, sections 6–27, ratification re-verification note) | +| P2 field registry not enumerated | fixed (v1 admitted set enumerated in-spec; growth is a spec change) | +| P2 domain/lifetime irreproducible | fixed (PSL-computed registrable domain, vendored PSL revision, Max-Age ≤ 34,214,400 s) | +| P2 skew window unvalued | fixed (normative 300 s constant with rationale) | +| P2 sign-off 23 says ratified | fixed (rewritten as an open question enumerating observers) | +| P2 revocation wording paraphrase | fixed (permission spec repeats "globally observable" verbatim) | +| P2 deferred residue in normative schemas | fixed (rewrite transaction and reservation wire schemas bracketed informative; rewrite links out of the migration expansion item) | +| P3 dangling sentences / key-table description / ledger reliability | fixed (fragments removed; key table says authority-state with positive summary; this section's mechanical-anchor rule) | From 3523b362c0daf41184c2b52a19a55d92010158ff Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Sun, 2 Aug 2026 23:47:07 -0700 Subject: [PATCH 14/24] Address thirteenth review: coherent suppression expiry, provable rowless prerequisites, honest failure semantics P1 fixes: - Suppression entries carry their evidence class's valid_until and go inert at expiry (lazily GC'd) - resolving the contradiction where an expired TCF refusal under a granted baseline would deny forever while the normalization table promised the baseline grant; normalization wins. - Rowless classification gains real prerequisites: the flag may be set only after full N+2 convergence (an N+1 fleet still minting v1 rows never classifies rowless) and an idempotent stub-backfill has stamped an authority-state existence stub on every existing row - only then does 'no record' actually mean graphless-era. A per-deployment flag alone cannot prove a per-cookie fact. - The mint-path recovery claim is retracted: a failed authority-state commit after the row commit leaves an orphan no later request can find (no cookie was emitted); it authorizes nothing, expires by TTL, is counted, and has its own runtime-failure-matrix row. The eligibility-at-graph-commit leftover is swept. - The authority-state wire schema now carries every field the permission protocol consumes: negative entries with cause/source/ timestamps/valid_until/referenced revision; the positive summary with kind, grant basis, policy identity, valid_until, revision, evidence timestamp, and the semantic digest with pinned first-seen; plus the backfill stub marker. - Negative-record creation is admission-controlled (existing family on a strong read, or a verified identifier - fabrications write nothing), and rowless withdrawal collapses to one capped per-prefix record (8 suffix hashes; saturation escalates to prefix-wide rowless revocation as the declared abuse response, harming only the abuser's own same-IP graphless cohort) - closing the storage-amplification surface that per-variant exact-cookie records opened. - Embedded GPP GPC is mapped: Gpc=true in any section is the same destructive global opt-out as the header (OR-aggregated); GpcSegmentIncluded=false/absent contributes nothing; malformed GPC segments render the section malformed-present. Sign-off 26. - Batch S2S jurisdiction ages: stored jurisdiction older than the consent-TTL horizon fails closed pending a live refresh; the horizon and its two-sided trade-off are sign-off 25. - Observability sinks join the egress inventory: raw EC values never reach logs/traces/metrics/errors, logging boundaries take hash-only types, the existing PR #838 logging site is cited, and a log-schema denylist test enforces the row. - 304 persisted metadata is versioned by (integration-registry, config, invariant) revisions with mismatch = cache miss, so normal and conditional hits cannot serve different policy metadata; cache-relevant fields are defined. - The DataDome contract aligns with documented vendor behavior where hardening was not intended (configurable SameSite, one-year 31,536,000 s cap replacing the over-vendor 396-day figure, 512-byte size per the current Fastly module, Domain per vendor guidance PSL-validated); the deliberately reduced pointer allowlist requires product AND vendor acceptance (sign-off 28); and the publisher origin is named in sign-off 23 as a ClientID observer - the overlay is the mechanism, the row now names the recipient. P2/P3: deployment-metadata 'm' key class with the graphless flag's full wire lifecycle (N+2 + backfill attested in the value, operator CAS clearing); N+1 rollback tests aligned to read-and-fail-closed for authority state; AuthorityRefresh's access set enumerated (row CAS + authority-record CAS - the old wording forbade a read its own protocol needs); expired-live-plus-persisted fallback decided (expired live does not suppress fallback); the skew constant became an algorithm (beyond-window malformed, no clamping, expiry grace, within-window equality routing to the restrictive tie rule); policy revisions have canonical identity (content digest + activation generation); provider-code-registry.md (hmac allocated) and psl-snapshot-ref.md (ICANN+private, IDNA, IP/single-label host-only) created; the GPP decoder gap (sections 24-27, usnat-v2-decodable-but-unpinned) is an explicit prerequisite; sign-off rows 25-28 added; adapter qualification is a pre-ratification prerequisite; telemetry residue removed; the hook fragment, provenance comma, test host-equivalents, and client-draft Axum claim are fixed. Ledger: Round 13 adopts the added-vs-verified vocabulary - rows are 'text-added' until a subsequent review declines to reopen them; all R12 rows retroactively so marked. --- ...26-07-30-client-cycle-ec-resolve-design.md | 4 +- ...integration-response-header-hook-design.md | 53 ++++---- .../2026-07-30-permission-model-design.md | 105 +++++++++++----- .../2026-07-30-pluggable-providers-design.md | 117 +++++++++++------- ...07-30-provider-migration-rollout-design.md | 96 +++++++------- docs/superpowers/specs/pr986-review-ledger.md | 26 ++++ .../specs/provider-code-registry.md | 10 ++ docs/superpowers/specs/psl-snapshot-ref.md | 13 ++ 8 files changed, 287 insertions(+), 137 deletions(-) create mode 100644 docs/superpowers/specs/provider-code-registry.md create mode 100644 docs/superpowers/specs/psl-snapshot-ref.md diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index eda8268eb..d6a5a487d 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -100,8 +100,8 @@ Everything in this spec follows from that. 5. **Exist on every adapter — where parity means identical behavior, including identical refusal.** Route registration goes through shared route wiring. On adapters whose capability matrix rows are green - (today only the dev adapter has the required CAS class — providers - spec §7), the parity suite asserts identical endpoint behavior; on + (today **no adapter** has the required CAS class — the normative + matrix marks even Axum's storage unavailable; providers spec §7), the parity suite asserts identical endpoint behavior; on adapters without them, parity means **identical startup rejection of the client-cycle selection** — not a proxied 404 (PR #838's failure mode: Fastly-only registration let the Axum dev server proxy the POST diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 3d5caa061..c290b0c3d 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -126,8 +126,8 @@ mutators to the outbound response for HTML document responses it processed. which corrupts attribution and budgets), with a duplicate-ID test in the done-when. Until then the operation set is headers-only, and `Set-Cookie` is fully reserved. - cookie via any operation — `Set-Cookie` is fully reserved in v1 (§3 - deferral). Violations are rejected + `Set-Cookie` is fully reserved in v1 (§3 deferral). Violations are + rejected at the operation layer (§2) and logged at `warn` with the integration id. The reserved lists are single constants next to the definitions they protect, not duplicated in the @@ -202,17 +202,17 @@ mutators to the outbound response for HTML document responses it processed. Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Yes, applied at serve time (mutations are not cached) | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation; a 304 re-emits those persisted finals — deterministic, no mutator re-run, no representation reconstruction. If no persisted metadata exists, the conditional request is treated as a **cache miss** (full 200 fetched and processed); the earlier "respond 200 instead" without saying whence was not implementable | -| `HEAD` of a processed document | **Yes — header parity with GET is mandatory** (a cache may refresh stored GET metadata from HEAD, and divergent CSP/privacy metadata between the two is a leak) | -| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | +| Response | Hook runs? | +| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Yes, applied at serve time (mutations are not cached) | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by (integration-registry revision, config revision, invariant revision)**; a 304 re-emits those persisted finals only when all three match the serving instance — a mismatch (integration or config changed since caching) is a **cache miss**, so a normal hit and a conditional hit can never return different policy metadata for one representation. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | +| `HEAD` of a processed document | **Yes — header parity with GET is mandatory** (a cache may refresh stored GET metadata from HEAD, and divergent CSP/privacy metadata between the two is a leak) | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). @@ -225,13 +225,22 @@ degree of freedom is closed: - **Typed security-cookie operation with a concrete lifecycle, not header strings.** The channel emits cookies only through a typed operation, and the registration is not a placeholder — for DataDome - it pins: cookie name exactly `datadome`; `Domain` set to the - registrable domain computed against the **Mozilla Public Suffix List** - (vendored revision named by the implementation; host-only is not used - because DataDome requires site-wide scope), path `/`; mandatory - `Secure` and `SameSite=Lax`; `Max-Age` at most **34,214,400 seconds** - (396 days — the thirteen-month ceiling, as an exact number); size ≤ - 4 KiB; + it pins, **aligned to documented vendor behavior where hardening was + not intended**: cookie name exactly `datadome`; `Domain` per + DataDome's own guidance (the module sets it; TS validates it does not + exceed the registrable domain, computed against the **vendored + Mozilla PSL snapshot** `docs/superpowers/specs/psl-snapshot-ref.md` — + ICANN + private sections, IDNA-mapped; IP-literal or single-label + hosts fall back to host-only), path `/`; `Secure` mandatory; + `SameSite` configurable `Lax` (default) / `Strict` / `None` + (`None` requires `Secure`), matching the vendor's endpoint options; + `Max-Age` at most **31,536,000 seconds** (the vendor's one-year cap — + the earlier 396-day figure exceeded it); size ≤ **512 bytes** + (DataDome's current Fastly-module limit; 4 KiB was ours, not theirs). + Where the contract **is** deliberately narrower than the vendor — the + spec-pinned pointer allowlist starting at ClientID-only against + DataDome's mandatory response-directed mapping set — that reduction + needs explicit product **and vendor** acceptance: sign-off item 28; a violating operation is rejected whole (the batch rule). Every `ts-*` name is rejected. **Read is owner-only, and the strip inventory is exhaustive, not integration-scoped** — the browser sends `datadome` in the ordinary @@ -317,8 +326,8 @@ degree of freedom is closed: origin's cache restrictions + public replacement → restriction preserved (pass-through responses never run the hook, §3a); a cache-hit serve re-applying mutations without weakening the stored classification; a `Vary` mutation neither - dropping core-required values nor bypassing the snapshot; each CDN - directive (`Surrogate-Control`, `CDN-Cache-Control`, host equivalents) + dropping core-required values nor bypassing the snapshot; each of the four enumerated CDN fields (`Surrogate-Control`, + `CDN-Cache-Control`, `Cloudflare-CDN-Cache-Control`, `Edge-Control`) individually stripped; and a rejected `Content-Encoding` mutation. ## 5. Size and sequencing diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 360df42ad..b9614430f 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -451,6 +451,16 @@ and the fail-closed marker: entry's; ties resolve to the more restrictive state. So a delayed grant with `LastUpdated = 100` never clears a suppression whose refusal carried `200`, while a genuine re-consent at `300` does. The + **Every suppression entry carries its own `valid_until`, derived from + its evidence class's TTL, and an expired entry is inert** — treated as + cleared without a write, lazily garbage-collected. Without this, an + expired TCF refusal under a `granted` baseline would deny forever: + normalization says an expired record is absent and "must not revoke + indefinitely", yet the surviving suppression would block the baseline + grant that same table promises — the two contracts now agree, in the + normalization table's favor. (Destructive opt-outs tombstone and need + no suppression longevity; non-destructive opt-out entries expire on + the consent-TTL horizon of the evidence that created them.) The transition table (causes without an intrinsic timestamp — malformed records decode no `LastUpdated`, absence has no source — use their **observation timestamp**, server receipt on the shared clock basis @@ -490,8 +500,12 @@ and the fail-closed marker: `GraphOps`, and clearing first is forbidden — so recovery has its own narrow write path: **`AuthorityRefresh`**, permission-exempt but strictly scoped to committing provenance from the _current live - resolution_ (nothing else: no partner writes, no egress, no reads - beyond the row being refreshed) while suppression remains effective; + resolution_ — its exact access set: **read + generation-CAS of the row + being refreshed, and read + CAS of the family's authority-state + record** (its own clearing protocol requires both; the earlier "no + reads beyond the row" wording forbade a read its own CAS needs); + nothing else — no partner writes, no egress — while suppression + remains effective; the clear then references that provenance's revision. Revisions are an **application-level monotonic counter written with the row** — never backend generation markers, which (per Fastly's own contract) only @@ -597,6 +611,7 @@ an expired record before clearing both sources; expiry-first is a | Expired consent record | Treated as **absent entirely** — grants nothing, refuses nothing, withdraws nothing; the baseline applies. Under a `granted` baseline that means the grant stands: an expired refusal is not current evidence and must not revoke indefinitely | Preserved | | One valid record + a second malformed record of the same family | The **valid record governs**; the malformed one is ignored with a `warn` log. Fail-closed-on-malformed (below) applies only when no valid record of that family exists | Decided here | | One valid record + one **expired** record of the same family | The valid record governs — the expired one dropped at pipeline step 2, before conflict resolution ever saw it | **Changed (declared)** — current runtime resolves the conflict first and can select the expired record | +| **Expired** live record + still-valid persisted-KV record | The expired live record is absent entirely (step 2), so it does **not** suppress the fallback: the persisted record substitutes, subject to its own TTL and the full pipeline — "live wins" applies to live records that still exist after expiry filtering | Decided here | | Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | | Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**, then flows through the full normalization pipeline (syntax, expiry, conflict) like any live record; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | **Changed (declared)**: current code returns immediately after the KV load, bypassing expiry and conflict normalization | | Proxy/mirror mode | **Minimal opt-out extraction still runs; full semantic decoding is skipped.** Because opt-outs are globally authoritative (§4), proxy mode must not suppress them: the §4.5-mapped opt-out fields (GPP US sections) and the US Privacy string are decoded — nothing else — alongside syntax validation, so a valid SaleOptOut or USP opt-out revokes and withdraws exactly as outside proxy mode. No grants are ever derived from records in proxy mode; a present record otherwise blocks grants (fail-closed); absent → baseline. GPC needs no decoding | **Changed (declared)**: today proxy mode skips decoding entirely — fail-open under permissive baselines and, worse, opt-out-blind | @@ -614,7 +629,7 @@ the single normative statement; an earlier "N/A contributes nothing" rule is dead, and the P4-authorizing consequence is sign-off item 17) — and only the fields marked destructive trigger withdrawal. Section IDs and versions are those of the IAB GPP -specification current at implementation time; adding a section or field is +specification pinned by the vendored snapshot; adding a section or field is a change to this table. | Source · field | Value | `store-on-device` (P1) | `select-personalised-ads` (P4) | Destructive withdrawal? | @@ -634,6 +649,16 @@ a change to this table. Applicable_ = grant-class; absent = nothing; a non-applicable section's fields grant nothing (their opt-outs still count, per step 2). +**Embedded GPC is mapped, not ignored.** The US sections carry +`GpcSegmentIncluded` and `Gpc` fields; a request with embedded +`Gpc = true` and no `Sec-GPC` header was previously unspecified despite +the global-GPC rule. Normatively: embedded `Gpc = true` in **any** +section is the same **destructive global opt-out** as the header +(aggregated with it by OR — opt-outs are never jurisdiction-filtered); +`GpcSegmentIncluded = false`, an absent segment, or `Gpc = false` +contributes nothing; a malformed optional GPC segment renders that +section malformed-present (blocks grants, never withdraws). + **Applicability and aggregation — ordered algorithm:** 1. **Section map (normative, pinned here — not "whatever GPP is @@ -647,7 +672,12 @@ fields grant nothing (their opt-outs still count, per step 2). `US/NJ` ↔ 21, `US/TN` ↔ 22, `US/MN` ↔ 23, **`US/MD` ↔ 24, `US/IN` ↔ 25, `US/KY` ↔ 26, `US/RI` ↔ 27** (an earlier draft wrongly claimed MD/IN/KY/RI had no section). A truncated map silently loses - opt-outs — a Texas (16) or Maryland (24) sale opt-out must not vanish. + opt-outs — a Texas (16) or Maryland (24) sale opt-out must not + vanish. **The current decoder is an explicit prerequisite gap**: it + (and `iab_gpp` 0.1.2) supports sections 7–23 only and models `usnat` + v2 while the snapshot pins v1 — implementation must extend or replace + the decoder for 24–27 _and_ reject versions the library happens to + decode but the snapshot disallows. The implementation PR cross-checks this list against both the current decoder's section set and the official registry, and the accepted version per section is **pinned to the vendored registry snapshot @@ -751,6 +781,14 @@ migration story unresolvable (migration spec §2, rows 5 and 7). ### 5.5 Policy revision activation +A **policy revision** has a defined identity: the canonical content +digest of the `[permissions]` section (identity — republishing identical +policy yields the same digest) paired with the config-store activation +generation (ordering — strictly monotonic per instance). Provenance +stores both; comparisons order by generation and equate by digest, so a +rollback is a _new_ generation carrying an _old_ digest, with defined +semantics on both axes. + A policy edit propagates through the config store, so a fleet briefly mixes revisions. The contract: instances stamp every resolution and every provenance write with the policy revision they used (already required by @@ -817,23 +855,24 @@ Consumers of the resolved set in this epic: inventory, normative per path (one test per row; a denylist check proves no ungated egress exists): - | Path | Required permissions | Notes | - | ---------------------------------------------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | OpenRTB `user.id` | `store-on-device` ∧ `select-personalised-ads` | Raw EC is identity in the bidstream — gated exactly as EIDs. PR #838 gated only EIDs, leaving `user.id` reachable with Purpose 4 refused | - | EC-derived auction request IDs | both purposes | Derived values are identity | - | Page-bids path | both purposes | | - | Bidstream EIDs | both purposes | The one gate PR #838 had | - | Proxy / click / Testlight forwarding of the EC cookie or headers | both purposes | **New hardening, declared change** — these paths extract the raw cookie/header without today's jurisdiction gate (migration spec §2 row 11b) | - | Identify endpoint (partner-facing) | both purposes | Partner identity exchange, not a first-party lookup — decided here | - | Pull sync (browser-request-scoped partner exchange) | both purposes, from the **live** request resolution | Pull sync is created from a browser request and checks the live `EcContext` today — it keeps using the live P1 ∧ P4 decision plus the family revocation state (§4.3); stored provenance is never a substitute for available live evidence | - | Batch sync (context-free S2S partner exchange) | both purposes, from **stored provenance** | The only truly signal-less path; authority rules below. Today's handler only authenticates and checks row state, so this gate is **declared hardening** (migration spec §2) | - | Request-scoped graph reads/writes (non-revocation) | `store-on-device` | | - | Revocation paths (tombstones, withdrawal reads) | **exempt** | Must work when permissions are unset | - | Stored consent-state lookup (§4.4) | **exempt**, narrowly scoped | Determining `store-on-device` cannot require `store-on-device` | - | Integration persistent response cookies | `store-on-device` (+ P4 where the cookie is an advertising identifier) | **Deferred with the hook's cookie surface** — the write-side gate alone was insufficient (read/use/forward/withdrawal unmodeled), so cookie operations ship only with the full model; this row and the client-cycle **page leg** (module injection gated on the provider's full declaration) join the inventory when their features do, and the §5.3 no-geo guard's consumer list grows with them | - | Suppression-record writes (§4.3) | **exempt** | Clearing authority is protective, like revocation | - | Authority-state / suppression-decision read (§4.3) | **exempt**, narrowly scoped | Returns only family ID, per-permission authority summary, and suppression entries — no identity values, no partner data; a test proves nothing else escapes | - | `AuthorityRefresh` provenance write (§4.3) | **exempt**, strictly scoped | Commits current-live-resolution provenance only; enables suppression recovery without reopening `GraphOps` | + | Path | Required permissions | Notes | + | ------------------------------------------------------------------ | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | OpenRTB `user.id` | `store-on-device` ∧ `select-personalised-ads` | Raw EC is identity in the bidstream — gated exactly as EIDs. PR #838 gated only EIDs, leaving `user.id` reachable with Purpose 4 refused | + | EC-derived auction request IDs | both purposes | Derived values are identity | + | Page-bids path | both purposes | | + | Bidstream EIDs | both purposes | The one gate PR #838 had | + | Proxy / click / Testlight forwarding of the EC cookie or headers | both purposes | **New hardening, declared change** — these paths extract the raw cookie/header without today's jurisdiction gate (migration spec §2 row 11b) | + | Identify endpoint (partner-facing) | both purposes | Partner identity exchange, not a first-party lookup — decided here | + | Pull sync (browser-request-scoped partner exchange) | both purposes, from the **live** request resolution | Pull sync is created from a browser request and checks the live `EcContext` today — it keeps using the live P1 ∧ P4 decision plus the family revocation state (§4.3); stored provenance is never a substitute for available live evidence | + | Batch sync (context-free S2S partner exchange) | both purposes, from **stored provenance** | The only truly signal-less path; authority rules below. Today's handler only authenticates and checks row state, so this gate is **declared hardening** (migration spec §2) | + | Request-scoped graph reads/writes (non-revocation) | `store-on-device` | | + | Revocation paths (tombstones, withdrawal reads) | **exempt** | Must work when permissions are unset | + | **Observability sinks — logs, traces, metrics, error attachments** | Never — no permission authorizes them | Raw EC values (and derived URLs embedding them) must not reach any observability sink: logging boundaries accept redacted/hash-only types, not `&str` (PR #838 logs a redirect URL containing the EC and the raw `ec_id` field — the motivating counterexample); a log-schema denylist test enforces the row | + | Stored consent-state lookup (§4.4) | **exempt**, narrowly scoped | Determining `store-on-device` cannot require `store-on-device` | + | Integration persistent response cookies | `store-on-device` (+ P4 where the cookie is an advertising identifier) | **Deferred with the hook's cookie surface** — the write-side gate alone was insufficient (read/use/forward/withdrawal unmodeled), so cookie operations ship only with the full model; this row and the client-cycle **page leg** (module injection gated on the provider's full declaration) join the inventory when their features do, and the §5.3 no-geo guard's consumer list grows with them | + | Suppression-record writes (§4.3) | **exempt** | Clearing authority is protective, like revocation | + | Authority-state / suppression-decision read (§4.3) | **exempt**, narrowly scoped | Returns only family ID, per-permission authority summary, and suppression entries — no identity values, no partner data; a test proves nothing else escapes | + | `AuthorityRefresh` provenance write (§4.3) | **exempt**, strictly scoped | Commits current-live-resolution provenance only; enables suppression recovery without reopening `GraphOps` | With **no EC provider configured**, identity use fails closed: a cookie value present on the request never egresses anywhere — never vacuously @@ -858,12 +897,14 @@ Consumers of the resolved set in this epic: | GPP / USP values (no intrinsic timestamp) | **First-seen**: when TS first observed this exact normalized value (a **per-permission equality digest computed over only the applicable, aggregated §4.5 fields for that permission** — never the whole GPP record, or a CMP touching an unrelated notice field would mint a new digest and reset first-seen forever) | Re-presenting an identical digest **keeps the original first-seen**; a different value is new evidence with a new first-seen | Consent TTL (same as TCF) | | Policy-baseline grant (`granted` rule, no signal) | The policy revision that granted | Re-derived on every recompute against the current revision — policy is not user evidence and does not age; it changes | n/a | - Timestamps are compared with a bounded clock-skew tolerance that is a - **normative constant — 300 seconds** (five minutes, applied - symmetrically; a spec-level value because suppression precedence, - malformed classification, and future-date handling all hinge on it, - and per-deployment values would give the same input different privacy - outcomes); + Timestamp handling is an **algorithm, not just a constant**: with + skew S = 300 s (normative), a timestamp `t > now + S` renders its + record malformed-present; `t` in `(now, now + S]` is used as-is (not + clamped — clamping re-freshens replays); expiry checks grant a grace + of S (`expired` means `valid_until < now − S`); and two evidence + timestamps within S of each other **compare equal**, which routes + the comparison to the tie rule (restrictive) — so a slightly + future-dated consent cannot out-order a just-observed opt-out; beyond-window future-dated records are **rejected as malformed**, and within the window a record's first normalized timestamp is pinned to its digest and never advanced by re-presentation (§4.3's anti-replay @@ -880,7 +921,15 @@ Consumers of the resolved set in this epic: stored evidence has **expired**, or when the regime no longer accepts the stored grant's source class (§4's regime-scoped table). Any of these → no update, row flagged for the operational cleanup of §4.2 - trigger 3. Sync never mints authority of its own. + trigger 3. Sync never mints authority of its own. **Stored + jurisdiction ages too**: batch sync has no live geo, so the + jurisdiction it recomputes against is the one from the last browser + visit — and a visitor who moved from a permissive into a GDPR + jurisdiction would otherwise keep old-rule egress for up to the row + lifetime. A stored jurisdiction older than the **consent-TTL + horizon** fails closed pending a live refresh (the inverse — denying + a visitor who moved the other way — is the accepted cost); the + horizon choice and its legal trade-off are **sign-off item 25**. **Legacy (pre-epic) rows** carry none of these fields. They are treated as reserved `hmac-v0` provenance with **no stored grant evidence**, so diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index fbe362fcf..05e231e95 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -263,12 +263,18 @@ normative order is: gate → `generate` → graph-row commit → **authority-state commit** (the strong record reporting the row's revision — the commit point of the two-record protocol, permission model spec §4.3) → cookie scheduled → eligible for egress. Eligibility begins -at the authority-state commit, not the row commit: a row whose revision -the strong record has not reported is uncommitted detail, which is what -keeps S2S authorization and the absence decision consistent. "Cookie scheduled" means queued onto the +at the authority-state commit and nowhere earlier. **Mint-path failure +between the two writes is declared, not "recovered"**: no cookie was +emitted, so no later request can identify the orphan — the earlier +claim that recovery "re-runs step 2" is false for the mint path (it +holds only for _presented_ identities via `AuthorityRefresh`). An +orphaned row (or orphaned pending record, if the order's first write +succeeded alone) authorizes nothing — the strong record never reported +it — and is bounded by its `expires_at`/retention TTL; the failure is +counted (a first-class metric) and appears in the §6.2 runtime matrix. "Cookie scheduled" means queued onto the final response — `Set-Cookie` is physically emitted after first-request -processing, so egress eligibility begins at **graph commit**, not at -header emission; the identity exists durably from that moment. A +processing, so egress eligibility begins at the **authority-state commit** +(§5 mint order — not graph commit, and not header emission). A graph-commit failure means the mint never happened: no cookie, no egress, error logged, the next request retries. @@ -285,14 +291,21 @@ variant**. Therefore: not-found proves nothing — a just-minted row invisible on a stale replica would classify its own cookie as rowless and fork the identity, and "the backend's strongest read" over eventual storage is - not an authoritative primitive. The proof uses what the protocol - already guarantees: **every post-upgrade identity has an - authority-state record** (the commit point, §5 mint order) under its + not an authoritative primitive. The proof needs more than the flag — + a per-deployment flag cannot prove a per-cookie fact, and "no + authority-state record" alone would misclassify every graph-backed + legacy row and every N+1-minted v1 row (neither has one). The + protocol closes both gaps with **prerequisites for setting the + flag**: (1) the fleet has fully converged on **N+2** (no v1 minting + anywhere — an N+1 fleet must never run rowless classification), and + (2) an idempotent **stub-backfill scan** has stamped a minimal + authority-state existence stub onto **every existing identity row** + (legacy and N+1-minted alike). Only then does the invariant hold: + every row-backed identity has an authority-state record under its derivable family ID, in the globally-strong class — so _rowless_ = - the deployment-metadata **graphless-migration flag** is set AND the - strong read finds **no authority-state record** for the cookie's - derived family ID. Graphless-era cookies never had one; no eventual - read participates. The flag itself is specified: a named + flag set AND the strong read finds **no record** for the cookie's + derived family ID. Graphless-era cookies never got a stub because + they have no row for the scan to find; no eventual read participates. The flag itself is specified: a named deployment-metadata key (write-once/CAS class), set by the §4.2 readiness step only on deployments that actually ran graphless (requires the deployment-metadata capability), surviving binary @@ -308,19 +321,29 @@ variant**. Therefore: deliberately not preserved (migration matrix row 13, sign-off 21). An unverifiable cookie (including the declared roaming false-negative) is simply expired. -- **Rowless withdrawal writes an exact-cookie family record, then - expires the cookie** — one contract, aligned with the family-first - rule of the permission spec (cookie-only expiry would be best-effort: - a lost response leaves the "withdrawn" cookie usable on its next - presentation). The family ID uses the **same full-graph-key derivation - as row-backed identities** — one derivation everywhere — so the record - revokes exactly the presented cookie value: no per-IP blast (the - earlier prefix derivation would have revoked every identity behind one - IP), and bounded minting, because only **prefix-verified** cookies may - write one — an attacker can fabricate suffix variants only for their - own evidence's prefix, spending their own withdrawal on themselves. - A re-presented withdrawn variant finds its family record and stays - dead. +- **Rowless withdrawal writes into one capped per-prefix record, then + expires the cookie** — durable (cookie-only expiry is best-effort: a + lost response leaves the "withdrawn" cookie usable), and **bounded in + storage**, which separate exact-cookie records were not: a holder of + one valid prefix can fabricate billions of suffix variants, and + per-variant records would be attacker-priced strong storage. The + record (strong class, keyed on the verified prefix) holds a bounded + list (cap 8) of withdrawn-suffix hashes; writes are admitted only for + **prefix-verified** cookies; **saturation escalates to prefix-wide + rowless revocation** — every rowless cookie under that prefix is + treated withdrawn, which harms only the abuser's own same-IP graphless + cohort and is the declared abuse response (legitimate users hold one + or two variants ever). A re-presented withdrawn variant finds its + entry (or the saturated record) and stays dead. Row-backed + withdrawal is untouched: full-graph-key family records, one derivation + everywhere. +- **Negative-record creation has admission rules everywhere**: + suppression and family records may be written only for (a) an + existing family — the authority-state record exists on a strong + read — or (b) a **verified** identifier (`verify`); a fabricated, + unverifiable cookie writes nothing. This bounds record creation to + real identities plus the writer's own evidence, and per-prefix + rate limits apply to the rowless path above. **Egress is typed, not policed.** The inventory-and-denylist test (permission model spec §7) is a backstop, but conventions do not survive @@ -494,15 +517,16 @@ Startup validation (§6) covers configuration; this covers what happens when a healthy configuration meets an unhealthy runtime. Every row logs at `error` with a metric; none is silent: -| Failure | Behavior | -| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | -| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | -| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | -| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | -| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | -| Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | -| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | -| Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | +| Failure | Behavior | +| ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | +| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | +| Authority-state commit fails after the row commit at mint | No cookie, no eligibility (the strong record never reported the revision); the orphan row expires by TTL; counted — **no recovery claim**, nothing can find it | +| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | +| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | +| Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | +| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | +| Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | The **degraded-graph health signal** referenced above and by the withdrawal contract is a defined state machine, not a vibe: it is @@ -539,7 +563,8 @@ the bounded suffix: | Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | | Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | | Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | -| Rewrite transaction | `rwx/` | One in-flight rewrite per family | +| Deployment metadata | `m` + fixed metadata name (fixed-width grammar; schema floor, graphless-migration flag) | Write-once/CAS class; value carries schema version, state, epoch, set-at; the graphless flag's lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded | +| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | | Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | Grammars are pairwise non-intersecting by their literal prefixes (plus @@ -558,8 +583,8 @@ Physical keys are **delimiter-free with fixed-width segments**: a generated key can begin with 64 hex characters, which is what makes disjointness from the legacy `{64hex}.{6alnum}` grammar _provable_ rather than asserted (an earlier `f` tag was itself a hex digit) — then -a **4-character provider code from a checked-in, append-only, -never-reused registry file** (allocation is a reviewed commit; +a **4-character provider code from the checked-in, append-only, +never-reused registry `docs/superpowers/specs/provider-code-registry.md`** (allocation is a reviewed commit; codes are immutable and never recycled, including for retired providers), then the suffix. Segment boundaries are positional, so no segment can contain or escape a delimiter, prefix queries are plain @@ -573,12 +598,18 @@ deadline, and fencing epoch; the **family revocation record** holds the family ID, revoked-at, triggering signal class (§4.5 destructive column), and a **family epoch** bumped on every revocation-state change (the client-cycle commit CAS is conditioned on it) — deliberately no identity -data, so it can outlive its members; the **authority-state (suppression) record** holds, per permission: +data, so it can outlive its members; the **authority-state record** holds, per permission — negative side: state (`suppressed`/`cleared`), cause, source class, authoritative or -observation evidence timestamp, the **application-level provenance -revision** a clear references, and the positive-authority summary -(revision + evidence timestamp) — plus the record-level CAS version -counter and schema version; unknown-field and range validation apply +observation evidence timestamp, entry `valid_until` (evidence-class TTL; +expired entries are inert), and the provenance revision a clear +references; positive side (the summary, **every field the permission +spec's absence/replay decisions consume — a reduced schema cannot +reproduce them**): kind (user evidence vs policy baseline), grant +basis/source class, policy revision (digest + activation generation), +`valid_until`, provenance revision, evidence timestamp, and the +per-permission **semantic digest with its pinned first-seen/ +first-normalized timestamps** (anti-replay); record level: family ID, +CAS version counter, schema version, stub marker (backfill, §5); unknown-field and range validation apply like every class (strong class, permission-exempt writes per the permission spec's inventory; revisions are app-level counters because backend generation markers detect change without ordering); @@ -608,7 +639,7 @@ achievable through a structured serializer). | `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | | New: **immutable mint tag** (`mint_provider`, `mint_version`) | Credential retirement and audit — write-once at mint (legacy backfill may populate a missing tag once); **never part of the replaceable snapshot**, or a v1 identity revisited after rotation would be restamped v2 | Mint (or one-time backfill) | — | Immutable | — | Retained | | New: **provenance revision** (application-level monotonic counter, u64, initialized at 1, incremented by every provenance-bearing write, CAS'd with the row generation, serialized as an integer; overflow is a hard error, not a wrap) | Orders clears vs. snapshots (permission spec §4.3) | Core | Read by S2S/clears | Monotonic | — | Retained | -| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision, ) | S2S authority | Live resolution only | Read by S2S recompute | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | +| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision) | S2S authority | Live resolution only | Read by S2S recompute | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | | New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | | `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | | `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 474275616..f9bdbb36f 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -161,8 +161,10 @@ Requirements: later than the model it protects. Live-request paths keep v1 semantics until N+2. - Rollback tests therefore run the family-revocation and suppression - paths — read **and write** — plus v1-minting behavior, on N+1 + Rollback tests therefore run: family-revocation read **and + write**; authority-state/suppression **read-and-fail-closed only** + (N+1 writes none — the earlier read-and-write test requirement + contradicted this contract); and v1-minting behavior — all on N+1 against N+2-written data. **Rollback is binaries-first too, in the other direction** — N+2 → N+1 binaries roll back keeping the new config (N+1 reads it fully; reverting config first would hand the old shape to N+2 @@ -194,7 +196,15 @@ Requirements: a deprecated `passphrase` field whose presence triggers the custom error). -2. **Revocation-eligible storage is a per-adapter gate, and ungated +2. **Adapter qualification is a pre-ratification prerequisite, not a + footnote.** No adapter is presently proven eligible for the complete + identity protocol — Fastly's global-read/retention cells are + unverified and its deployment-metadata primitive unwired; every + other adapter is unavailable or needs a new primitive. Ratifying + before at least one adapter qualifies risks an epic with no + selectable identity provider, so Fastly qualification (or an + explicit decision to proceed without it) gates ratification. +3. **Revocation-eligible storage is a per-adapter gate, and ungated adapters migrate stateless.** Identity features require the adapter's strong-consistency rows in the capability matrix (providers spec §7) to be green: today that means Fastly must _verify_ its KV read @@ -206,7 +216,7 @@ Requirements: would make the required fixtures self-contradictory. Whether ungated adapters go stateless or block the release is product sign-off item 12. -3. **Graph-store readiness precedes everything.** Today the graph store +4. **Graph-store readiness precedes everything.** Today the graph store is optional and EC generation succeeds without one; the epic's no-active-until-commit invariant (providers spec §5) makes it mandatory wherever a minting provider is configured — so a currently @@ -217,7 +227,7 @@ Requirements: row supports the features in use, providers spec §7) _before_ rolling N+1. This is a **declared breaking change** for graphless deployments (matrix row 12), not a side effect discovered at boot. -4. **The graph schema change is expand-contract, in lockstep with the +5. **The graph schema change is expand-contract, in lockstep with the binary sequence.** New rows carry fields v1 rows never had — provider/ version, per-permission grant evidence, policy revision, family ID — and two failure modes must be engineered away: a naive schema-version bump makes old readers fail closed on new rows, and an @@ -256,32 +266,32 @@ Requirements: new fields untouched if read-only, preserved semantically if read-modify-write on N+1, **test-proven lost on pre-N+1** (documenting why the floor is a floor); N+2-reader/N+1-written-row → full function. -5. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no +6. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no `provider = "hmac"` selector is a startup error (providers spec §6). In PR #838 this configuration — the exact state an operator following the docs reaches if they miss one line — validated green and silently minted zero ECs. -6. **PR #838-era keys fail loud.** `provider = "host-signals"` (shipped by +7. **PR #838-era keys fail loud.** `provider = "host-signals"` (shipped by PR #838, deliberately not carried into this epic — providers spec §2) and `provider = "client-fixed"` are unknown keys and rejected like any other, so a config written against the PR #838 example cannot silently select a provider that no longer exists. -7. **Provider switches go through legacy readers.** Changing +8. **Provider switches go through legacy readers.** Changing `[ec] provider` on a deployment with live identities requires listing the outgoing provider in `[ec] legacy_providers` (providers spec §6.1) so existing cookies keep resolving and stay withdrawable; the guide documents the switch sequence and the retirement/cleanup step that ends it. -8. **The example config ships the migrated happy path**, uncommented: +9. **The example config ships the migrated happy path**, uncommented: `provider = "hmac"` with its block, `[geo] default_country`, and (for Fastly) the behavior-preserving `[device] provider = "fastly"` and `[geo] provider = "platform"` lines present with a comment stating what removing them changes. PR #838's example shipped the passphrase block uncommented with the selector commented out — steering operators directly into the silent-stateless state. -9. Every misconfiguration in the providers spec §6 table fails at - **startup**. Request-time failure for a configuration error is a defect. -10. Validation is split into two named layers, because "the same +10. Every misconfiguration in the providers spec §6 table fails at + **startup**. Request-time failure for a configuration error is a defect. +11. Validation is split into two named layers, because "the same validation at push and startup" is not implementable: **structural validation** (schema, types, `[permissions]` policy — permission spec §3.3) runs at `ts config push` and again at startup; @@ -378,10 +388,8 @@ global honoring of opt-out signals is unconditional. maximum cookie/row lifetime plus rollout skew** is the **only retirement-readiness** bar for a legacy provider ("trending to ~zero" is not evidence; a yearly visitor - is not churn), (rewrite-based backfill and its metrics left with the rewrite - deferral). The telemetry set also includes: graph read/commit failures, - stored-provenance denials, schema-migration failures, and - replay-reservation recoveries. **Each rollout-gate metric ships with a + is not churn), . The telemetry set also includes: graph read/commit failures, + stored-provenance denials, and schema-migration failures. **Each rollout-gate metric ships with a threshold, an evaluation window, and a named action** (pause rollout / roll back / block retirement) in the migration guide — a metric with a "healthy range" but no action is dashboard decoration; the two already @@ -450,29 +458,33 @@ recording the decision, the deciders, and the date) — the table links them as rows close; an unratified row reverts to open, not to silently implemented. -| # | Decision | Where | Owner | Status | -| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | -| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | -| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | -| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | maintainers + legal | open | -| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | -| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | -| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | -| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | -| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | -| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | -| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | -| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | -| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | -| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | -| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | -| 16 | Sticky opt-out for timestamp-less sources: a GPP/USP-only re-consent does not restore authority without a timestamped grant | permission §4.3 | maintainers + legal | open | -| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | maintainers + legal | open | -| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | maintainers + legal | open | -| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | maintainers + product | open | -| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | maintainers + product | open | -| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | maintainers + product | open | -| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | maintainers + legal | open | -| 23 | **Open question, not ratified**: may DataDome's security identifier (tag injection, `datadome` cookie, `X-DataDome-ClientID` read and vendor egress) operate outside the permission model? The decision must enumerate exactly which consumers may observe the cookie/ClientID — today's spec allows only the security channel itself and strips every other surface (hook §4a) — plus retention and whether TS withdrawal expires it | hook §4a; permission §7 | maintainers + legal | open | -| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | maintainers + legal | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | +| # | Decision | Where | Owner | Status | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | +| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | +| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | +| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | maintainers + legal | open | +| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | +| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | +| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | +| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | +| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | +| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | +| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | +| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | +| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | +| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | +| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | +| 16 | Sticky opt-out for timestamp-less sources: a GPP/USP-only re-consent does not restore authority without a timestamped grant | permission §4.3 | maintainers + legal | open | +| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | maintainers + legal | open | +| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | maintainers + legal | open | +| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | maintainers + product | open | +| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | maintainers + product | open | +| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | maintainers + product | open | +| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | maintainers + legal | open | +| 23 | **Open question, not ratified**: may DataDome's security identifier (tag injection, `datadome` cookie, `X-DataDome-ClientID` read and vendor egress) operate outside the permission model? The decision must enumerate exactly which consumers may observe the cookie/ClientID — the enumerated observers are the security channel **and the publisher origin, which receives `X-DataDome-ClientID` via the upstream overlay** — "owner-scoped overlay" names the mechanism, this row names the recipient — plus retention and whether TS withdrawal expires it | hook §4a; permission §7 | maintainers + legal | open | +| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | maintainers + legal | open | +| 25 | Batch-sync stored-jurisdiction maximum age (consent-TTL horizon): a mover into GDPR stops old-rule egress at the horizon; a mover out is denied until a live visit | permission §7 | maintainers + legal | open | +| 26 | Embedded GPP GPC maps to the destructive global opt-out (header-OR-embedded aggregation) | permission §4.5 | maintainers + legal | open | +| 27 | Proxy-mode minimal opt-out extraction (decode only §4.5-mapped opt-out fields; no grants) | permission §4.4 | maintainers + legal | open | +| 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance | hook §4a; `datadome-header-allowlist.md` | maintainers + product | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index f0c1291af..9f2b04669 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -240,3 +240,29 @@ instead of trusting prose. | P2 revocation wording paraphrase | fixed (permission spec repeats "globally observable" verbatim) | | P2 deferred residue in normative schemas | fixed (rewrite transaction and reservation wire schemas bracketed informative; rewrite links out of the migration expansion item) | | P3 dangling sentences / key-table description / ledger reliability | fixed (fragments removed; key table says authority-state with positive summary; this section's mechanical-anchor rule) | + +## Round 13 — review at ba25ba85 + +**Status vocabulary change (per this round's P3):** ledger rows now +distinguish **text-added** (normative text landed; cross-document +coherence pending the next review) from **verified-closed** (a later +review re-examined and did not reopen). A greppable anchor proves +presence, not coherence — R12's own rows demonstrated the difference. +All R12 rows are retroactively **text-added**; rows below are +text-added unless marked otherwise. + +| Finding | Status | +| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| P1 expired refusal suppresses forever | text-added: suppression entries carry evidence-class `valid_until`, expired entries inert — normalization wins | +| P1 graphless flag can't prove per-cookie fact | text-added: rowless gated on N+2 convergence + idempotent stub-backfill of every existing row; N+1 fleets never classify rowless | +| P1 mint authority-write failure unrecoverable | text-added: false recovery claim retracted; declared TTL-bounded orphan + failure-matrix row; eligibility-at-graph-commit leftover swept | +| P1 wire schema can't represent the protocol | text-added: full authority-state schema (negative entries with valid_until; positive summary with kind/basis/policy identity/digest/first-seen; stub marker) | +| P1 negative-record amplification | text-added: admission rules (existing family or verified id); rowless withdrawal = one capped per-prefix record (8), saturation → prefix-wide rowless revocation as declared abuse response | +| P1 embedded GPP GPC unmapped | text-added: Gpc=true in any section = destructive global opt-out, header-OR-embedded; GpcSegmentIncluded/malformed rules; sign-off 26 | +| P1 stale batch jurisdiction | text-added: consent-TTL jurisdiction-age horizon, fail closed pending live refresh; sign-off 25 | +| P1 observability egress | text-added: inventory row, hash-only logging types, #838 logging site cited, denylist test | +| P1 200-vs-304 divergence | text-added: persisted metadata versioned by (registry, config, invariant) revisions; mismatch = miss; cache-relevant fields defined | +| P1 DataDome vendor incompatibility | text-added: SameSite configurable, 1-year cap (31,536,000 s), 512-byte size, Domain per vendor guidance PSL-validated; reduced pointer set → product **and vendor** acceptance (sign-off 28) | +| P1 publisher origin as ClientID observer | text-added: sign-off 23 names the recipient | +| P2 batch (flag wire contract + `m` class; N+1 test alignment; AuthorityRefresh read-set; expired-live fallback row; skew algorithm; policy-revision identity = digest + activation generation; provider-code registry + PSL reference files created; decoder prerequisite noted; sign-off rows 25–28; adapter qualification as pre-ratification prerequisite; telemetry residue swept) | text-added | +| P3 batch (hook fragment; provenance comma; host-equivalents in tests; client-draft Axum claim; this vocabulary change) | text-added | diff --git a/docs/superpowers/specs/provider-code-registry.md b/docs/superpowers/specs/provider-code-registry.md new file mode 100644 index 000000000..fb373f32e --- /dev/null +++ b/docs/superpowers/specs/provider-code-registry.md @@ -0,0 +1,10 @@ +# Provider-code registry (normative, append-only, never reused) + +Four-character codes (`[a-z0-9]`, zero-padded) used in physical graph +keys (providers spec §6.3). Allocation is a reviewed commit to this +file; codes are immutable and never recycled, including for retired +providers. + +| Code | Provider | Allocated | Status | +| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------ | +| `hmac` | Built-in HMAC EC provider (note: hmac identities use the reserved verbatim key grammar, so this code appears in non-key contexts — provenance, registries — not in physical keys) | 2026-08-02 | active | diff --git a/docs/superpowers/specs/psl-snapshot-ref.md b/docs/superpowers/specs/psl-snapshot-ref.md new file mode 100644 index 000000000..72bb3a519 --- /dev/null +++ b/docs/superpowers/specs/psl-snapshot-ref.md @@ -0,0 +1,13 @@ +# Public Suffix List snapshot reference (normative) + +The vendored Mozilla PSL revision used for registrable-domain +computation (hook spec §4a): the implementation PR vendors the list file +and records its upstream commit hash here. Rules: ICANN **and** private +sections apply; hostnames are IDNA-mapped before matching; IP literals +and single-label hosts have no registrable domain (cookie falls back to +host-only). Updating the snapshot is a reviewed spec change. + +| Field | Value | +| --------------- | --------------------------------------------------------- | +| Upstream commit | _recorded by the implementation PR that vendors the list_ | +| Vendored path | _recorded alongside_ | From f3eacf5902d8b45cfe62b0c59fd1691fc5c8c0ea Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Mon, 3 Aug 2026 09:10:10 -0700 Subject: [PATCH 15/24] Address fourteenth review: monotone commits, replay history, one opt-out lifetime, and closed DataDome directions P1 fixes: - The authority-state transition rejects revision regression (incoming < stored refused; equal is idempotent and payload-equivalent, with a named delayed-r2-after-r3 test) - CAS alone let a delayed older commit restore older authority. - Anti-replay keeps a bounded replay history keyed by (source class, semantic digest) - horizon-retained, capped at 16 per permission-source, saturation fails restrictive - because a single current digest forgets grant A the moment refusal B displaces it, and replayed A would then look novel. - Timestamp-less opt-out has one lifetime contract, chosen explicitly: TTL-sticky (inert at consent-TTL valid_until; within lifetime only a newer timestamped grant clears; administrative clear is an optional early exit). The migration spec's 'irreversible artifact requiring administrative clear' wording is superseded; sign-off 16 rewritten. - The stub-backfill invariant is establishable: N+2 convergence, the backend's documented listing settle window (listing completeness is a declared capability - an unboundable backend cannot host this migration), repeated scans to two consecutive zero-discovery passes, attestation in the flag value - and misses reconcile rather than fail: a late-surfacing real row for a withdrawn suffix promotes the prefix entry to a full family revocation at first sight. - The rowless withdrawal record is one class end to end: 'w' + provider code + prefix in the physical grammar, wire schema (capped list of 8, saturation flag, CAS version, valid_until >= max cookie lifetime), linearizable CAS in the capability matrix, N+1 fail-closed reader, and migration row 13 aligned (exact-cookie records superseded). - Roaming unverifiable cookies: cookie-only expiry is a disclosed residual with re-attempt on every re-presentation - sign-off 29. - Verified suffix variants cannot amplify suppression: durable per-family negative records require row-backed families; rowless negative state is exclusively the capped prefix record. - DataDome's browser-response direction is a decision-scoped positive allowlist (Respond: Location/Content-Type/Cache-Control/Pragma + typed cookie + enumerated vendor headers from the allowlist file's new response section; Continue: cookie + vendor headers only) - the atomic-302 example's Location is admitted, not assumed. - Incoming X-DataDome-ClientID is owner-only like the cookie: extracted into the DataDome-only view, removed from the shared request and upstream routing, added to RedactedRequestView's strip set - DataDome itself prioritizes the header over the cookie. - Normal and conditional cache hits serve the same persisted post-hook finals captured at cache fill (revision-versioned; mismatch = miss); metadata identity holds by construction with no mutator-purity assumption. P2/P3: absence suppression is one-shot per positive summary (writing retires the summary; entry valid_until capped by the retired horizon); policy activation uses the config store's globally assigned push version (per-instance monotonicity ordered nothing across a fleet); unknown GPP section IDs contribute nothing - with the honest bound that embedded GPC in an unparseable section is undetectable; the physical grammar has complete byte-level constructors (i/r/s/x/m/w, family-id = 64-hex SHA-256, m-names 16-char padded, 128-byte cap, per-class parsers); DataDome cookies must RFC 6265 domain-match the request host with Expires-to-Max-Age normalization (both present: Max-Age wins; ceiling violations reject the batch); Cache-Control/Vary parsing is one shared fail-closed core parser with fixtures (Vary: * = shared-uncacheable); NAT saturation collateral is sign-off 30; PSL fill, decision records, and adapter qualification are named ratification gates; the provider section 5 lead now states the authority-state commit point, section 4's 'cannot be gated at all' distinguishes geo (cannot) from device (deliberately not); typo/punctuation/metric residue swept; the client draft's resv notation is marked superseded; and the old DataDome spec carries an explicit supersession banner so 'applies last' cannot be read as current. Per maintainer direction, the sign-off table is decision-centric: the Owner column is replaced by a Decision-record link column - the table tracks decisions; the records capture deciders. --- ...-datadome-server-side-protection-design.md | 11 ++ ...26-07-30-client-cycle-ec-resolve-design.md | 5 +- ...integration-response-header-hook-design.md | 59 ++++++-- .../2026-07-30-permission-model-design.md | 49 +++++-- .../2026-07-30-pluggable-providers-design.md | 87 +++++++++--- ...07-30-provider-migration-rollout-design.md | 130 +++++++++--------- .../specs/datadome-header-allowlist.md | 12 ++ docs/superpowers/specs/pr986-review-ledger.md | 20 +++ 8 files changed, 266 insertions(+), 107 deletions(-) diff --git a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md index f8d582a4f..5d59e2ef6 100644 --- a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md +++ b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md @@ -1,5 +1,16 @@ # DataDome Server-Side Protection API Integration +> **Supersession note (PR #986):** the response-effects portions of this +> document — in particular "DataDome headers/cookies apply last and win" +> and any post-finalization ordering — are **superseded** by the +> response-header hook spec's §4a security-channel contract +> (`2026-07-30-integration-response-header-hook-design.md`): one global +> order applies (core finalization → ordinary mutators → security +> effects → final cache/privacy invariant pass, unconditionally last), +> with typed cookie/header operations, enumerated allowlists +> (`datadome-header-allowlist.md`), and owner-only identifier +> boundaries. Where this document conflicts, the hook spec governs. + **Issue:** #317 **Date:** 2026-06-11 **Status:** In Progress diff --git a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md index d6a5a487d..7630ebb26 100644 --- a/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md +++ b/docs/superpowers/specs/2026-07-30-client-cycle-ec-resolve-design.md @@ -153,8 +153,9 @@ Everything in this spec follows from that. happens under the reservation and is deterministic under its key, so any retry converges on the same row; reservations are retained at least through the token's expiry. Reservation keys are **namespaced** - per the providers spec §6.3 grammar - (`resv////`), so + (illustrated here with the superseded slash notation; the actual + grammar at revival follows the providers spec's delimiter-free + fixed-width scheme), so payloads cannot collide across publishers, providers, or versions. Ownership conflicts are terminal per state: a non-owner hitting `pending` gets `409` (retry only after lease expiry); a non-owner diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index c290b0c3d..574a9b268 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -85,7 +85,14 @@ mutators to the outbound response for HTML document responses it processed. they are additionally stripped from any restricted response; and the final `Vary` is the **union of the complete snapshot `Vary` set** — origin-supplied members included, not only core-required ones — and - the mutation. Middle-stage placement also keeps + the mutation. Parsing itself is a **shared core parser with + fail-closed normalization**, not four adapter interpretations: + invalid `Cache-Control` syntax normalizes to the most restrictive + reading; duplicate directives keep the strongest; quoted and unquoted + forms are equivalent; conflicting `max-age` values keep the smallest; + unknown extension directives are dropped at merge; and `Vary: *` is + treated as uncacheable-by-shared-caches (no-store-equivalent for the + invariant). Conformance fixtures cover each rule. Middle-stage placement also keeps the earlier property: an integration mutation is not silently stripped by ordinary core handling — only by the invariant pass, which logs the downgrade it applies. @@ -202,17 +209,17 @@ mutators to the outbound response for HTML document responses it processed. Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Yes, applied at serve time (mutations are not cached) | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by (integration-registry revision, config revision, invariant revision)**; a 304 re-emits those persisted finals only when all three match the serving instance — a mismatch (integration or config changed since caching) is a **cache miss**, so a normal hit and a conditional hit can never return different policy metadata for one representation. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | -| `HEAD` of a processed document | **Yes — header parity with GET is mandatory** (a cache may refresh stored GET metadata from HEAD, and divergent CSP/privacy metadata between the two is a leak) | -| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | +| Response | Hook runs? | +| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by (integration-registry revision, config revision, invariant revision)**; a 304 — like a **normal cache hit** (§3a), which serves the same persisted finals rather than re-running mutators — re-emits them only when all three match the serving instance; a mismatch is a **cache miss**. Identity of normal-hit and conditional-hit metadata holds by construction (one persisted artifact serves both), not by an unsupported determinism claim about mutators. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | +| `HEAD` of a processed document | **Yes — header parity with GET is mandatory** (a cache may refresh stored GET metadata from HEAD, and divergent CSP/privacy metadata between the two is a leak) | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). @@ -234,6 +241,12 @@ degree of freedom is closed: hosts fall back to host-only), path `/`; `Secure` mandatory; `SameSite` configurable `Lax` (default) / `Strict` / `None` (`None` requires `Secure`), matching the vendor's endpoint options; + the returned `Domain` must additionally **domain-match the current + request host** per RFC 6265 (domain-match and the PSL boundary check + are separate requirements) and a vendor cookie using `Expires` is + normalized to its Max-Age equivalent (both present → `Max-Age` wins, + per RFC 6265); a normalized lifetime exceeding the ceiling rejects + the whole operation batch; `Max-Age` at most **31,536,000 seconds** (the vendor's one-year cap — the earlier 396-day figure exceeded it); size ≤ **512 bytes** (DataDome's current Fastly-module limit; 4 KiB was ours, not theirs). @@ -254,6 +267,15 @@ degree of freedom is closed: **sign-off item 23** — the carve-out is _pending ratification_, not ratified, and the permission inventory's cookie deferral stands until it closes. No other request filter inherits the cookie capability. +- **The incoming `X-DataDome-ClientID` request header is owner-only, + like the cookie.** DataDome prioritizes the header over the cookie, + so leaving it in the shared request would hand other integrations and + upstream routing the same identifier the cookie boundary strips: core + **extracts it into the DataDome-only view and removes it from the + shared request** before integrations and upstream routing run — + it joins `RedactedRequestView`'s enumerated strip set (providers + spec) — and only DataDome-returned overlay data reaches the + publisher, never the raw browser-supplied header. - **Request-header pointers are a positive, enumerated allowlist.** "Documented enrichment headers" is not enforceable; the registration enumerates the exact names from the **checked-in allowlist file @@ -272,6 +294,19 @@ degree of freedom is closed: routing-authority fields — is rejected by name and by class: a compromised endpoint must not replace origin credentials, inject `ts-ec`, or spoof client location. +- **Browser-response headers are a decision-scoped positive allowlist + too** — request pointers were enumerated, response headers were not, + leaving either the six-field ordinary registry (which would reject a + challenge's `Location`) or an open door. Normatively, per decision: + a _Respond_ (challenge/deny) may set exactly `Location` (replace; + 3xx only), `Content-Type` (its own body, per the representation rule + below), `Cache-Control`/`Pragma` (through the restricted merge; the + invariant pass still runs last), the typed security cookie (above), + and the vendor response headers enumerated in the **response section + of `datadome-header-allowlist.md`**; a _Continue_ may set only the + typed cookie and those enumerated vendor headers. Everything else is + rejected — the atomic-302 example's `Location` is hereby admitted + rather than assumed. - **Representation rules are decision-scoped.** A _Respond_ decision (challenge/deny) **owns its body** and may set representation headers (`Content-Type`, encoding, validators) for it — the hook's diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index b9614430f..bf3d57e9a 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -435,7 +435,14 @@ and the fail-closed marker: clearing a previously positive permission — uses a narrow **permission-exempt suppression-decision read** exposing only the family ID and authority metadata (an undeclared exempt read was the - alternative, and skipping it leaves stale S2S authority). + alternative, and skipping it leaves stale S2S authority). Absence is + **one-shot per positive summary**: writing the entry also retires the + summary that justified it (recorded as retired-at), so later + signal-less requests find no positive authority and write nothing — + otherwise each would re-observe and extend the denial forever — and + the entry's `valid_until` is capped by the retired authority's own + original `valid_until` (absence retires an authority horizon; it does + not outlive it). **Policy-only tightening writes nothing**: a policy edit is not a user signal (§4.2 trigger 3), and a signal-less request after granted→denied must not create sticky user suppression that a policy @@ -450,8 +457,7 @@ and the fail-closed marker: when its evidence timestamp is **newer than or equal to** the stored entry's; ties resolve to the more restrictive state. So a delayed grant with `LastUpdated = 100` never clears a suppression whose - refusal carried `200`, while a genuine re-consent at `300` does. The - **Every suppression entry carries its own `valid_until`, derived from + refusal carried `200`, while a genuine re-consent at `300` does. **Every suppression entry carries its own `valid_until`, derived from its evidence class's TTL, and an expired entry is inert** — treated as cleared without a write, lazily garbage-collected. Without this, an expired TCF refusal under a `granted` baseline would deny forever: @@ -467,9 +473,14 @@ and the fail-closed marker: within the skew window; cross-source comparison uses the authoritative timestamp where one exists, else the observation timestamp, ties restrictive), by stored cause: **opt-out from a timestamp-less - source** — cleared only by a grant with an authoritative timestamp - newer than the suppression's observation (sticky opt-out, sign-off - 16); **TCF refusal** — cleared by any regime-accepted grant with newer + source** — within its lifetime, cleared only by a grant with an + authoritative timestamp newer than its observation; its lifetime is + the ordinary consent-TTL `valid_until`, at which it goes inert + automatically (**TTL-sticky** — the one rule chosen among three that + circulated: not user-sticky-forever, and not the migration spec's + former "irreversible artifact requiring administrative clear", which + is superseded; administrative clear remains an optional early exit — + sign-off 16); **TCF refusal** — cleared by any regime-accepted grant with newer authoritative evidence; **malformed-present / absence** — cleared by any regime-accepted valid grant with newer evidence, including a timestamp-less grant whose first-seen is newer (these causes are not @@ -537,7 +548,13 @@ and the fail-closed marker: row and the strong authority-state record, in a fixed order with defined intermediate states: (1) the row commits at revision _r_ (generation-CAS); (2) the authority-state record CAS-updates its - summary to _r_. **Revision _r_ is committed — usable by S2S, visible + summary to _r_ — and that transition **rejects regression**: an + incoming revision lower than the stored one is refused (a delayed + commit for r2 arriving after r3's must not restore older authority or + an older `valid_until`), and an equal revision is idempotent and must + be payload-equivalent (a mismatch at equal revision is a hard error, + not a merge). The r2-row → r3-row → r3-authority → delayed + r2-authority schedule is a named test. **Revision _r_ is committed — usable by S2S, visible to the absence decision — only when the strong record reports it**; a row at _r_ whose summary still reads _r−1_ is simply uncommitted detail, and a crash between the writes leaves a recoverable state (the @@ -649,6 +666,17 @@ a change to this table. Applicable_ = grant-class; absent = nothing; a non-applicable section's fields grant nothing (their opt-outs still count, per step 2). +**Unknown section IDs contribute nothing — and bound what +embedded-GPC scanning can promise.** A section ID outside the pinned map +is ignored (its fields neither grant nor revoke; known sections in the +same string remain valid — an unknown _section_ is not a malformed +_family_, unlike a mapped section at an unpinned version). Consequence, +stated honestly: an embedded GPC bit inside an unknown section is +undetectable by a decoder that cannot parse it — global-GPC coverage is +bounded by the pinned map's currency, which is one reason snapshot +updates are reviewed spec changes. Mixed known/unknown strings resolve +per-section by these rules. + **Embedded GPC is mapped, not ignored.** The US sections carry `GpcSegmentIncluded` and `Gpc` fields; a request with embedded `Gpc = true` and no `Sec-GPC` header was previously unspecified despite @@ -783,8 +811,11 @@ migration story unresolvable (migration spec §2, rows 5 and 7). A **policy revision** has a defined identity: the canonical content digest of the `[permissions]` section (identity — republishing identical -policy yields the same digest) paired with the config-store activation -generation (ordering — strictly monotonic per instance). Provenance +policy yields the same digest) paired with the **config-store's globally assigned activation +version** (the `ts config push` version — one fleet-wide ordered +sequence, not a per-instance counter: "monotonic per instance" gave +generation 12 on one instance no relation to 12 on another, making +cross-instance provenance comparison undefined). Provenance stores both; comparisons order by generation and equate by digest, so a rollback is a _new_ generation carrying an _old_ digest, with defined semantics on both axes. diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index 05e231e95..95d2897e9 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -182,8 +182,9 @@ NOT ship without a caller: - `IdentityInput.permissions` / `IdentityInput.consent` (ignored by all built-ins), - `DeviceProvider::required_permissions` / `GeoProvider::required_permissions` - — dropped entirely, not deferred: §5 explains why these two kinds cannot - be permission-gated at all. + — dropped entirely, not deferred: §5 explains why — geo _cannot_ be + gated (circular), and device _could_ be but deliberately is not (a + recorded decision, not an impossibility). If a future feature needs one of these, it arrives with that feature. @@ -254,9 +255,10 @@ spy-provider test pins the split: with `store-on-device` unset, `generate` is never called while a withdrawal request still parses the cookie and writes tombstones. -**A generated identity is not active until its graph row commits.** No -cookie write, no egress, no auction use may observe a minted identifier -before its graph row (with provenance, §6.1) has committed — PR #838 let a +**A generated identity is not active until its authority-state record +commits** (the two-record commit point — the graph row alone is not +activation). No cookie write, no egress, no auction use may observe a +minted identifier before both writes have committed — PR #838 let a generated EC reach an auction before finalization refused the cookie, producing an identity that existed for one request and nowhere else. The normative order is: gate → `generate` → graph-row commit → @@ -300,7 +302,21 @@ variant**. Therefore: anywhere — an N+1 fleet must never run rowless classification), and (2) an idempotent **stub-backfill scan** has stamped a minimal authority-state existence stub onto **every existing identity row** - (legacy and N+1-minted alike). Only then does the invariant hold: + (legacy and N+1-minted alike) — and "every" is established by an + actual algorithm over an eventual store, not asserted: after N+2 + convergence at time T, wait the backend's **documented listing settle + window** (a listing-completeness bound is a declared adapter + capability; a backend that cannot bound listing visibility cannot host + this migration), then scan repeatedly until **two consecutive full + passes discover zero unstubbed rows**; the flag value attests the + watermark, pass count, and settle window. Rows minted after T carry + records by protocol. And because no scan over an eventual store is + provably perfect, misses are **reconciled, not fatal**: a per-prefix + withdrawal entry (below) doubles as pending intent — if a real row for + a withdrawn suffix ever surfaces, core **promotes** the entry to a + full family revocation on that row's family at first sight, so a + missed row cannot quietly keep S2S egress after its cookie was + withdrawn. Only then does the invariant hold: every row-backed identity has an authority-state record under its derivable family ID, in the globally-strong class — so _rowless_ = flag set AND the strong read finds **no record** for the cookie's @@ -319,8 +335,13 @@ variant**. Therefore: matched version) is **expired and replaced by a fresh mint through the ordinary graph-backed path** when permissions allow; continuity is deliberately not preserved (migration matrix row 13, sign-off 21). An - unverifiable cookie (including the declared roaming false-negative) is - simply expired. + unverifiable cookie (including the declared roaming false-negative) + gets **cookie-only expiry — a disclosed best-effort residual, not a + buried one**: admission rules forbid durable records for unverified + values, so if the expiry response is lost the cookie survives and may + resurface on the old network; every re-presentation re-attempts + expiry. The residual is bounded to graphless-era cookies from changed + networks and is **sign-off item 29**. - **Rowless withdrawal writes into one capped per-prefix record, then expires the cookie** — durable (cookie-only expiry is best-effort: a lost response leaves the "withdrawn" cookie usable), and **bounded in @@ -337,13 +358,18 @@ variant**. Therefore: entry (or the saturated record) and stays dead. Row-backed withdrawal is untouched: full-graph-key family records, one derivation everywhere. -- **Negative-record creation has admission rules everywhere**: - suppression and family records may be written only for (a) an - existing family — the authority-state record exists on a strong - read — or (b) a **verified** identifier (`verify`); a fabricated, - unverifiable cookie writes nothing. This bounds record creation to - real identities plus the writer's own evidence, and per-prefix - rate limits apply to the rowless path above. +- **Negative-record creation has admission rules everywhere — and + rowless identifiers get no per-family records at all.** Durable + suppression and family-revocation records may be written only for an + **existing, row-backed family** (authority-state record present on a + strong read). A rowless identifier — even a _verified_ one — creates + none: verification authenticates only the prefix, so per-identifier + records would let one prefix-holder fabricate unlimited suffixes into + unlimited strong-storage records (rate limits slow creation; they do + not bound cardinality), and a rowless identity has no authority to + suppress anyway. **The capped per-prefix withdrawal record is the + entirety of rowless negative state.** Fabricated, unverifiable + cookies write nothing anywhere. **Egress is typed, not policed.** The inventory-and-denylist test (permission model spec §7) is a backstop, but conventions do not survive @@ -368,7 +394,7 @@ assertion**: the current filter/proxy inputs expose the raw request snapshot redaction cannot undo that. The contract: integration-facing request access moves to a typed **`RedactedRequestView`** whose stripped set is enumerated — the `ts-ec` cookie and every `ts-*` cookie, `x-ts-*` -identity/consent headers, and the EIDs header — with identity reachable +identity/consent headers, the incoming `X-DataDome-ClientID` header (owner-only, hook spec §4a), and the EIDs header — with identity reachable only through a scoped `AuthorizedIdentity` parameter; the raw-request filter/proxy interfaces are migrated in the **same PR** as the typed egress boundary (they are the same boundary), and the tests are @@ -563,6 +589,7 @@ the bounded suffix: | Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | | Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | | Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | +| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8), saturation flag, CAS version, created-at, `valid_until` ≥ the maximum cookie lifetime; readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | | Deployment metadata | `m` + fixed metadata name (fixed-width grammar; schema floor, graphless-migration flag) | Write-once/CAS class; value carries schema version, state, epoch, set-at; the graphless flag's lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded | | Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | | Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | @@ -586,10 +613,19 @@ rather than asserted (an earlier `f` tag was itself a hex digit) — then a **4-character provider code from the checked-in, append-only, never-reused registry `docs/superpowers/specs/provider-code-registry.md`** (allocation is a reviewed commit; codes are immutable and never recycled, including for retired -providers), then the suffix. Segment boundaries are positional, so no -segment can contain or escape a delimiter, prefix queries are plain -string prefixes on every backend, and a grammar-disjointness test covers -every class against the legacy grammar. (hmac verbatim keys remain the +providers), then the suffix. The **complete physical constructor set** (logical +`fam/`-style sketches elsewhere are notation for these): `i` + +provider-code(4) + suffix(≤123) for rows; `r`/`s` + family-id(64 +lowercase hex — family IDs are canonically SHA-256 over the derivation +input) for revocation/authority records (no provider code: the family id +already encodes derivation); `w` + provider-code(4) + prefix(64 hex) for +rowless withdrawal; `x` + family-id(64) for transactions; `m` + +name(16, `[a-z0-9-]`, right-padded `-`) for deployment metadata. Maximum +physical key length **128 bytes**; every class has a total parser, and +segment boundaries are positional, so no segment can contain or escape a +delimiter, prefix queries are plain string prefixes on every backend, +and a grammar-disjointness test covers every class pair plus the legacy +grammar. (hmac verbatim keys remain the reserved exception, with the 64-hex cluster prefix at position zero.) **Wire schemas** (JSON, like identity rows; every class carries a schema @@ -607,8 +643,15 @@ spec's absence/replay decisions consume — a reduced schema cannot reproduce them**): kind (user evidence vs policy baseline), grant basis/source class, policy revision (digest + activation generation), `valid_until`, provenance revision, evidence timestamp, and the -per-permission **semantic digest with its pinned first-seen/ -first-normalized timestamps** (anti-replay); record level: family ID, +— because a _single current_ digest cannot uphold +"re-presentation never advances first-seen" (grant A → refusal B → +replayed A would look novel once B displaced A's slot) — a **bounded +replay history keyed by (source class, semantic digest)**: pinned +first-seen/first-normalized entries retained for at least the maximum +evidence/suppression horizon, capacity-capped at 16 per +permission·source (in-horizon entries are never evicted; cap saturation +fails restrictive — novel values cannot grant until the horizon +passes); record level: family ID, CAS version counter, schema version, stub marker (backfill, §5); unknown-field and range validation apply like every class (strong class, permission-exempt writes per the permission spec's inventory; revisions are app-level counters because diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index f9bdbb36f..3cbf9fae3 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -32,30 +32,30 @@ and whether that is a preservation or a declared change. **Silent changes are defects.** PR #838 changed six of these without declaring any; each was discoverable only because a deleted test had pinned the old behavior. -| # | Decision (today) | After epic | Status | -| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | -| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | -| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | -| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | -| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | -| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | -| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | -| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — opt-out effects are more protective; the same fields' not-opted-out values can also **newly grant P4**, which is not (both effects classified in permission spec §4.5) | -| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | -| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | -| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | -| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | -| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | -| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | -| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | -| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | -| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | -| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | -| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | -| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | -| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | -| 13 | Cookies minted by graphless deployments have no graph row | Rowless proof via the strong authority-state class under the graphless-migration flag (providers spec §5); verified cookies are expired and re-minted without continuity; **rowless withdrawal writes an exact-cookie family record, then expires** — full-key derivation, no prefix mechanism | **Declared** — pre-existing identities restart rather than carry over | +| # | Decision (today) | After epic | Status | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | +| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | +| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | +| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | +| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | +| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — opt-out effects are more protective; the same fields' not-opted-out values can also **newly grant P4**, which is not (both effects classified in permission spec §4.5) | +| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | +| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | +| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | +| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | +| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | +| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | +| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | +| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | +| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | +| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | +| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | +| 13 | Cookies minted by graphless deployments have no graph row | Rowless proof via strong-class records under the graphless-migration flag (N+2 convergence + attested stub-backfill first — providers spec §5); verified cookies expire and re-mint without continuity; **rowless withdrawal writes into the capped per-prefix `w` record, then expires** (exact-cookie family records are superseded); unverifiable roaming cookies get disclosed cookie-only expiry (sign-off 29) | **Declared** — pre-existing identities restart rather than carry over | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -203,7 +203,11 @@ Requirements: other adapter is unavailable or needs a new primitive. Ratifying before at least one adapter qualifies risks an epic with no selectable identity provider, so Fastly qualification (or an - explicit decision to proceed without it) gates ratification. + explicit decision to proceed without it) gates ratification — + together with **filling the PSL snapshot reference** + (`psl-snapshot-ref.md` is a placeholder; ratification cannot + reproduce the cookie-domain computation it approves until the + vendored commit is recorded) and creating the §8 decision records. 3. **Revocation-eligible storage is a per-adapter gate, and ungated adapters migrate stateless.** Identity features require the adapter's strong-consistency rows in the capability matrix (providers spec §7) @@ -388,12 +392,12 @@ global honoring of opt-out signals is unconditional. maximum cookie/row lifetime plus rollout skew** is the **only retirement-readiness** bar for a legacy provider ("trending to ~zero" is not evidence; a yearly visitor - is not churn), . The telemetry set also includes: graph read/commit failures, + is not churn). The telemetry set also includes: graph read/commit failures, stored-provenance denials, and schema-migration failures. **Each rollout-gate metric ships with a threshold, an evaluation window, and a named action** (pause rollout / roll back / block retirement) in the migration guide — a metric with a "healthy range" but no action is dashboard decoration; the two already - specified (legacy-reader quiet period, rewrite failures) are the + specified (legacy-reader quiet period) are the pattern the rest follow. 3. Startup logs always print: selected provider per concern, whether geo is live, the effective default baseline, and the count of granted-without- @@ -451,40 +455,42 @@ global honoring of opt-out signals is unconditional. These are decisions this spec set makes that #838 had not already made (or made differently). **Implementation is blocked while any row is `open`**; -each row needs an owner, a status, and a link to its decision record. -Decision records live as files under -`docs/superpowers/specs/decisions/` (one per row, `NN-title.md`, -recording the decision, the deciders, and the date) — the table links -them as rows close; an unratified row reverts to open, not to silently +each row is a **decision, not an assignment**: the table tracks the +decision and its record; _who_ decided is captured inside the record +itself (`docs/superpowers/specs/decisions/NN-title.md` — the decision, +the deciders, the date). The Decision-record column holds the link (`—` +while open); an unratified row reverts to open, not to silently implemented. -| # | Decision | Where | Owner | Status | -| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------------- | ------------------------------------------- | -| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | maintainers + legal | open | -| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | maintainers + legal | open | -| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | maintainers + legal | open | -| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | maintainers + product | open | -| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | maintainers + legal | open | -| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | maintainers + legal | open | -| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | maintainers + product | open | -| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | maintainers | open | -| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | maintainers | superseded by descope (ratify the deferral) | -| 10 | Session-cookie exemption question | hook §3 | maintainers + legal | deferred with item 9 | -| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | maintainers + legal | open | -| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | maintainers + product | open | -| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | maintainers + product | open | -| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | maintainers + product | open | -| 16 | Sticky opt-out for timestamp-less sources: a GPP/USP-only re-consent does not restore authority without a timestamped grant | permission §4.3 | maintainers + legal | open | -| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | maintainers + legal | open | -| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | maintainers + legal | open | -| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | maintainers + product | open | -| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | maintainers + product | open | -| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | maintainers + product | open | -| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | maintainers + legal | open | -| 23 | **Open question, not ratified**: may DataDome's security identifier (tag injection, `datadome` cookie, `X-DataDome-ClientID` read and vendor egress) operate outside the permission model? The decision must enumerate exactly which consumers may observe the cookie/ClientID — the enumerated observers are the security channel **and the publisher origin, which receives `X-DataDome-ClientID` via the upstream overlay** — "owner-scoped overlay" names the mechanism, this row names the recipient — plus retention and whether TS withdrawal expires it | hook §4a; permission §7 | maintainers + legal | open | -| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | maintainers + legal | open | -| 25 | Batch-sync stored-jurisdiction maximum age (consent-TTL horizon): a mover into GDPR stops old-rule egress at the horizon; a mover out is denied until a live visit | permission §7 | maintainers + legal | open | -| 26 | Embedded GPP GPC maps to the destructive global opt-out (header-OR-embedded aggregation) | permission §4.5 | maintainers + legal | open | -| 27 | Proxy-mode minimal opt-out extraction (decode only §4.5-mapped opt-out fields; no grants) | permission §4.4 | maintainers + legal | open | -| 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance | hook §4a; `datadome-header-allowlist.md` | maintainers + product | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | maintainers + legal | open | +| # | Decision | Where | Decision record | Status | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------- | ------------------------------------------- | +| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | — | open | +| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | — | open | +| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | — | open | +| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | — | open | +| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | — | open | +| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | — | open | +| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | — | open | +| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | — | open | +| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | — | superseded by descope (ratify the deferral) | +| 10 | Session-cookie exemption question | hook §3 | — | deferred with item 9 | +| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | — | open | +| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | — | open | +| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | — | open | +| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | — | open | +| 16 | Timestamp-less opt-out suppression is **TTL-sticky**: within its consent-TTL lifetime only a newer timestamped grant clears it; at `valid_until` it goes inert automatically (not user-sticky-forever, not administratively sticky — administrative clear is an optional early exit) | permission §4.3 | — | open | +| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | — | open | +| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | — | open | +| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | — | open | +| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | — | open | +| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | — | open | +| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | — | open | +| 23 | **Open question, not ratified**: may DataDome's security identifier (tag injection, `datadome` cookie, `X-DataDome-ClientID` read and vendor egress) operate outside the permission model? The decision must enumerate exactly which consumers may observe the cookie/ClientID — the enumerated observers are the security channel **and the publisher origin, which receives `X-DataDome-ClientID` via the upstream overlay** — "owner-scoped overlay" names the mechanism, this row names the recipient — plus retention and whether TS withdrawal expires it | hook §4a; permission §7 | — | open | +| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | — | open | +| 25 | Batch-sync stored-jurisdiction maximum age (consent-TTL horizon): a mover into GDPR stops old-rule egress at the horizon; a mover out is denied until a live visit | permission §7 | — | open | +| 26 | Embedded GPP GPC maps to the destructive global opt-out (header-OR-embedded aggregation) | permission §4.5 | — | open | +| 27 | Proxy-mode minimal opt-out extraction (decode only §4.5-mapped opt-out fields; no grants) | permission §4.4 | — | open | +| 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance | hook §4a; `datadome-header-allowlist.md` | — | open | +| 29 | Unverifiable roaming rowless cookies receive best-effort cookie-only expiry (admission rules forbid durable records for unverified values); a lost response can leave the cookie usable on the old network until re-presented | providers §5 | — | open | +| 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, and observability all in scope | providers §5 | — | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | — | open | diff --git a/docs/superpowers/specs/datadome-header-allowlist.md b/docs/superpowers/specs/datadome-header-allowlist.md index 9acb8c225..fecd3880a 100644 --- a/docs/superpowers/specs/datadome-header-allowlist.md +++ b/docs/superpowers/specs/datadome-header-allowlist.md @@ -8,3 +8,15 @@ name is a reviewed commit to this file and a spec change. | Header | Direction | Scope | | --------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------- | | `X-DataDome-ClientID` | response → upstream overlay | Owner-scoped overlay only; never the shared request view; vendor egress governed by sign-off item 23 | + +## Response-direction allowlist (browser-response pointers) + +Headers a DataDome decision may set on the outgoing response, beyond +the typed security cookie (hook spec §4a). Empty rows below the base +set mean: nothing else is accepted until a reviewed commit adds it. + +| Header | Decision | Semantics | +| ------------------------- | ---------------------------- | ------------------------------------------- | +| `Location` | Respond (3xx) only | replace | +| `Content-Type` | Respond only (owns its body) | replace | +| `Cache-Control`, `Pragma` | Respond only | restricted merge; invariant pass still last | diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index 9f2b04669..07e195664 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -266,3 +266,23 @@ text-added unless marked otherwise. | P1 publisher origin as ClientID observer | text-added: sign-off 23 names the recipient | | P2 batch (flag wire contract + `m` class; N+1 test alignment; AuthorityRefresh read-set; expired-live fallback row; skew algorithm; policy-revision identity = digest + activation generation; provider-code registry + PSL reference files created; decoder prerequisite noted; sign-off rows 25–28; adapter qualification as pre-ratification prerequisite; telemetry residue swept) | text-added | | P3 batch (hook fragment; provenance comma; host-equivalents in tests; client-draft Axum claim; this vocabulary change) | text-added | + +## Round 14 — review at 3523b362 + +All rows text-added per the R13 vocabulary. + +| Finding | Status | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| P1 authority CAS can regress revisions | text-added: reject `incoming < stored`, equal ⇒ idempotent + payload-equivalent, named delayed-r2 test schedule | +| P1 single digest breaks anti-replay | text-added: bounded replay history per (source, digest), horizon-retained, cap-16, saturation fails restrictive | +| P1 three opt-out lifetime contracts | text-added: **TTL-sticky chosen** — inert at consent-TTL `valid_until`; migration "irreversible artifact" wording superseded; sign-off 16 rewritten | +| P1 backfill invariant unestablishable | text-added: settle-window + two-idle-pass convergence + attestation in the flag value, listing-completeness as a capability, and **promotion reconciliation** (late-surfacing row for a withdrawn suffix upgrades to full family revocation) | +| P1 rowless record class incoherent | text-added: `w`-class end to end (grammar, wire schema, linearizable CAS, retention ≥ cookie lifetime, N+1 fail-closed reader); migration row 13 aligned | +| P1 roaming best-effort withdrawal | text-added: disclosed residual, re-attempt on re-presentation, sign-off 29 | +| P1 verified-suffix suppression amplification | text-added: durable per-family records require row-backed families; rowless negative state is exclusively the capped prefix record | +| P1 DataDome response boundary open | text-added: decision-scoped response allowlist (Location/Content-Type/Cache-Control/Pragma per Respond; cookie + enumerated vendor headers per Continue); allowlist file gains a response section | +| P1 incoming ClientID unstripped | text-added: extracted into the DataDome-only view, removed from shared request and upstream routing, joins RedactedRequestView's strip set | +| P1 normal-vs-conditional hit divergence | text-added: persisted post-hook finals serve **all** hit paths; identity by construction, no mutator-determinism assumption | +| P2 batch (absence one-shot with summary retirement + horizon cap; global config-push activation version; unknown GPP sections contribute nothing with the embedded-GPC coverage bound stated; complete byte-level constructors incl. `w`/`m`, 128-byte cap, family-id encoding; RFC 6265 domain-match + Expires normalization; shared fail-closed cache parser + `Vary: *`; NAT saturation sign-off 30; PSL fill + decision records + adapter qualification as explicit ratification gates) | text-added | +| P3 batch (§5 activation lead fixed to the commit point; §4 device wording; typo/punctuation/metric residue; client resv notation; old DataDome spec carries a supersession banner now) | text-added | +| Sign-off table restructured **decision-centric** (Owner column → Decision-record link; deciders live in the records) per maintainer direction | text-added | From 9a8596ee80917c04e8e50d68e3ba10f35c17725c Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Mon, 3 Aug 2026 12:38:37 -0700 Subject: [PATCH 16/24] Address fifteenth review: rollback-safe rowless proof, admission for observed rows, and vendor-faithful DataDome cookie handling P1 fixes: - N+1 rollback can no longer invalidate the rowless proof: any N+1 startup that observes the graphless flag active CASes it to suspended (rowless classification stops fleet-wide); re-activation after roll-forward requires complete re-attestation over the gap window; the N+2 -> N+1-mint -> N+2 schedule is a named test. - The admission rule gains the observed-row arm, adopted verbatim from the review: a successful row read is safe admission evidence (an eventual not-found is not), and the permission-exempt sequence - derive family ID -> create-if-absent minimal stub with no positive authority -> commit family revocation -> deny all use in between -> only then expire the cookie - unblocks both first-upgrade withdrawal of untouched v1 rows and promotion of late-surfacing rows. - The w record protocol is enforceable end to end: abstract capability row (globally strong reads + linearizable CAS + the bounded listing-visibility window, all graphless-migration eligibility gates, retention >= max cookie lifetime), four runtime-failure rows (read fail = indeterminate; write fail = cookie retained, client retries; saturation semantics; promotion failure = deny until commit), and the migration-window rule that every HMAC row discovery consults w state before use. Saturation promotes listed suffix-hash matches only - it never blanket-revokes row-backed identities, and the beyond-cap overflow residual is declared. - Replay-history saturation is a serialized state machine: 16 semantic-state slots where a same-semantics TCF renewal with newer LastUpdated updates its slot in place (ordinary renewals never consume capacity - the 16-genuine-renewals exhaustion is gone), plus a rolling restrictive slot outside the cap so a seventeenth refusal is stored under normal recency rules rather than dropped or replay-advanceable; saturated/saturated_until serialized, automatic slot-expiry recovery, first-class metric, and the cap's denial behavior is sign-off 31. - The migration spec's irreversible-artifact list is corrected: timestamp-less suppression is TTL-bounded and goes inert automatically - the 'irreversible, administrative clear required' entry contradicted the chosen TTL-sticky rule. - GPP sections 24-27 are demoted to reserved-pending-official-schema in both the map and the vendored snapshot: the official registries publish layouts only through 23, so IDs without binary layouts are not reproducible - those states behave national-only, and the official-coverage claim is retracted. - Both DataDome cookie-return forms (Set-Cookie and the session-by-header X-Set-Cookie field) lower into the one typed datadome cookie operation with identical validation; the X-Set-Cookie field itself is never forwarded - previously it would have forked implementations or failed the whole challenge open. - The Respond representation rule narrows to Content-Type only; encoding and validator fields stay reserved even for challenge bodies, with vendor needs arriving as reviewed allowlist-file additions - the prior wording could flip enforcement to fail-open on an ambiguity. - The cache merge preserves unknown snapshot directives verbatim (mutations still drop unknowns), folds Expires into the freshness bound, and forbids mutations introducing max-age/s-maxage where the snapshot had no upper bound (RFC 9111 5.2.3/5.3). P2/P3: HEAD serves the persisted GET artifact (parity by construction under RFC 9111 4.3.5); the cache revision tuple has fleet-stable identity (registry content hash, global config push version, build-time invariant constant); the 300-second tie preserves the winner's complete tuple (no near-window grant ratchet); the physical grammar is consistent (suffix <= 123; m + 2-digit registry index replaces padded names and their foo/foo- aliasing; the tag enumeration includes w and m; the provider-registry note acknowledges hmac codes in w keys; family-ID derivation is defined with domain tag tsfam1| and cross-language vectors); the graphless migration has an operational runbook with the correct section pointer, abort/suspension, re-attestation, and quiet-period clearing; the policy digest is defined (tspol1|, canonical JSON, vectors); the DataDome pointer protocol has a total parser contract (case, OWS, duplicates, count/byte limits, cookie-source priority, fail-open semantics); Pragma is dropped from the response allowlist (no standardized response meaning); the wire-schema sentence is restructured; row 3e's phrase is completed; and the sign-off rows are renumbered into order with row 31 added. --- ...integration-response-header-hook-design.md | 72 ++++++++---- .../2026-07-30-permission-model-design.md | 33 ++++-- .../2026-07-30-pluggable-providers-design.md | 107 +++++++++++++----- ...07-30-provider-migration-rollout-design.md | 78 ++++++++----- .../specs/datadome-header-allowlist.md | 4 + .../specs/gpp-registry-snapshot.md | 9 +- docs/superpowers/specs/pr986-review-ledger.md | 18 +++ .../specs/provider-code-registry.md | 6 +- 8 files changed, 234 insertions(+), 93 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 574a9b268..751922967 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -90,7 +90,16 @@ mutators to the outbound response for HTML document responses it processed. invalid `Cache-Control` syntax normalizes to the most restrictive reading; duplicate directives keep the strongest; quoted and unquoted forms are equivalent; conflicting `max-age` values keep the smallest; - unknown extension directives are dropped at merge; and `Vary: *` is + unknown extension directives are dropped **from mutations only — + unknown directives already in the snapshot are preserved verbatim** (a + downstream cache may honor a restrictive extension TS does not + recognize; dropping it would weaken origin policy, RFC 9111 §5.2.3); + `Expires` participates in the freshness bound (effective freshness = + the minimum across `max-age`, `s-maxage`, and the `Expires`-derived + lifetime) and a mutation may **not introduce `max-age`/`s-maxage` + where the snapshot supplied no upper bound** — HTTP prefers `max-age` + over `Expires` (RFC 9111 §5.3), so introducing one would override an + origin's shorter or already-expired `Expires`; and `Vary: *` is treated as uncacheable-by-shared-caches (no-store-equivalent for the invariant). Conformance fixtures cover each rule. Middle-stage placement also keeps the earlier property: an integration mutation is not silently stripped @@ -209,17 +218,17 @@ mutators to the outbound response for HTML document responses it processed. Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by (integration-registry revision, config revision, invariant revision)**; a 304 — like a **normal cache hit** (§3a), which serves the same persisted finals rather than re-running mutators — re-emits them only when all three match the serving instance; a mismatch is a **cache miss**. Identity of normal-hit and conditional-hit metadata holds by construction (one persisted artifact serves both), not by an unsupported determinism claim about mutators. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | -| `HEAD` of a processed document | **Yes — header parity with GET is mandatory** (a cache may refresh stored GET metadata from HEAD, and divergent CSP/privacy metadata between the two is a leak) | -| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | +| Response | Hook runs? | +| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by a fleet-stable tuple** — the integration-registry revision is a content hash over the ordered (integration ID, version) list; the config revision is the config store's globally assigned push version; the invariant revision is a build-time spec-version constant — local counters would collide across instances and binaries; a 304 — like a **normal cache hit** (§3a), which serves the same persisted finals rather than re-running mutators — re-emits them only when all three match the serving instance; a mismatch is a **cache miss**. Identity of normal-hit and conditional-hit metadata holds by construction (one persisted artifact serves both), not by an unsupported determinism claim about mutators. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | +| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals persisted | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). @@ -254,8 +263,16 @@ degree of freedom is closed: spec-pinned pointer allowlist starting at ClientID-only against DataDome's mandatory response-directed mapping set — that reduction needs explicit product **and vendor** acceptance: sign-off item 28; - a violating operation is rejected whole (the batch rule). Every - `ts-*` name is rejected. **Read is owner-only, and the strip inventory is exhaustive, not + a violating operation is rejected whole (the batch rule). **Both + documented cookie-return forms lower into this one typed operation**: + an ordinary `Set-Cookie` from the vendor response _and_ the + session-by-header form (`X-DataDome-X-Set-Cookie: true` request flag → + updated value returned in the vendor's `X-Set-Cookie` response field) + are parsed and validated identically as the typed `datadome` cookie — + and the `X-Set-Cookie` field itself is **never forwarded** to the + browser (unmapped, it would either fork implementations or invalidate + the whole batch and silently fail the challenge open). Every `ts-*` + name is rejected. **Read is owner-only, and the strip inventory is exhaustive, not integration-scoped** — the browser sends `datadome` in the ordinary `Cookie` header, so it is removed from **every non-DataDome surface**: other integrations' request views, publisher-origin proxy forwarding, @@ -276,6 +293,14 @@ degree of freedom is closed: it joins `RedactedRequestView`'s enumerated strip set (providers spec) — and only DataDome-returned overlay data reaches the publisher, never the raw browser-supplied header. +- **The pointer protocol has a total parser contract** — adapters + cannot differ where malformed batches fail open: pointer names + compare case-insensitively with OWS/tab trimmed; duplicate pointer + names, invalid names, more than 16 pointers, or more than 4 KiB of + pointer payload render the batch invalid (→ Continue, the vendor's + fail-open); when both cookie sources arrive (header form and + `Set-Cookie`), the header form wins, matching the vendor's documented + priority. - **Request-header pointers are a positive, enumerated allowlist.** "Documented enrichment headers" is not enforceable; the registration enumerates the exact names from the **checked-in allowlist file @@ -300,18 +325,23 @@ degree of freedom is closed: challenge's `Location`) or an open door. Normatively, per decision: a _Respond_ (challenge/deny) may set exactly `Location` (replace; 3xx only), `Content-Type` (its own body, per the representation rule - below), `Cache-Control`/`Pragma` (through the restricted merge; the - invariant pass still runs last), the typed security cookie (above), + below), `Cache-Control` (through the restricted merge; the invariant pass + still runs last — `Pragma` is dropped from the allowlist: response + `Pragma: no-cache` has no standardized meaning, RFC 9111 §5.4), the typed security cookie (above), and the vendor response headers enumerated in the **response section of `datadome-header-allowlist.md`**; a _Continue_ may set only the typed cookie and those enumerated vendor headers. Everything else is rejected — the atomic-302 example's `Location` is hereby admitted rather than assumed. -- **Representation rules are decision-scoped.** A _Respond_ decision - (challenge/deny) **owns its body** and may set representation headers - (`Content-Type`, encoding, validators) for it — the hook's - representation reservation exists because ordinary mutators do not own - the body, and this one does. A _Continue_ decision may not touch +- **Representation rules are decision-scoped and narrow.** A _Respond_ + decision (challenge/deny) owns its body but may describe it with + **`Content-Type` only** — encoding and validator fields + (`Content-Encoding`, `ETag`, `Last-Modified`, digests) stay reserved + even for Respond: challenge bodies are simple and uncacheable, the + allowlist does not admit those fields, and ambiguity here decides + whether a challenge enforces or silently fails open (batch rejection → + Continue). If the vendor ever requires more, it arrives as a reviewed + allowlist-file addition. A _Continue_ decision may not touch representation metadata of publisher bytes. - **One global order:** core finalization → ordinary mutators → security effects → **final cache/privacy invariant pass, diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index bf3d57e9a..5aca80c88 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -406,7 +406,12 @@ and the fail-closed marker: GPC-carrying visitor whose v1 row has no family field and has never been backfilled — computes the same family ID that every future reader of that row computes, so the revocation record is discoverable even if the - writer crashes before ever touching the member row. A **random** ID + writer crashes before ever touching the member row — and the write is + admitted through the **observed-row sequence** (providers spec §5: + successful row read → create-if-absent stub with no positive + authority → family revocation → use denied in between), since an + untouched v1 row has no authority-state record for the plain + admission arm to find. A **random** ID would recreate the exact partial-withdrawal orphan this design exists to eliminate. Revocation writes one record keyed by the family ID; that single write is the withdrawal. Per-member tombstones are cleanup that @@ -697,9 +702,15 @@ section malformed-present (blocks grants, never withdraws). the state sections — `US/CA` ↔ 8, `US/VA` ↔ 9, `US/CO` ↔ 10, `US/UT` ↔ 11, `US/CT` ↔ 12, `US/FL` ↔ 13, `US/MT` ↔ 14, `US/OR` ↔ 15, `US/TX` ↔ 16, `US/DE` ↔ 17, `US/IA` ↔ 18, `US/NE` ↔ 19, `US/NH` ↔ 20, - `US/NJ` ↔ 21, `US/TN` ↔ 22, `US/MN` ↔ 23, **`US/MD` ↔ 24, - `US/IN` ↔ 25, `US/KY` ↔ 26, `US/RI` ↔ 27** (an earlier draft wrongly - claimed MD/IN/KY/RI had no section). A truncated map silently loses + `US/NJ` ↔ 21, `US/TN` ↔ 22, `US/MN` ↔ 23; **IDs 24–27 (MD/IN/KY/RI) are mapped as + _reserved-pending-official-schema_** — the public official registries + currently expose sections only through 23, so 24–27 have IDs but no + reproducible published binary layout; until the vendored snapshot can + carry an official layout, those four states behave as + no-section states (national section only) and the map does **not** + claim official-registry coverage for them (an earlier revision + claimed both "no section" and later "official through 27" — each + wrong in its own direction). A truncated map silently loses opt-outs — a Texas (16) or Maryland (24) sale opt-out must not vanish. **The current decoder is an explicit prerequisite gap**: it (and `iab_gpp` 0.1.2) supports sections 7–23 only and models `usnat` @@ -810,7 +821,10 @@ migration story unresolvable (migration spec §2, rows 5 and 7). ### 5.5 Policy revision activation A **policy revision** has a defined identity: the canonical content -digest of the `[permissions]` section (identity — republishing identical +digest of the `[permissions]` section — **defined**: SHA-256 with domain +tag `tspol1|` over the canonical JSON of the parsed policy (keys sorted, +UTF-8, defaults materialized, no insignificant whitespace), with +cross-language test vectors a conformance requirement (identity — republishing identical policy yields the same digest) paired with the **config-store's globally assigned activation version** (the `ts config push` version — one fleet-wide ordered sequence, not a per-instance counter: "monotonic per instance" gave @@ -934,8 +948,13 @@ Consumers of the resolved set in this epic: clamped — clamping re-freshens replays); expiry checks grant a grace of S (`expired` means `valid_until < now − S`); and two evidence timestamps within S of each other **compare equal**, which routes - the comparison to the tie rule (restrictive) — so a slightly - future-dated consent cannot out-order a just-observed opt-out; + the comparison to the tie rule (restrictive) — and the tie winner's + **complete tuple survives unchanged** (state, source, timestamp, + digest, `valid_until`): the losing grant's timestamp is never merged + into the surviving refusal, or a sequence of near-window grants would + ratchet the refusal's effective age forward and prolong it + indefinitely. A slightly future-dated consent cannot out-order a + just-observed opt-out; beyond-window future-dated records are **rejected as malformed**, and within the window a record's first normalized timestamp is pinned to its digest and never advanced by re-presentation (§4.3's anti-replay diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index 95d2897e9..41521835a 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -310,7 +310,16 @@ variant**. Therefore: this migration), then scan repeatedly until **two consecutive full passes discover zero unstubbed rows**; the flag value attests the watermark, pass count, and settle window. Rows minted after T carry - records by protocol. And because no scan over an eventual store is + records by protocol — **and rollback cannot silently break that + invariant**: an N+1 binary mints v1 rows with no records, so any N+1 + instance that starts while the flag is active **CASes it to + `suspended` at startup** (N+1 reads deployment metadata; this is one + of its reader duties). Suspended = rowless classification off + fleet-wide; re-activation after roll-forward requires a **complete + re-attestation** (a fresh scan to two idle passes covering the + rollback window's mints). The N+2 → rollback-to-N+1 → mint → + roll-forward-to-N+2 schedule is a named test proving those rows are + never classified rowless. And because no scan over an eventual store is provably perfect, misses are **reconciled, not fatal**: a per-prefix withdrawal entry (below) doubles as pending intent — if a real row for a withdrawn suffix ever surfaces, core **promotes** the entry to a @@ -320,7 +329,11 @@ variant**. Therefore: every row-backed identity has an authority-state record under its derivable family ID, in the globally-strong class — so _rowless_ = flag set AND the strong read finds **no record** for the cookie's - derived family ID. Graphless-era cookies never got a stub because + derived family ID — and, while the flag is active, **every HMAC row + discovery consults the prefix's `w` state before any live or S2S + use** (a pending or saturated entry means promotion-then-denial per + the runtime matrix, §6.2), so a withdrawn suffix cannot slip into use + through the row path during the window. Graphless-era cookies never got a stub because they have no row for the scan to find; no eventual read participates. The flag itself is specified: a named deployment-metadata key (write-once/CAS class), set by the §4.2 readiness step only on deployments that actually ran graphless @@ -360,9 +373,18 @@ variant**. Therefore: everywhere. - **Negative-record creation has admission rules everywhere — and rowless identifiers get no per-family records at all.** Durable - suppression and family-revocation records may be written only for an + suppression and family-revocation records may be written for an **existing, row-backed family** (authority-state record present on a - strong read). A rowless identifier — even a _verified_ one — creates + strong read) — **or for a positively observed real row**: a successful + row read is safe admission evidence (an eventual not-found is not), + and without this arm the first post-upgrade GPC request could not + revoke an untouched v1 row, and the promotion path could not promote a + late-surfacing row (neither has a stub by definition). The + permission-exempt sequence for an observed row: derive the family ID → + **create-if-absent a minimal strong stub carrying no positive + authority** → commit the family revocation → the identity is denied + all use between discovery and revocation commit → only then expire the + browser cookie. A rowless identifier — even a _verified_ one — creates none: verification authenticates only the prefix, so per-identifier records would let one prefix-holder fabricate unlimited suffixes into unlimited strong-storage records (rate limits slow creation; they do @@ -543,16 +565,20 @@ Startup validation (§6) covers configuration; this covers what happens when a healthy configuration meets an unhealthy runtime. Every row logs at `error` with a metric; none is silent: -| Failure | Behavior | -| ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | -| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | -| Authority-state commit fails after the row commit at mint | No cookie, no eligibility (the strong record never reported the revision); the orphan row expires by TTL; counted — **no recovery claim**, nothing can find it | -| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | -| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | -| Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | -| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | -| Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | +| Failure | Behavior | +| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | +| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | +| Authority-state commit fails after the row commit at mint | No cookie, no eligibility (the strong record never reported the revision); the orphan row expires by TTL; counted — **no recovery claim**, nothing can find it | +| `w` read fails (rowless path or migration-window row discovery) | Fail closed: the cookie/row is **indeterminate** — no use, no mint, no expiry this request | +| `w` write fails at rowless withdrawal | Cookie retained (withdrawal did not durably occur); durable client signal retries next presentation | +| `w` saturation encountered | Rowless cookies under the prefix are treated withdrawn (§5); later-discovered **real rows** promote only on a listed suffix-hash match — saturation never blanket-revokes row-backed identities, and overflow suffixes beyond the cap lose promotion (declared residual) | +| Promotion (listed suffix-hash matches a discovered row) fails | The row is denied all use until the promoted family revocation commits; retried on next observation | +| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | +| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | +| Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | +| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | +| Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | The **degraded-graph health signal** referenced above and by the withdrawal contract is a defined state machine, not a vibe: it is @@ -605,8 +631,10 @@ logical identity different physical keys on different adapters, breaking migration, shared storage, and parity; and Fastly's prefix queries reject both `/` and `:`, so no delimiter character is safely portable). Physical keys are **delimiter-free with fixed-width segments**: a -1-character class tag — `i` row, `r` family revocation, `s` suppression, -`x` transaction, every tag chosen **outside the hex alphabet** so no +1-character class tag — `i` row, `r` family revocation, `s` +authority-state, `x` transaction, `w` rowless prefix-withdrawal, `m` +deployment metadata (the full enumeration; earlier lists omitted `w` and +`m`), every tag chosen **outside the hex alphabet** so no generated key can begin with 64 hex characters, which is what makes disjointness from the legacy `{64hex}.{6alnum}` grammar _provable_ rather than asserted (an earlier `f` tag was itself a hex digit) — then @@ -615,12 +643,20 @@ never-reused registry `docs/superpowers/specs/provider-code-registry.md`** (allo codes are immutable and never recycled, including for retired providers), then the suffix. The **complete physical constructor set** (logical `fam/`-style sketches elsewhere are notation for these): `i` + -provider-code(4) + suffix(≤123) for rows; `r`/`s` + family-id(64 -lowercase hex — family IDs are canonically SHA-256 over the derivation -input) for revocation/authority records (no provider code: the family id +provider-code(4) + suffix(≤123 — the 128-byte total minus tag and code; +an earlier "suffix ≤128" was inconsistent with its own cap) for rows; `r`/`s` + family-id(64 +lowercase hex — family IDs are canonically +SHA-256 over a **defined derivation input**: the domain tag `tsfam1|`, +then the record-kind byte, then the provider code (4 bytes), then the +canonical graph-key bytes — concatenated in that order, UTF-8, no +separators beyond the tag's `|`; cross-language test vectors are a +conformance requirement) for revocation/authority records (no provider code: the family id already encodes derivation); `w` + provider-code(4) + prefix(64 hex) for -rowless withdrawal; `x` + family-id(64) for transactions; `m` + -name(16, `[a-z0-9-]`, right-padded `-`) for deployment metadata. Maximum +rowless withdrawal; `x` + family-id(64) for transactions; `m` + a **2-digit +registry-assigned index** for deployment metadata (a closed name +registry in this spec: `00` schema floor, `01` graphless-migration +flag — padding-based names aliased `foo` and `foo-`, so names are not +encoded in keys at all). Maximum physical key length **128 bytes**; every class has a total parser, and segment boundaries are positional, so no segment can contain or escape a delimiter, prefix queries are plain string prefixes on every backend, @@ -643,15 +679,25 @@ spec's absence/replay decisions consume — a reduced schema cannot reproduce them**): kind (user evidence vs policy baseline), grant basis/source class, policy revision (digest + activation generation), `valid_until`, provenance revision, evidence timestamp, and the -— because a _single current_ digest cannot uphold -"re-presentation never advances first-seen" (grant A → refusal B → -replayed A would look novel once B displaced A's slot) — a **bounded -replay history keyed by (source class, semantic digest)**: pinned -first-seen/first-normalized entries retained for at least the maximum -evidence/suppression horizon, capacity-capped at 16 per -permission·source (in-horizon entries are never evicted; cap saturation -fails restrictive — novel values cannot grant until the horizon -passes); record level: family ID, +and a **bounded replay history**, record-level, keyed by (source +class, semantic digest) — because a _single current_ digest cannot +uphold "re-presentation never advances first-seen" (grant A → refusal +B → replayed A would look novel once B displaced A's slot). Entries pin +first-seen/first-normalized timestamps, retained for at least the +maximum evidence/suppression horizon. Capacity and saturation are a +**serialized state machine, not a shrug**: 16 slots per +permission·source, each holding a _semantic state_ — a TCF renewal with +the same semantic result and newer `LastUpdated` **updates its slot in +place** (ordinary repeated renewals never consume capacity; replays +cannot advance the slot, their `LastUpdated` is not newer) — plus one +dedicated **rolling restrictive slot** outside the 16, so a seventeenth +refusal is stored under normal recency rules, never dropped and never +replay-advanceable. All slots holding distinct in-horizon states sets +`saturated: true` with `saturated_until` = the earliest slot +`valid_until`: while saturated, novel values cannot grant (fail +restrictive), restrictive evidence still lands in the rolling slot, +recovery is automatic as slots expire, and saturation is a first-class +metric — the cap and its denial behavior are **sign-off item 31**; record level: family ID, CAS version counter, schema version, stub marker (backfill, §5); unknown-field and range validation apply like every class (strong class, permission-exempt writes per the permission spec's inventory; revisions are app-level counters because @@ -723,6 +769,7 @@ Requirements: | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | | Deployment metadata (schema floor) | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | + | Rowless prefix-withdrawal (`w`) records | **Globally strong reads + linearizable CAS** (same bar as authority-state), plus the **bounded listing-visibility window** the backfill scan depends on — all three are eligibility gates for the graphless migration; retention ≥ maximum cookie lifetime | | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 3cbf9fae3..11c95bd50 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -32,30 +32,30 @@ and whether that is a preservation or a declared change. **Silent changes are defects.** PR #838 changed six of these without declaring any; each was discoverable only because a deleted test had pinned the old behavior. -| # | Decision (today) | After epic | Status | -| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | -| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | -| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | -| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | -| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | -| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | -| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | -| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — opt-out effects are more protective; the same fields' not-opted-out values can also **newly grant P4**, which is not (both effects classified in permission spec §4.5) | -| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | -| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | -| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | -| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | -| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | -| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | -| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | -| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | -| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | -| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | -| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | -| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | -| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | -| 13 | Cookies minted by graphless deployments have no graph row | Rowless proof via strong-class records under the graphless-migration flag (N+2 convergence + attested stub-backfill first — providers spec §5); verified cookies expire and re-mint without continuity; **rowless withdrawal writes into the capped per-prefix `w` record, then expires** (exact-cookie family records are superseded); unverifiable roaming cookies get disclosed cookie-only expiry (sign-off 29) | **Declared** — pre-existing identities restart rather than carry over | +| # | Decision (today) | After epic | Status | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | +| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | +| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | +| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | +| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | +| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — opt-out effects are more protective; the same fields' not-opted-out values can also **newly grant P4**, which is not protective — both directions are classified in permission spec §4.5 | +| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | +| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | +| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | +| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | +| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | +| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | +| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | +| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | +| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | +| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | +| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | +| 13 | Cookies minted by graphless deployments have no graph row | Rowless proof via strong-class records under the graphless-migration flag (N+2 convergence + attested stub-backfill first — providers spec §5); verified cookies expire and re-mint without continuity; **rowless withdrawal writes into the capped per-prefix `w` record, then expires** (exact-cookie family records are superseded); unverifiable roaming cookies get disclosed cookie-only expiry (sign-off 29) | **Declared** — pre-existing identities restart rather than carry over | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -220,7 +220,22 @@ Requirements: would make the required fixtures self-contradictory. Whether ungated adapters go stateless or block the release is product sign-off item 12. -4. **Graph-store readiness precedes everything.** Today the graph store +4. **The graphless migration has an operational runbook, not a + pointer to the wrong section.** (The providers spec's earlier "§4.2 + readiness step" reference pointed at adapter qualification.) The + sequence, attested where noted: (a) confirm N+2 fleet convergence + (deploy records); (b) measure/confirm the backend's listing settle + window (capability declaration); (c) run the stub-backfill scan to + **two consecutive zero-discovery passes**, recording watermark and + pass count; (d) CAS-create the graphless flag with the attestation in + its value; (e) rowless classification active. **Abort/rollback:** any + N+1 startup suspends the flag automatically (providers spec §5); a + failed or interrupted pass restarts from (c) — passes are idempotent; + re-attestation after any suspension repeats (c)–(d) over the gap + window. **Clearing:** after the quiet-period criterion (no rowless + classifications for a full cookie lifetime), the operator CASes the + flag cleared, permanently ending rowless classification. + 4b. **Graph-store readiness precedes everything.** Today the graph store is optional and EC generation succeeds without one; the epic's no-active-until-commit invariant (providers spec §5) makes it mandatory wherever a minting provider is configured — so a currently @@ -425,9 +440,13 @@ global honoring of opt-out signals is unconditional. irreversible artifacts are enumerated — not "one": **family revocation records and member tombstones** (no recovery; that is their purpose), the **schema-floor marker** (write-once by design; - no administrative clear), and **sticky timestamp-less suppression** - (administrative clear procedure documented in the guide, requiring - recorded operator intent). Withdrawal tombstones — which is why the + no administrative clear), and — corrected from the former "sticky + timestamp-less suppression" entry, which the permission spec's + TTL-sticky rule supersedes — nothing suppression-shaped: + timestamp-less opt-out suppression is **TTL-bounded and goes inert + automatically** (administrative clear is an optional early exit, not + a requirement, and the guide's cleanup and expiry tests follow the + TTL rule). Withdrawal tombstones — which is why the withdrawal triggers (permission spec §4.2) are exhaustive, why partial withdrawal failure has an explicit tombstones-first, browser-retries contract (permission spec §4.3), and why §2 rows 6 and 8 call out @@ -477,6 +496,7 @@ implemented. | 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | — | open | | 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | — | open | | 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | — | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | — | open | | 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | — | open | | 16 | Timestamp-less opt-out suppression is **TTL-sticky**: within its consent-TTL lifetime only a newer timestamped grant clears it; at `valid_until` it goes inert automatically (not user-sticky-forever, not administratively sticky — administrative clear is an optional early exit) | permission §4.3 | — | open | | 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | — | open | @@ -493,4 +513,4 @@ implemented. | 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance | hook §4a; `datadome-header-allowlist.md` | — | open | | 29 | Unverifiable roaming rowless cookies receive best-effort cookie-only expiry (admission rules forbid durable records for unverified values); a lost response can leave the cookie usable on the old network until re-presented | providers §5 | — | open | | 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, and observability all in scope | providers §5 | — | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | — | open | +| 31 | Replay-history capacity (16 semantic-state slots + rolling restrictive slot): while saturated, novel values cannot grant — fresh consent in unusual multi-CMP setups can be rejected until slots expire | permission §4.3; providers wire schema | — | open | diff --git a/docs/superpowers/specs/datadome-header-allowlist.md b/docs/superpowers/specs/datadome-header-allowlist.md index fecd3880a..31936a5e7 100644 --- a/docs/superpowers/specs/datadome-header-allowlist.md +++ b/docs/superpowers/specs/datadome-header-allowlist.md @@ -20,3 +20,7 @@ set mean: nothing else is accepted until a reviewed commit adds it. | `Location` | Respond (3xx) only | replace | | `Content-Type` | Respond only (owns its body) | replace | | `Cache-Control`, `Pragma` | Respond only | restricted merge; invariant pass still last | + +Note: the vendor's `X-Set-Cookie` response field is **not** a +forwardable header — it lowers into the typed `datadome` cookie +operation (hook spec §4a) and never reaches the browser as a header. diff --git a/docs/superpowers/specs/gpp-registry-snapshot.md b/docs/superpowers/specs/gpp-registry-snapshot.md index 103e8ca78..8565e6bd2 100644 --- a/docs/superpowers/specs/gpp-registry-snapshot.md +++ b/docs/superpowers/specs/gpp-registry-snapshot.md @@ -30,6 +30,9 @@ here is treated as malformed-present (permission spec §4.4). | 26 | usky | 1 | | 27 | usri | 1 | -Version values were captured from the IAB registry at the time of -writing and are re-verified against the official registry as part of -ratification review; any correction is a change to this file. +Version values for sections 6–23 were captured from the IAB registry at +the time of writing and are re-verified against the official registry as +part of ratification review. Sections 24–27 have assigned IDs but no +reproducibly published binary layouts in the official sources as of this +snapshot; they are reserved and inert until an official layout can be +vendored here. Any change is a reviewed change to this file. diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index 07e195664..c4a9b8481 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -286,3 +286,21 @@ All rows text-added per the R13 vocabulary. | P2 batch (absence one-shot with summary retirement + horizon cap; global config-push activation version; unknown GPP sections contribute nothing with the embedded-GPC coverage bound stated; complete byte-level constructors incl. `w`/`m`, 128-byte cap, family-id encoding; RFC 6265 domain-match + Expires normalization; shared fail-closed cache parser + `Vary: *`; NAT saturation sign-off 30; PSL fill + decision records + adapter qualification as explicit ratification gates) | text-added | | P3 batch (§5 activation lead fixed to the commit point; §4 device wording; typo/punctuation/metric residue; client resv notation; old DataDome spec carries a supersession banner now) | text-added | | Sign-off table restructured **decision-centric** (Owner column → Decision-record link; deciders live in the records) per maintainer direction | text-added | + +## Round 15 — review at f3eacf59 + +All rows text-added per the R13 vocabulary. + +| Finding | Status | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | +| P1 N+1 rollback invalidates rowless proof | text-added: any N+1 startup CASes the flag to suspended; re-activation needs full re-attestation; N+2→N+1-mint→N+2 named test | +| P1 admission rule blocks first-upgrade withdrawal / promotion | text-added: observed-row sequence adopted verbatim (row read → stub create-if-absent, no positive authority → family revocation → use denied between → cookie expiry); permission spec §4.3 references it | +| P1 `w` protocol absent from matrices/runtime/tests | text-added: abstract row (global reads + CAS + listing bound as migration gates), four runtime-failure rows, migration-window consult rule, saturation promotes listed hashes only (blanket row revocation excluded; overflow residual declared) | +| P1 replay saturation unimplementable | text-added: semantic-state slots with in-place TCF renewal updates (ordinary renewals never consume capacity), rolling restrictive slot outside the 16, serialized `saturated`/`saturated_until`, automatic recovery, metric; sign-off 31 | +| P1 TTL-sticky contradiction in migration | text-added: irreversible-artifact list corrected — suppression is TTL-bounded, administrative clear optional | +| P1 GPP 24–27 not reproducibly official | text-added: demoted to reserved-pending-official-schema in map and snapshot; those states behave national-only; the official-coverage claim retracted | +| P1 X-Set-Cookie unmapped | text-added: both cookie-return forms lower into the typed operation; the field itself never forwarded; allowlist file noted | +| P1 response allowlist vs representation rule | text-added: Respond may describe its body with `Content-Type` only; encoding/validators stay reserved; vendor needs arrive as allowlist-file review | +| P1 cache merge weakens origin policy | text-added: unknown snapshot directives preserved verbatim (mutations still drop unknowns); `Expires` in the freshness bound; no introduced `max-age`/`s-maxage` absent a snapshot upper bound (RFC 9111 §5.2.3/§5.3 cited) | +| P2 batch (HEAD serves the persisted GET artifact; fleet-stable revision tuple — registry content hash, global push version, build constant; tie winner's complete tuple survives (no timestamp ratchet); grammar consistency — suffix ≤123, `m` + registry index instead of padded names, full tag enumeration incl. `w`/`m`, registry-file note fixed, family-ID derivation with domain tag `tsfam1 | ` and vectors; graphless runbook with correct pointer, abort/re-attestation/clearing; policy digest defined (`tspol1 | `, canonical JSON, vectors); pointer parser total contract with fail-open semantics and cookie-source priority; `Pragma` dropped from the response allowlist) | text-added | +| P3 batch (wire-schema sentence restructured with the saturation rewrite; row 3e phrase completed; sign-off rows renumbered into order) | text-added | diff --git a/docs/superpowers/specs/provider-code-registry.md b/docs/superpowers/specs/provider-code-registry.md index fb373f32e..92748b476 100644 --- a/docs/superpowers/specs/provider-code-registry.md +++ b/docs/superpowers/specs/provider-code-registry.md @@ -5,6 +5,6 @@ keys (providers spec §6.3). Allocation is a reviewed commit to this file; codes are immutable and never recycled, including for retired providers. -| Code | Provider | Allocated | Status | -| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------ | -| `hmac` | Built-in HMAC EC provider (note: hmac identities use the reserved verbatim key grammar, so this code appears in non-key contexts — provenance, registries — not in physical keys) | 2026-08-02 | active | +| Code | Provider | Allocated | Status | +| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------ | +| `hmac` | Built-in HMAC EC provider (hmac identity rows use the reserved verbatim key grammar; this code does appear in `w` rowless-withdrawal keys and in non-key contexts — provenance, registries) | 2026-08-02 | active | From 21885221733bac3c6a22fb2afdc54dd2a1db2456 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Mon, 3 Aug 2026 17:51:33 -0700 Subject: [PATCH 17/24] Address sixteenth review: dual admission sequences, fleet-linearizable suspension, and representable replay state P1 fixes: - Suppression admission has one coherent contract with two observed-row sequences: destructive (stub -> family revocation -> expiry) and non-destructive (stub -> per-permission suppression CAS -> deny-while-incomplete, family and cookie retained) - the single revocation-shaped sequence either destroyed identities for SharingOptOut/TargetedAdvertisingOptOut or dropped their suppression. 'Unconditional' now explicitly means independent of prior positive authority, never of family admission, closing the fabricated-suffix strong-record path the permission wording had reopened. - The graphless flag is a globally-strong state machine (absent -> active -> suspended -> re-attested -> cleared, epoch per transition; globally observable strong reads are a capability cell for this key): N+2 proves active through a bounded lease revalidated by strong read; N+1 CASes suspension, reads it back, and waits one full lease window before minting - no unstubbed cookie can exist while any active lease survives, closing the stale-instance misclassification race by construction. w consultation continues under suspended. - GPP 24-27 are removed from decoder work items and accepted versions: reserved and inert, national-only for MD/IN/KY/RI (sign-off 32); the 'Maryland opt-out must not vanish / extend the decoder' phrasing is withdrawn as incompatible with reserved status. - Replay slots are keyed by a timestamp-independent state_key (semantic result excluding LastUpdated) distinct from the evidence digest - renewals genuinely update one slot; replay defense derives from recency comparison, needing no per-value history - and the unrepresentable rolling slot is replaced by a fixed saturation_epoch (saturated_until = entry + consent TTL, never extended by overflow values; overflow evidence affects the live request, is not stored, and advances nothing on replay). - Mutation-introduced must-understand is rejected: under RFC 9111 5.2.2.3 a cache understanding the status may ignore an accompanying no-store, so 'adding' it can weaken a stored no-store - it is not an additive restriction; snapshot-present survives. - The Pragma contradiction is resolved with a drop-individually list (known-harmless standard vendor fields are dropped and logged, never batch-invalidating; unknown/active fields still invalidate to Continue), the allowlist file corrected and retitled for both directions, DataDome's documented response (Set-Cookie, Pragma, X-DataDome, Cache-Control) becomes a verbatim test fixture, and the batch-invalidation fail-open consequence is inside sign-off 28. - Identifier derivation is fully assigned with computed known-answer vectors embedded: record-kind byte = class-tag ASCII; family-ID vector for tsfam1|i|hmac|{64xa}.AbC123 = e90616c381f64965b8326f17108c3c481cee932b2d7f8af783c7bdc2e21591ef; w suffix hashes = tswsx1|-tagged SHA-256 truncated to 16 bytes (AbC123 -> 08cb55acf42929772862e82b0960c134); the permission spec's stale fam:v0 example is replaced. P2/P3: registration versions gain a bump contract (content-hash where declarative, review-checklist otherwise; invariant revision bumps with parser/merge changes); head-only artifacts never satisfy a later GET and validator mismatches invalidate rather than update (RFC 9111 4.3.5); w retention >= max(cookie lifetime, row/S2S horizon) with the overflow-resurrection residual added to sign-off 30; the Fastly listing cell requires a cited platform completeness bound, pagination, and failure semantics; the suffix limit is 123 everywhere with boundary fixtures; Expires participates via RFC 9111 4.2.1 with conservative invalid-date handling; the pointer list has a tokenization grammar (repeated fields joined by SP, SP/HTAB runs, empty tokens ignored, ASCII lowercase before duplicate detection); the ledger's R15 malformed rows and false Pragma claim are corrected; the wire-schema grammar break, stale readiness-step pointer, allowlist title, and migration tombstone fragment are repaired; sign-off row 32 added. --- ...integration-response-header-hook-design.md | 60 +++++-- .../2026-07-30-permission-model-design.md | 17 +- .../2026-07-30-pluggable-providers-design.md | 162 +++++++++++------- ...07-30-provider-migration-rollout-design.md | 5 +- .../specs/datadome-header-allowlist.md | 2 +- docs/superpowers/specs/pr986-review-ledger.md | 20 ++- 6 files changed, 175 insertions(+), 91 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 751922967..057e22cb6 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -68,8 +68,13 @@ mutators to the outbound response for HTML document responses it processed. storage subject to revalidation; `private` forbids shared storage), so "replace `private` with the stronger `no-cache`" would make a personalized response shared-storable. The merge: each of `no-store`, - `no-cache`, `private`, `must-revalidate`, `proxy-revalidate`, - `must-understand`, and `no-transform` is **sticky** — present in the snapshot or the + `no-cache`, `private`, `must-revalidate`, `proxy-revalidate`, and + `no-transform` is **sticky** — and `must-understand` is the deliberate + exception: **mutation-introduced `must-understand` is rejected** + (snapshot-present survives untouched), because under RFC 9111 + §5.2.2.3 a cache that understands the status may then ignore an + accompanying `no-store` — "adding" it can _weaken_ a stored `no-store` + response, so it is not an additive restriction at all — present in the snapshot or the mutation ⇒ present in the final response, independently; `public` is dropped whenever any restriction is present; `max-age`/`s-maxage` may only shrink relative to the snapshot; `stale-while-revalidate`/ @@ -94,9 +99,14 @@ mutators to the outbound response for HTML document responses it processed. unknown directives already in the snapshot are preserved verbatim** (a downstream cache may honor a restrictive extension TS does not recognize; dropping it would weaken origin policy, RFC 9111 §5.2.3); - `Expires` participates in the freshness bound (effective freshness = - the minimum across `max-age`, `s-maxage`, and the `Expires`-derived - lifetime) and a mutation may **not introduce `max-age`/`s-maxage` + `Expires` participates in the freshness bound via **RFC 9111 §4.2.1's + freshness-lifetime algorithm, referenced directly**: the + `Expires`-derived lifetime is `Expires − Date` (absent `Date` → + response receipt time), invalid or duplicate date values are treated + as already expired (the RFC's conservative option, chosen + normatively), and `Age` is handled per the RFC — effective freshness + is the minimum across `max-age`, `s-maxage`, and that derived + lifetime and a mutation may **not introduce `max-age`/`s-maxage` where the snapshot supplied no upper bound** — HTTP prefers `max-age` over `Expires` (RFC 9111 §5.3), so introducing one would override an origin's shorter or already-expired `Expires`; and `Vary: *` is @@ -140,7 +150,15 @@ mutators to the outbound response for HTML document responses it processed. Integration IDs are **startup-unique, enforced**: registry construction rejects a duplicate ID (current code silently coalesces, which corrupts attribution and budgets), with a duplicate-ID test in - the done-when. Until then the + the done-when. The registration also carries the **version the cache + tuple consumes, with a bump contract**: the version MUST change with + every output-semantic change of the mutator (review-checklist item; + where the mutator's behavior is fully declared configuration, the + version is a content hash of that declaration, making the bump + automatic), and the build-time invariant revision MUST bump with any + parser or merge-rule change — otherwise a deploy silently reuses old + post-hook finals and a new privacy restriction waits for cache + expiry. Until then the operation set is headers-only, and `Set-Cookie` is fully reserved. `Set-Cookie` is fully reserved in v1 (§3 deferral). Violations are rejected @@ -294,13 +312,16 @@ degree of freedom is closed: spec) — and only DataDome-returned overlay data reaches the publisher, never the raw browser-supplied header. - **The pointer protocol has a total parser contract** — adapters - cannot differ where malformed batches fail open: pointer names - compare case-insensitively with OWS/tab trimmed; duplicate pointer - names, invalid names, more than 16 pointers, or more than 4 KiB of - pointer payload render the batch invalid (→ Continue, the vendor's - fail-open); when both cookie sources arrive (header form and - `Set-Cookie`), the header form wins, matching the vendor's documented - priority. + cannot differ where malformed batches fail open: the pointer list is + tokenized by the vendor's documented space separation — repeated + pointer header fields are concatenated with a single SP before + tokenizing, tokens split on runs of SP/HTAB, empty tokens ignored — + then names are ASCII-lowercased before duplicate detection; duplicate + names after normalization, invalid names, more than 16 pointers, or + more than 4 KiB of pointer payload render the batch invalid + (→ Continue, the vendor's fail-open); when both cookie sources arrive + (header form and `Set-Cookie`), the header form wins, matching the + vendor's documented priority. - **Request-header pointers are a positive, enumerated allowlist.** "Documented enrichment headers" is not enforceable; the registration enumerates the exact names from the **checked-in allowlist file @@ -326,8 +347,17 @@ degree of freedom is closed: a _Respond_ (challenge/deny) may set exactly `Location` (replace; 3xx only), `Content-Type` (its own body, per the representation rule below), `Cache-Control` (through the restricted merge; the invariant pass - still runs last — `Pragma` is dropped from the allowlist: response - `Pragma: no-cache` has no standardized meaning, RFC 9111 §5.4), the typed security cookie (above), + still runs last). **`Pragma` and its kin get a defined middle path**: + allowlist-absent fields that are known-harmless standard cache + metadata (`Pragma` is the enumerated case — response `Pragma: +no-cache` has no standardized meaning, RFC 9111 §5.4) are **dropped + individually and logged**, never batch-invalidating; genuinely + unknown or active fields still invalidate the batch (→ Continue). + Without this split, DataDome's own documented response — which points + at `Set-Cookie`, `Pragma`, `X-DataDome`, and `Cache-Control` — would + fail every challenge open; that documented vendor response is a + **verbatim test fixture**, and the fail-open consequence of + batch-invalidation is explicitly within sign-off 28's scope, the typed security cookie (above), and the vendor response headers enumerated in the **response section of `datadome-header-allowlist.md`**; a _Continue_ may set only the typed cookie and those enumerated vendor headers. Everything else is diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 5aca80c88..e5936f105 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -401,7 +401,7 @@ and the fail-closed marker: a stable **family ID**: minted rows store it, and — the case that makes or breaks the protocol — **rows that lack the field derive it deterministically** as a function of (record kind, provider namespace, - canonical graph key), e.g. `fam:v0:hmac:`. Determinism is the + canonical graph key), per the providers spec §6.3 derivation (`tsfam1|` + record-kind byte + provider code + graph key). Determinism is the point: a withdrawal arriving on the **first post-upgrade request** — a GPC-carrying visitor whose v1 row has no family field and has never been backfilled — computes the same family ID that every future reader of @@ -433,7 +433,7 @@ and the fail-closed marker: **Creation is cause-aware and mostly read-free.** A live resolution whose outcome for a permission is unset writes suppression when the cause is a **signal state** — refusal, non-destructive opt-out, - malformed-present — **unconditionally**, with no row read: conditioning + malformed-present — **unconditionally** — meaning independent of _prior positive authority_, never independent of **family admission** (every durable write still passes the providers spec §5 admission arms; for an observed v1 row the non-destructive sequence applies) — with no row read needed for the decision itself: conditioning on observing positive provenance through an eventually consistent row loses the race where a stale replica hides a just-committed grant. The one cause that inherently needs prior state — applicable **absence** @@ -711,12 +711,15 @@ section malformed-present (blocks grants, never withdraws). claim official-registry coverage for them (an earlier revision claimed both "no section" and later "official through 27" — each wrong in its own direction). A truncated map silently loses - opt-outs — a Texas (16) or Maryland (24) sale opt-out must not - vanish. **The current decoder is an explicit prerequisite gap**: it + opt-outs — a Texas (16) sale opt-out must not vanish. **The current decoder is an explicit prerequisite gap**: it (and `iab_gpp` 0.1.2) supports sections 7–23 only and models `usnat` - v2 while the snapshot pins v1 — implementation must extend or replace - the decoder for 24–27 _and_ reject versions the library happens to - decode but the snapshot disallows. + v2 while the snapshot pins v1 — implementation must reject versions + the library happens to decode but the snapshot disallows. Sections + 24–27 are **not** a decoder work item: with no reproducible official + layout they are reserved and inert (national-only for those states — + an accepted limitation, sign-off 32); the earlier "Maryland opt-out + must not vanish / extend the decoder for 24–27" reading is withdrawn + as incompatible with reserved status. The implementation PR cross-checks this list against both the current decoder's section set and the official registry, and the accepted version per section is **pinned to the vendored registry snapshot diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index 41521835a..3ae232c8c 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -311,15 +311,26 @@ variant**. Therefore: passes discover zero unstubbed rows**; the flag value attests the watermark, pass count, and settle window. Rows minted after T carry records by protocol — **and rollback cannot silently break that - invariant**: an N+1 binary mints v1 rows with no records, so any N+1 - instance that starts while the flag is active **CASes it to - `suspended` at startup** (N+1 reads deployment metadata; this is one - of its reader duties). Suspended = rowless classification off - fleet-wide; re-activation after roll-forward requires a **complete - re-attestation** (a fresh scan to two idle passes covering the - rollback window's mints). The N+2 → rollback-to-N+1 → mint → - roll-forward-to-N+2 schedule is a named test proving those rows are - never classified rowless. And because no scan over an eventual store is + invariant — with fleet-linearizable mechanics, not a hopeful CAS**: + the flag is a **globally-strong state machine** + (absent → active → suspended → re-attested-active → cleared, each + transition bumping an epoch; this metadata key requires globally + observable strong reads _in addition to_ CAS — a capability cell, + since CAS alone says nothing about what other instances currently + see). An N+1 instance starting while the flag is active CASes it to + `suspended`, **reads the committed suspension back, then waits one + full lease window before minting**. N+2 instances prove `active` at + classification time through a **bounded lease** (strong read at lease + expiry, lease ≤ L): suspension is therefore fleet-effective within L, + and because N+1 does not mint until L has elapsed after its committed + suspension, **no unstubbed cookie can exist while any instance still + holds an `active` lease** — the stale-lease misclassification race is + closed by construction, not by luck. Under `suspended`, rowless + _classification_ stops but **`w` consultation and enforcement + continue** (withdrawn stays withdrawn). Re-activation after + roll-forward requires complete re-attestation over the gap window. + The N+2 → rollback-to-N+1 → mint → roll-forward-to-N+2 schedule is a + named test proving those rows are never classified rowless. And because no scan over an eventual store is provably perfect, misses are **reconciled, not fatal**: a per-prefix withdrawal entry (below) doubles as pending intent — if a real row for a withdrawn suffix ever surfaces, core **promotes** the entry to a @@ -336,7 +347,7 @@ variant**. Therefore: through the row path during the window. Graphless-era cookies never got a stub because they have no row for the scan to find; no eventual read participates. The flag itself is specified: a named deployment-metadata key (write-once/CAS class), set by the §4.2 - readiness step only on deployments that actually ran graphless + migration runbook step (migration spec §4) only on deployments that actually ran graphless (requires the deployment-metadata capability), surviving binary rollback, and **cleared by an explicit operator action** once the migration window closes (quiet-period criterion in the guide) — @@ -379,12 +390,24 @@ variant**. Therefore: row read is safe admission evidence (an eventual not-found is not), and without this arm the first post-upgrade GPC request could not revoke an untouched v1 row, and the promotion path could not promote a - late-surfacing row (neither has a stub by definition). The - permission-exempt sequence for an observed row: derive the family ID → - **create-if-absent a minimal strong stub carrying no positive - authority** → commit the family revocation → the identity is denied - all use between discovery and revocation commit → only then expire the - browser cookie. A rowless identifier — even a _verified_ one — creates + late-surfacing row (neither has a stub by definition). There are **two** + permission-exempt observed-row sequences, because one shape cannot + serve both signal classes (the single revocation-shaped sequence + either destroyed identities for non-destructive opt-outs or dropped + their suppression entirely). **Destructive** (GPC, sale, USP): derive + the family ID → create-if-absent a minimal strong stub carrying no + positive authority → commit the family revocation → the identity is + denied all use between discovery and revocation commit → only then + expire the browser cookie. **Non-destructive** (SharingOptOut / + TargetedAdvertisingOptOut, refusal, malformed): row read → same stub + create-if-absent → **CAS the per-permission suppression entry** → + the suppressed permission is denied use while the sequence is + incomplete → **the family and the cookie are retained** (nothing is + revoked or expired). The permission spec's "unconditional" creation + means **independent of prior positive authority — never independent + of family admission**: every durable negative write passes one of + these admission arms, which is what keeps fabricated HMAC suffixes + from minting unbounded strong records. A rowless identifier — even a _verified_ one — creates none: verification authenticates only the prefix, so per-identifier records would let one prefix-holder fabricate unlimited suffixes into unlimited strong-storage records (rate limits slow creation; they do @@ -608,17 +631,17 @@ solves. **Physical key grammar.** Core constructs every key; providers supply only the bounded suffix: -| Record class | Key | Notes | -| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ 128 bytes, KV-safe alphabet. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | -| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | -| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | -| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | -| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | -| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8), saturation flag, CAS version, created-at, `valid_until` ≥ the maximum cookie lifetime; readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | -| Deployment metadata | `m` + fixed metadata name (fixed-width grammar; schema floor, graphless-migration flag) | Write-once/CAS class; value carries schema version, state, epoch, set-at; the graphless flag's lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded | -| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | -| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | +| Record class | Key | Notes | +| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ **123** bytes (128-byte total key cap minus tag and provider code — an earlier 128 here contradicted the constructor; 123/124 boundary fixtures required), KV-safe alphabet. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | +| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | +| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | +| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | +| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | +| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8), saturation flag, CAS version, created-at, `valid_until` ≥ **max(cookie lifetime, row/S2S authority horizon)** — a `w` expiring before the row horizon would let an overflow row's identity resurface (the overflow non-promotion residual is sign-off 30); readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | +| Deployment metadata | `m` + fixed metadata name (fixed-width grammar; schema floor, graphless-migration flag) | Write-once/CAS class; value carries schema version, state, epoch, set-at; the graphless flag's lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded | +| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | +| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | Grammars are pairwise non-intersecting by their literal prefixes (plus the reserved hmac grammar), and every record value carries a `kind` @@ -646,11 +669,18 @@ providers), then the suffix. The **complete physical constructor set** (logical provider-code(4) + suffix(≤123 — the 128-byte total minus tag and code; an earlier "suffix ≤128" was inconsistent with its own cap) for rows; `r`/`s` + family-id(64 lowercase hex — family IDs are canonically -SHA-256 over a **defined derivation input**: the domain tag `tsfam1|`, -then the record-kind byte, then the provider code (4 bytes), then the -canonical graph-key bytes — concatenated in that order, UTF-8, no -separators beyond the tag's `|`; cross-language test vectors are a -conformance requirement) for revocation/authority records (no provider code: the family id +SHA-256 over a **fully assigned derivation input**: the domain tag +`tsfam1|`, then the record-kind byte — **assigned: the class tag's +ASCII byte** (`i` = 0x69 for identity-derived families) — then the +provider code (4 bytes), then the canonical graph-key bytes, +concatenated in that order, no separators beyond the tag's `|`. +**Known-answer vector**: input `tsfam1|` + `i` + `hmac` + +`{64×"a"}.AbC123` → +`e90616c381f64965b8326f17108c3c481cee932b2d7f8af783c7bdc2e21591ef`. +`w` suffix hashes are likewise assigned: SHA-256 with domain tag +`tswsx1|` over the raw suffix bytes, truncated to 16 bytes, lowercase +hex (32 chars) — vector: `AbC123` → `08cb55acf42929772862e82b0960c134`. +Cross-language vector suites extend these) for revocation/authority records (no provider code: the family id already encodes derivation); `w` + provider-code(4) + prefix(64 hex) for rowless withdrawal; `x` + family-id(64) for transactions; `m` + a **2-digit registry-assigned index** for deployment metadata (a closed name @@ -679,25 +709,29 @@ spec's absence/replay decisions consume — a reduced schema cannot reproduce them**): kind (user evidence vs policy baseline), grant basis/source class, policy revision (digest + activation generation), `valid_until`, provenance revision, evidence timestamp, and the -and a **bounded replay history**, record-level, keyed by (source -class, semantic digest) — because a _single current_ digest cannot -uphold "re-presentation never advances first-seen" (grant A → refusal -B → replayed A would look novel once B displaced A's slot). Entries pin -first-seen/first-normalized timestamps, retained for at least the -maximum evidence/suppression horizon. Capacity and saturation are a -**serialized state machine, not a shrug**: 16 slots per -permission·source, each holding a _semantic state_ — a TCF renewal with -the same semantic result and newer `LastUpdated` **updates its slot in -place** (ordinary repeated renewals never consume capacity; replays -cannot advance the slot, their `LastUpdated` is not newer) — plus one -dedicated **rolling restrictive slot** outside the 16, so a seventeenth -refusal is stored under normal recency rules, never dropped and never -replay-advanceable. All slots holding distinct in-horizon states sets -`saturated: true` with `saturated_until` = the earliest slot -`valid_until`: while saturated, novel values cannot grant (fail -restrictive), restrictive evidence still lands in the rolling slot, -recovery is automatic as slots expire, and saturation is a first-class -metric — the cap and its denial behavior are **sign-off item 31**; record level: family ID, +and a **bounded replay history** whose slots are keyed by a +**timestamp-independent `state_key`** — (source class, semantic result +digest _excluding_ `LastUpdated`) — distinct from the _evidence digest_ +(which for TCF includes `LastUpdated` for recency): keying slots on the +evidence digest would give every renewal a fresh key and make +"updates its slot in place" impossible, the incompatibility an earlier +draft shipped. A slot stores its `state_key`, the current evidence +digest, that digest's pinned first-seen, and the newest authoritative +timestamp observed; a TCF renewal (same `state_key`, newer +`LastUpdated`) updates the slot in place, while a replay (not newer) +changes nothing — replay protection derives from recency comparison, +not per-value history, so no per-digest sublists are needed. 16 slots +per permission·source; entries live to the evidence/suppression +horizon. **Saturation is a fixed epoch, not a rolling slot** (one slot +cannot hold independent timestamps for multiple overflow digests): when +all slots hold distinct in-horizon states, the record sets a +`saturation_epoch` with `saturated_until = now + consent TTL`, **fixed +at entry and never extended by later overflow values**; while +saturated, novel values cannot grant (fail restrictive — overflow +evidence of either polarity affects the live request but is not +stored, so replaying it advances nothing), recovery is automatic at +epoch expiry as slots free, and saturation is a first-class metric — +the cap and its denial behavior are **sign-off item 31**; record level: family ID, CAS version counter, schema version, stub marker (backfill, §5); unknown-field and range validation apply like every class (strong class, permission-exempt writes per the permission spec's inventory; revisions are app-level counters because @@ -792,19 +826,19 @@ Requirements: them is how a "yes" cell hides an unusable feature. Feature eligibility requires wired, not merely available: - | Capability | Fastly | Axum (dev) | Cloudflare | Spin | - | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | - | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | - | Prefix listing (cluster) | Yes (used today) | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | - | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | - | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | - | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | - | Device host evidence (JA4/H2) | Yes | No | No | No | - | Authority-state: global strong reads + CAS | Conditional writes available (generation marker); **globally current read semantics to verify** — both required, wiring to verify | **Unavailable** (no store wired) | Workers KV: **ineligible**; Durable Objects: feasible, not wired | **Unavailable** | - | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | - | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | - | Deployment metadata (write-once/CAS) | **Not wired** — needs a primitive distinct from the config store | **Unavailable** | DO: feasible, not wired | **Unavailable** | - | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | + | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | + | Prefix listing (cluster) | Used today, but **ratification requires a cited platform completeness bound, pagination behavior, and failure semantics** — the graphless scan's settle window cannot be derived from "works in practice" | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | + | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | + | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | + | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | + | Device host evidence (JA4/H2) | Yes | No | No | No | + | Authority-state: global strong reads + CAS | Conditional writes available (generation marker); **globally current read semantics to verify** — both required, wiring to verify | **Unavailable** (no store wired) | Workers KV: **ineligible**; Durable Objects: feasible, not wired | **Unavailable** | + | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Deployment metadata (write-once/CAS) | **Not wired** — needs a primitive distinct from the config store | **Unavailable** | DO: feasible, not wired | **Unavailable** | + | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 11c95bd50..2b8bd1147 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -446,7 +446,7 @@ global honoring of opt-out signals is unconditional. timestamp-less opt-out suppression is **TTL-bounded and goes inert automatically** (administrative clear is an optional early exit, not a requirement, and the guide's cleanup and expiry tests follow the - TTL rule). Withdrawal tombstones — which is why the + TTL rule). The irreversibility of revocation is also why the withdrawal triggers (permission spec §4.2) are exhaustive, why partial withdrawal failure has an explicit tombstones-first, browser-retries contract (permission spec §4.3), and why §2 rows 6 and 8 call out @@ -512,5 +512,6 @@ implemented. | 27 | Proxy-mode minimal opt-out extraction (decode only §4.5-mapped opt-out fields; no grants) | permission §4.4 | — | open | | 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance | hook §4a; `datadome-header-allowlist.md` | — | open | | 29 | Unverifiable roaming rowless cookies receive best-effort cookie-only expiry (admission rules forbid durable records for unverified values); a lost response can leave the cookie usable on the old network until re-presented | providers §5 | — | open | -| 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, and observability all in scope | providers §5 | — | open | +| 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, observability, **and the overflow residual: a real row beyond the listed hashes permanently loses promotion and could resurface if `w` retention or the flag lifecycle were mishandled** — all in scope | providers §5 | — | open | | 31 | Replay-history capacity (16 semantic-state slots + rolling restrictive slot): while saturated, novel values cannot grant — fresh consent in unusual multi-CMP setups can be rejected until slots expire | permission §4.3; providers wire schema | — | open | +| 32 | GPP sections 24–27 (MD/IN/KY/RI) are reserved with **national-section-only** handling until an official binary layout can be vendored — a state-specific opt-out expressed only in an undecodable state section is not honored | permission §4.5; `gpp-registry-snapshot.md` | — | open | diff --git a/docs/superpowers/specs/datadome-header-allowlist.md b/docs/superpowers/specs/datadome-header-allowlist.md index 31936a5e7..1f8c24395 100644 --- a/docs/superpowers/specs/datadome-header-allowlist.md +++ b/docs/superpowers/specs/datadome-header-allowlist.md @@ -1,4 +1,4 @@ -# DataDome request-header allowlist (normative, checked-in) +# DataDome header allowlist (normative, checked-in — request and response directions) The complete set of response-named header pointers the security channel (hook spec §4a) may copy into the owner-scoped publisher-upstream diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index c4a9b8481..bff46332a 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -292,7 +292,7 @@ All rows text-added per the R13 vocabulary. All rows text-added per the R13 vocabulary. | Finding | Status | -| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | P1 N+1 rollback invalidates rowless proof | text-added: any N+1 startup CASes the flag to suspended; re-activation needs full re-attestation; N+2→N+1-mint→N+2 named test | | P1 admission rule blocks first-upgrade withdrawal / promotion | text-added: observed-row sequence adopted verbatim (row read → stub create-if-absent, no positive authority → family revocation → use denied between → cookie expiry); permission spec §4.3 references it | | P1 `w` protocol absent from matrices/runtime/tests | text-added: abstract row (global reads + CAS + listing bound as migration gates), four runtime-failure rows, migration-window consult rule, saturation promotes listed hashes only (blanket row revocation excluded; overflow residual declared) | @@ -302,5 +302,21 @@ All rows text-added per the R13 vocabulary. | P1 X-Set-Cookie unmapped | text-added: both cookie-return forms lower into the typed operation; the field itself never forwarded; allowlist file noted | | P1 response allowlist vs representation rule | text-added: Respond may describe its body with `Content-Type` only; encoding/validators stay reserved; vendor needs arrive as allowlist-file review | | P1 cache merge weakens origin policy | text-added: unknown snapshot directives preserved verbatim (mutations still drop unknowns); `Expires` in the freshness bound; no introduced `max-age`/`s-maxage` absent a snapshot upper bound (RFC 9111 §5.2.3/§5.3 cited) | -| P2 batch (HEAD serves the persisted GET artifact; fleet-stable revision tuple — registry content hash, global push version, build constant; tie winner's complete tuple survives (no timestamp ratchet); grammar consistency — suffix ≤123, `m` + registry index instead of padded names, full tag enumeration incl. `w`/`m`, registry-file note fixed, family-ID derivation with domain tag `tsfam1 | ` and vectors; graphless runbook with correct pointer, abort/re-attestation/clearing; policy digest defined (`tspol1 | `, canonical JSON, vectors); pointer parser total contract with fail-open semantics and cookie-source priority; `Pragma` dropped from the response allowlist) | text-added | +| P2 batch (HEAD serves the persisted GET artifact; fleet-stable revision tuple — registry content hash, global push version, build constant; tie winner's complete tuple survives (no timestamp ratchet); grammar consistency — suffix ≤123, `m` + registry index instead of padded names, full tag enumeration incl. `w`/`m`, registry-file note fixed, family-ID derivation with domain tag `tsfam1 | ` and vectors; graphless runbook with correct pointer, abort/re-attestation/clearing; policy digest defined (`tspol1 | `, canonical JSON, vectors); pointer parser total contract with fail-open semantics and cookie-source priority; `Pragma` moved to the drop-individually list — **the R15 ledger falsely said the allowlist file was updated; it was not until R16**) | text-added | | P3 batch (wire-schema sentence restructured with the saturation rewrite; row 3e phrase completed; sign-off rows renumbered into order) | text-added | + +## Round 16 — review at 9a8596ee + +All rows text-added per the R13 vocabulary. + +| Finding | Status | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| P1 dual suppression admission contracts | text-added: two observed-row sequences (destructive: stub → revocation → expiry; non-destructive: stub → suppression CAS → deny-while-incomplete, family and cookie retained); "unconditional" clarified as independent-of-prior-authority, never of family admission | +| P1 flag suspension not fleet-linearizable | text-added: globally-strong state machine with epochs; N+2 validates active on a bounded lease; N+1 reads back suspension then waits one lease window before minting — no unstubbed cookie can coexist with a live active lease; `w` enforcement continues under suspended | +| P1 GPP 24–27 accept/inert/decode contradiction | text-added: removed from decoder work items and accepted versions; national-only limitation is sign-off 32; "Maryland must not vanish" withdrawn | +| P1 replay keys incompatible / rolling slot unrepresentable | text-added: timestamp-independent `state_key` (semantic result sans LastUpdated) keys slots; recency, not per-value history, is the replay defense; rolling slot replaced by a fixed `saturation_epoch` never extended by overflow values | +| P1 must-understand weakens no-store | text-added: mutation-introduced `must-understand` rejected (RFC 9111 §5.2.2.3); snapshot-present preserved | +| P1 Pragma fork / fail-open | text-added: drop-individually list for known-harmless standard fields (Pragma enumerated) vs batch-invalidating unknowns; the documented vendor response is a verbatim fixture; allowlist file corrected and retitled; consequence in sign-off 28 | +| P1 nondeterministic identifiers | text-added: record-kind byte assigned (class-tag ASCII), `tswsx1 | `16-byte truncated suffix hashes, **computed known-answer vectors embedded** for family ID and suffix hash; stale`fam:v0` example replaced | +| P2 batch (registration version bump contract + invariant-revision bump; head-only artifacts never satisfy GET, validator-mismatch invalidates; `w` retention ≥ max(cookie, row/S2S horizon) + overflow residual added to sign-off 30; Fastly listing cell requires a cited platform bound; suffix 123 everywhere + boundary fixtures; Expires via RFC 9111 §4.2.1 with conservative invalid-date handling; pointer-list tokenization grammar) | text-added | +| P3 batch (ledger R15 pipes/falsity corrected; wire-schema grammar; providers stale §4.2 pointer; allowlist title; migration tombstone fragment repaired) | text-added | From 184c9d9b63e99073229d31e460ff2a2eba5dda43 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 4 Aug 2026 09:55:39 -0700 Subject: [PATCH 18/24] Address seventeenth review: one N+1 contract, crash-safe suspension, revision-matched S2S, and fully closed DataDome/GPP boundaries P1 fixes (second-order interactions the prior repairs left open): - N+1 has one contract: it may create s-class stubs and write negative suppression entries, but never commit or clear positive authority until N+2. A first post-upgrade GPC or SharingOptOut now executes fully on N+1 - the earlier 'neither creates nor clears' wording made first-upgrade withdrawal unexecutable. - Graphless suspension is crash- and concurrency-safe: the suspended epoch stores a fleet-stable not_before = suspend + L (L = 120 s, assigned and stored) that every N+1 observer honors, not just the suspending instance; the state chain is cyclic (re-attested-active -> suspended for a second rollback) with CAS-loser retry; and the graphless flag requires globally observable strong reads, now a capability cell rather than write-once/CAS alone. - One w saturation outcome: any real row surfacing under a saturated prefix is denied and its family revoked (listed or overflow), closing the resurrection where a completed rowless withdrawal became usable the instant its row appeared; w consultation keys on the record's valid_until, not the rowless flag, so a late row after the flag clears still finds a live w. - Restrictive replay overflow is recorded under an epoch-pinned marker (timestamp and valid_until fixed at saturation entry), so replaying the same overflow value can no longer advance observation time and renew denial indefinitely; sign-off 31 text updated. - Two-record use requires row.provenance_revision == authority.summary_revision, failing closed both ways, and the strong summary now carries resolved jurisdiction - so a newer committed summary can never be combined with a stale row's jurisdiction to authorize egress under the wrong rule. - The cache invariant includes request-side authority (RFC 9111 3.5): on a response to an Authorization-bearing request the origin did not mark shared-cacheable, integrations may not introduce public, must-revalidate, or s-maxage; the invariant forces private, no-store. - The 304 contract splits the two cases: a local conditional hit re-emits persisted finals, while an origin-revalidation 304 updates the stored base from the 304 first (RFC 9111 4.3.4) and reruns processing or refetches a full 200 if cache-relevant fields changed - a cached public response can no longer survive an origin private, no-store revalidation. - DataDome sessionByHeader is unsupported by default (TS never requests it); enabling it forwards typed X-Set-Cookie/X-DD-B and accepts a JS/local-storage observer as an explicit sign-off-23 opt-in. - Every documented DataDome pointer has exactly one assigned outcome (outcome table in the allowlist file: cookie / forward / merge / drop / invalidate), the documented vendor response is a Respond-asserting fixture, and every security Respond ends Cache-Control: private, no-store with CDN fields stripped regardless of status or pointers. - GPP sections 24-27 are fully reserved: removed from the accepted- version table into a separate reserved table (no accepted version), reserved-vs-unknown difference defined (logging only), and supported sections pin to an immutable registry commit with vendored encoded vectors. P2/P3: HEAD-only artifacts are a distinct type that never satisfies a GET, with validator/Content-Length update rules; Vary ordering reaches TS's own cache key (mutation -> Vary -> key -> commit); state_key and evidence digests have canonical construction (tsstk1|/tsevd1|) with a vector; a non-HMAC family vector disambiguates the graph-key bytes; the suffix alphabet is the portable [A-Za-z0-9._~-]; revision hashes are domain-separated and adapter-independent (tspol1|/tsreg1|/tscfg1|); the must-understand clause is scoped to the six sticky directives; Respond transport is size/deadline/encoding bounded; sign-off rows 9/10/31 read open per the decisions README; and the 'and the and a' fragment, the providers/hook stale section-4.2 pointers, and the malformed R15/R16 ledger tables are corrected (the ledger's false 'Pragma removed from allowlist' claim recorded, tables rebuilt as prose). --- ...integration-response-header-hook-design.md | 99 +++++++---- .../2026-07-30-permission-model-design.md | 50 +++--- .../2026-07-30-pluggable-providers-design.md | 159 ++++++++++-------- ...07-30-provider-migration-rollout-design.md | 94 ++++++----- .../specs/datadome-header-allowlist.md | 16 ++ .../specs/gpp-registry-snapshot.md | 28 ++- docs/superpowers/specs/pr986-review-ledger.md | 107 ++++++++---- 7 files changed, 353 insertions(+), 200 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 057e22cb6..0af57dfc2 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -67,16 +67,24 @@ mutators to the outbound response for HTML document responses it processed. are orthogonal constraints (RFC 9111: `no-cache` permits shared storage subject to revalidation; `private` forbids shared storage), so "replace `private` with the stronger `no-cache`" would make a - personalized response shared-storable. The merge: each of `no-store`, - `no-cache`, `private`, `must-revalidate`, `proxy-revalidate`, and - `no-transform` is **sticky** — and `must-understand` is the deliberate + personalized response shared-storable. The merge: each of the **six sticky directives** — `no-store`, `no-cache`, + `private`, `must-revalidate`, `proxy-revalidate`, `no-transform` — is + present-in-snapshot-or-mutation ⇒ present-in-final (this + "snapshot or mutation ⇒ final" rule scopes to exactly these six) — and `must-understand` is the deliberate exception: **mutation-introduced `must-understand` is rejected** (snapshot-present survives untouched), because under RFC 9111 §5.2.2.3 a cache that understands the status may then ignore an accompanying `no-store` — "adding" it can _weaken_ a stored `no-store` response, so it is not an additive restriction at all — present in the snapshot or the mutation ⇒ present in the final response, independently; `public` is - dropped whenever any restriction is present; `max-age`/`s-maxage` may + dropped whenever any restriction is present; **request-side authority + is part of the invariant** — if the request carried `Authorization` + and the origin did not itself authorize shared reuse (no `public`, + `must-revalidate`, or `s-maxage` from the origin), an integration may + not introduce `public`, `must-revalidate`, or `s-maxage` (RFC 9111 + §3.5 makes those the very directives that unlock shared caching of + authenticated responses); the invariant pass forces `private, +no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may only shrink relative to the snapshot; `stale-while-revalidate`/ `stale-if-error` may appear only if the snapshot had them **and their durations may only shrink** (present-at-1s must not become @@ -90,7 +98,12 @@ mutators to the outbound response for HTML document responses it processed. they are additionally stripped from any restricted response; and the final `Vary` is the **union of the complete snapshot `Vary` set** — origin-supplied members included, not only core-required ones — and - the mutation. Parsing itself is a **shared core parser with + the mutation. Ordering is normative so the final `Vary` reaches TS's + own cache key, not just the wire: **mutation/invariant → final `Vary` + computation → cache-key construction → body/metadata commit**, and the + nominated request values are stored with the artifact (emitting `Vary` + downstream is useless if the body was already keyed less specifically + internally). Parsing itself is a **shared core parser with fail-closed normalization**, not four adapter interpretations: invalid `Cache-Control` syntax normalizes to the most restrictive reading; duplicate directives keep the strongest; quoted and unquoted @@ -236,17 +249,17 @@ mutators to the outbound response for HTML document responses it processed. Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by a fleet-stable tuple** — the integration-registry revision is a content hash over the ordered (integration ID, version) list; the config revision is the config store's globally assigned push version; the invariant revision is a build-time spec-version constant — local counters would collide across instances and binaries; a 304 — like a **normal cache hit** (§3a), which serves the same persisted finals rather than re-running mutators — re-emits them only when all three match the serving instance; a mismatch is a **cache miss**. Identity of normal-hit and conditional-hit metadata holds by construction (one persisted artifact serves both), not by an unsupported determinism claim about mutators. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | -| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals persisted | -| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | +| Response | Hook runs? | +| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by a fleet-stable tuple** — the integration-registry revision is a content hash over the ordered (integration ID, version) list; the config revision is the config store's globally assigned push version; the invariant revision is a build-time spec-version constant — local counters would collide across instances and binaries; there are **two distinct 304 cases**. A **locally generated conditional hit** (TS answers the client's `If-*` from its own fresh stored artifact) re-emits the persisted finals when all three revisions match, else cache-miss — as before. An **origin-revalidation 304** (TS revalidated upstream and the origin returned 304 with possibly new `Cache-Control`/`Vary`/`Expires`/validators) is different: RFC 9111 §4.3.4 requires the stored response to be **updated from the current 304 before serving**, so TS updates the stored base with the 304's metadata first; if any admitted/cache-relevant field changed, it **reruns the relevant processing or refetches a full 200** rather than re-emitting stale finals (a cached `public` followed by an origin `304 Cache-Control: private, no-store` must not keep serving the old public policy). Artifact absence or a revision mismatch strips internal preconditions before obtaining that full response. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | +| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals stored as a **distinct head-only artifact type that never satisfies a later GET** — when a stored GET artifact exists a HEAD may **update** it only if validators and `Content-Length` match (RFC 9111 §4.3.5), a mismatch **invalidating** the stored GET artifact rather than updating it | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). @@ -282,14 +295,17 @@ degree of freedom is closed: DataDome's mandatory response-directed mapping set — that reduction needs explicit product **and vendor** acceptance: sign-off item 28; a violating operation is rejected whole (the batch rule). **Both - documented cookie-return forms lower into this one typed operation**: - an ordinary `Set-Cookie` from the vendor response _and_ the - session-by-header form (`X-DataDome-X-Set-Cookie: true` request flag → - updated value returned in the vendor's `X-Set-Cookie` response field) - are parsed and validated identically as the typed `datadome` cookie — - and the `X-Set-Cookie` field itself is **never forwarded** to the - browser (unmapped, it would either fork implementations or invalidate - the whole batch and silently fail the challenge open). Every `ts-*` + sessionByHeader is unsupported by default and never requested** — TS + does not send `X-DataDome-X-Set-Cookie: true`, so the vendor uses + ordinary `Set-Cookie`, which lowers into the typed `datadome` + operation. Header-session mode exists for clients that cannot use a + cookie and expects JavaScript to receive `X-Set-Cookie` and `X-DD-B` + into local storage; supporting it means forwarding those as typed, + owner-scoped headers **and** accepting a JavaScript/local-storage + identifier observer — an explicit opt-in under **sign-off 23** (which + now enumerates that observer), not a silent cookie substitution. + Without the opt-in, an incoming `X-Set-Cookie` is unclassified → + Continue. Every `ts-*` name is rejected. **Read is owner-only, and the strip inventory is exhaustive, not integration-scoped** — the browser sends `datadome` in the ordinary `Cookie` header, so it is removed from **every non-DataDome surface**: @@ -373,6 +389,31 @@ no-cache` has no standardized meaning, RFC 9111 §5.4) are **dropped Continue). If the vendor ever requires more, it arrives as a reviewed allowlist-file addition. A _Continue_ decision may not touch representation metadata of publisher bytes. +- **Respond transport is bounded.** The challenge body has a maximum + size (64 KiB) and a complete-response deadline; TS sends + `Accept-Encoding: identity` on challenge fetches so there is no + encoded body to reframe (`Content-Encoding` is reserved anyway), and + `Content-Length` is recomputed from the actual bytes before Respond + commits. Exceeding the size or deadline fails the batch → Continue. +- **Every documented DataDome pointer has exactly one assigned + outcome, and the vendor's documented response is a passing fixture.** + The outcome table (also in `datadome-header-allowlist.md`): + `Set-Cookie` / `X-Set-Cookie` → typed `datadome` cookie operation; + `Location`, `Content-Type` → forward (Respond only); `Cache-Control` → + restricted merge; `Pragma` → drop-individually (logged); `X-DataDome`, + `X-DD-*` → **forward as owner-scoped typed headers** (they are DataDome + telemetry, not publisher policy); anything unclassified → invalidate + (→ Continue). The fixture asserts DataDome's documented example + (`Set-Cookie`, `Pragma`, `X-DataDome`, `Cache-Control`) stays + **Respond** and emits exactly the mapped fields — none of it drops to + Continue. +- **Every security Respond ends uncacheable, unconditionally.** After + the decision's fields are applied, the invariant pass forces + `Cache-Control: private, no-store` and strips all CDN cache fields on + **every** Respond regardless of status, vendor pointers, or cookie + emission — a cookie-less `301` challenge with no effective vendor + cache header could otherwise be heuristically stored and served to + unrelated clients. - **One global order:** core finalization → ordinary mutators → security effects → **final cache/privacy invariant pass, unconditionally last**. Security precedence over publisher-facing @@ -429,7 +470,7 @@ no-cache` has no standardized meaning, RFC 9111 §5.4) are **dropped This is a modest feature plus tests with zero coupling to the provider architecture or, in its v1 headers-only form (§3), to the permission -model. It lands whenever its first real consumer is identified (§4.2); +model. It lands whenever its first real consumer is identified (§4, item 3); cookie operations arrive only with their own follow-up spec (§3) and its permission-model coupling. If no consumer materializes, it does not land; being unblocked is not a reason to ship @@ -440,7 +481,7 @@ scaffolding. This spec supersedes #782 on the following points; the issue is updated to reference this spec when the PR merges: -| #782 says | This spec says | Why | -| -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | -| Mutations apply to the outbound response generally | Eligibility is the explicit §3a matrix, centered on processed documents | Pass-through/error/304 mutation has different semantics per adapter; enumerating beats implying | -| Ship the trait + registry + adapter application | Additionally: structured operations API (§2), reserved surface (§3), and a real consumer in the same PR (§4.2) | PR #838 shipped the trait with zero call sites; an unrestricted `&mut HeaderMap` cannot enforce any collision policy | +| #782 says | This spec says | Why | +| -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | +| Mutations apply to the outbound response generally | Eligibility is the explicit §3a matrix, centered on processed documents | Pass-through/error/304 mutation has different semantics per adapter; enumerating beats implying | +| Ship the trait + registry + adapter application | Additionally: structured operations API (§2), reserved surface (§3), and a real consumer in the same PR (§4, item 3) | PR #838 shipped the trait with zero call sites; an unrestricted `&mut HeaderMap` cannot enforce any collision policy | diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index e5936f105..6744912e7 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -559,8 +559,15 @@ and the fail-closed marker: an older `valid_until`), and an equal revision is idempotent and must be payload-equivalent (a mismatch at equal revision is a hard error, not a merge). The r2-row → r3-row → r3-authority → delayed - r2-authority schedule is a named test. **Revision _r_ is committed — usable by S2S, visible - to the absence decision — only when the strong record reports it**; a + r2-authority schedule is a named test. **Revision _r_ is committed — usable by S2S, visible to the absence + decision — only when the strong record reports it, and identity/S2S + use requires `row.provenance_revision == authority.summary_revision`, + failing closed in _both_ mismatch directions** (row r2 + summary r1 is + the ordinary uncommitted case; summary r2 + eventually-stale row r1 is + the inverse another region can strongly read — so the recompute takes + **every** input, jurisdiction included, from the strong summary, never + the row, and refuses on any revision mismatch, closing the + wrong-jurisdiction egress). A row at _r_ whose summary still reads _r−1_ is simply uncommitted detail, and a crash between the writes leaves a recoverable state (the next live resolution re-runs step 2 via `AuthorityRefresh`), never a @@ -715,11 +722,16 @@ section malformed-present (blocks grants, never withdraws). (and `iab_gpp` 0.1.2) supports sections 7–23 only and models `usnat` v2 while the snapshot pins v1 — implementation must reject versions the library happens to decode but the snapshot disallows. Sections - 24–27 are **not** a decoder work item: with no reproducible official - layout they are reserved and inert (national-only for those states — - an accepted limitation, sign-off 32); the earlier "Maryland opt-out - must not vanish / extend the decoder for 24–27" reading is withdrawn - as incompatible with reserved status. + 24–27 are **not** a decoder work item and carry **no accepted + version** (the snapshot lists them in a separate _reserved_ table, + not the accepted-version table — an accepted-version entry plus + "inert" prose let two implementations diverge): with no reproducible + official layout they are reserved and inert (national-only for those + states — sign-off 32). Their **presence differs from an unknown + section only in logging**: both contribute nothing, but a reserved + ID is expected-inert while an unknown ID is flagged for snapshot + review. The earlier "Maryland opt-out must not vanish / extend the + decoder" reading is withdrawn. The implementation PR cross-checks this list against both the current decoder's section set and the official registry, and the accepted version per section is **pinned to the vendored registry snapshot @@ -825,21 +837,15 @@ migration story unresolvable (migration spec §2, rows 5 and 7). A **policy revision** has a defined identity: the canonical content digest of the `[permissions]` section — **defined**: SHA-256 with domain -tag `tspol1|` over the canonical JSON of the parsed policy (keys sorted, -UTF-8, defaults materialized, no insignificant whitespace), with -cross-language test vectors a conformance requirement (identity — republishing identical -policy yields the same digest) paired with the **config-store's globally assigned activation -version** (the `ts config push` version — one fleet-wide ordered -sequence, not a per-instance counter: "monotonic per instance" gave -generation 12 on one instance no relation to 12 on another, making -cross-instance provenance comparison undefined). Provenance -stores both; comparisons order by generation and equate by digest, so a -rollback is a _new_ generation carrying an _old_ digest, with defined -semantics on both axes. - -A policy edit propagates through the config store, so a fleet briefly -mixes revisions. The contract: instances stamp every resolution and every -provenance write with the policy revision they used (already required by +tag `tspol1|` over the canonical JSON of the parsed policy (keys sorted +lexicographically by UTF-8 code unit, numbers shortest round-trip, +defaults materialized, no insignificant whitespace), cross-language +vectors required. **The other cache-tuple inputs are domain-separated +hashes of effective configuration, adapter-independent**: +integration-registry revision = `tsreg1|` over the canonical-JSON +`(id, version)` list; config revision = `tscfg1|` over the effective +config blob — so adapters without a native push version still derive +identical revisions from identical configuration by §7); the mixing window is bounded by config propagation and observable via the config-version metric; and mixed-revision irreversibility is bounded and **accepted, not denied** (sign-off 19): destructive withdrawal triggers are user diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index 3ae232c8c..cee6c23ed 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -317,15 +317,22 @@ variant**. Therefore: transition bumping an epoch; this metadata key requires globally observable strong reads _in addition to_ CAS — a capability cell, since CAS alone says nothing about what other instances currently - see). An N+1 instance starting while the flag is active CASes it to - `suspended`, **reads the committed suspension back, then waits one - full lease window before minting**. N+2 instances prove `active` at - classification time through a **bounded lease** (strong read at lease - expiry, lease ≤ L): suspension is therefore fleet-effective within L, - and because N+1 does not mint until L has elapsed after its committed - suspension, **no unstubbed cookie can exist while any instance still - holds an `active` lease** — the stale-lease misclassification race is - closed by construction, not by luck. Under `suspended`, rowless + see). The suspension transition stamps a **fleet-stable `not_before` + deadline** = commit time + L into the suspended epoch, where **L is an + assigned constant (120 s) stored in the metadata**, not a per-instance + timer: **every** N+1 instance — the one that suspended, a second that + starts and reads an already-suspended state, or one recovering after + the suspender crashed — refuses to mint until `now ≥ not_before` + (globally strong read of the epoch). N+2 instances prove `active` at + classification time through a **bounded lease ≤ L** (strong read at + lease expiry). So suspension is fleet-effective within L, no minting + occurs before `not_before` = suspension + L, and **no unstubbed + cookie can exist while any instance still holds an `active` lease** — + closed by construction, crash-safe, and independent of which instance + suspended. The state chain is cyclic: + absent → active → suspended → re-attested-active → **suspended** + (a second rollback) → … ; CAS losers on any transition re-read and + retry against the winner's epoch. Under `suspended`, rowless _classification_ stops but **`w` consultation and enforcement continue** (withdrawn stays withdrawn). Re-activation after roll-forward requires complete re-attestation over the gap window. @@ -344,7 +351,7 @@ variant**. Therefore: discovery consults the prefix's `w` state before any live or S2S use** (a pending or saturated entry means promotion-then-denial per the runtime matrix, §6.2), so a withdrawn suffix cannot slip into use - through the row path during the window. Graphless-era cookies never got a stub because + through the row path. **`w` consultation is keyed on the record's `valid_until`, not the rowless-classification flag** — the flag may clear after one cookie lifetime while `w` is retained through the longer max(cookie, row, S2S) horizon, and a late row must still find a live `w`; enforcement ends only when the `w` record itself expires. Graphless-era cookies never got a stub because they have no row for the scan to find; no eventual read participates. The flag itself is specified: a named deployment-metadata key (write-once/CAS class), set by the §4.2 migration runbook step (migration spec §4) only on deployments that actually ran graphless @@ -588,20 +595,20 @@ Startup validation (§6) covers configuration; this covers what happens when a healthy configuration meets an unhealthy runtime. Every row logs at `error` with a metric; none is silent: -| Failure | Behavior | -| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | -| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | -| Authority-state commit fails after the row commit at mint | No cookie, no eligibility (the strong record never reported the revision); the orphan row expires by TTL; counted — **no recovery claim**, nothing can find it | -| `w` read fails (rowless path or migration-window row discovery) | Fail closed: the cookie/row is **indeterminate** — no use, no mint, no expiry this request | -| `w` write fails at rowless withdrawal | Cookie retained (withdrawal did not durably occur); durable client signal retries next presentation | -| `w` saturation encountered | Rowless cookies under the prefix are treated withdrawn (§5); later-discovered **real rows** promote only on a listed suffix-hash match — saturation never blanket-revokes row-backed identities, and overflow suffixes beyond the cap lose promotion (declared residual) | -| Promotion (listed suffix-hash matches a discovered row) fails | The row is denied all use until the promoted family revocation commits; retried on next observation | -| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | -| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | -| Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | -| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | -| Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | +| Failure | Behavior | +| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | +| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | +| Authority-state commit fails after the row commit at mint | No cookie, no eligibility (the strong record never reported the revision); the orphan row expires by TTL; counted — **no recovery claim**, nothing can find it | +| `w` read fails (rowless path or migration-window row discovery) | Fail closed: the cookie/row is **indeterminate** — no use, no mint, no expiry this request | +| `w` write fails at rowless withdrawal | Cookie retained (withdrawal did not durably occur); durable client signal retries next presentation | +| `w` saturation encountered, a real row surfaces under the prefix | **Denied, then promoted to a family revocation** — a saturated prefix means the safe assumption is "withdrawn", so any real row under it is denied all use and its family revoked, listed-hash or overflow alike. The earlier "overflow loses promotion, never blanket-denies" rule left a completed rowless withdrawal usable the instant its row surfaced; that resurrection is closed here, not left to retention. The collateral — a non-abuser row under a saturated NAT prefix is revoked — is sign-off 30 | +| Promotion (listed suffix-hash matches a discovered row) fails | The row is denied all use until the promoted family revocation commits; retried on next observation | +| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | +| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | +| Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | +| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | +| Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | The **degraded-graph health signal** referenced above and by the withdrawal contract is a defined state machine, not a vibe: it is @@ -631,17 +638,17 @@ solves. **Physical key grammar.** Core constructs every key; providers supply only the bounded suffix: -| Record class | Key | Notes | -| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ **123** bytes (128-byte total key cap minus tag and provider code — an earlier 128 here contradicted the constructor; 123/124 boundary fixtures required), KV-safe alphabet. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | -| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | -| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | -| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | -| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | -| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8), saturation flag, CAS version, created-at, `valid_until` ≥ **max(cookie lifetime, row/S2S authority horizon)** — a `w` expiring before the row horizon would let an overflow row's identity resurface (the overflow non-promotion residual is sign-off 30); readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | -| Deployment metadata | `m` + fixed metadata name (fixed-width grammar; schema floor, graphless-migration flag) | Write-once/CAS class; value carries schema version, state, epoch, set-at; the graphless flag's lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded | -| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | -| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | +| Record class | Key | Notes | +| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ **123** bytes (128-byte total key cap minus tag and provider code — an earlier 128 here contradicted the constructor; 123/124 boundary fixtures required), portable alphabet **`[A-Za-z0-9._~-]`** (not "KV-safe per host", which would break the single cross-adapter grammar; providers with native identifiers outside it emit base64url suffixes); ≤ **123** bytes with 123/124 boundary fixtures. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | +| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | +| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | +| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | +| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | +| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8), saturation flag, CAS version, created-at, `valid_until` ≥ **max(cookie lifetime, row/S2S authority horizon)** — a `w` expiring before the row horizon would let an overflow row's identity resurface (the overflow non-promotion residual is sign-off 30); readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | +| Deployment metadata | `m` + fixed metadata name (fixed-width grammar; schema floor, graphless-migration flag) | Write-once/CAS class; value carries schema version, state, epoch, set-at; the graphless flag's lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded | +| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | +| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | Grammars are pairwise non-intersecting by their literal prefixes (plus the reserved hmac grammar), and every record value carries a `kind` @@ -677,6 +684,11 @@ concatenated in that order, no separators beyond the tag's `|`. **Known-answer vector**: input `tsfam1|` + `i` + `hmac` + `{64×"a"}.AbC123` → `e90616c381f64965b8326f17108c3c481cee932b2d7f8af783c7bdc2e21591ef`. +A **non-HMAC vector** disambiguates "canonical graph-key bytes" = +the provider's `graph_key_suffix` bytes (not the full physical key — +the tag and provider code are already separate derivation fields): +input `tsfam1|` + `i` + `vend` + `abcdef` → +`278e67d721babaee94690cd246ee567d6ce709c43f8737c2e9dce1e1119c6be1`. `w` suffix hashes are likewise assigned: SHA-256 with domain tag `tswsx1|` over the raw suffix bytes, truncated to 16 bytes, lowercase hex (32 chars) — vector: `AbC123` → `08cb55acf42929772862e82b0960c134`. @@ -707,12 +719,20 @@ expired entries are inert), and the provenance revision a clear references; positive side (the summary, **every field the permission spec's absence/replay decisions consume — a reduced schema cannot reproduce them**): kind (user evidence vs policy baseline), grant -basis/source class, policy revision (digest + activation generation), -`valid_until`, provenance revision, evidence timestamp, and the +basis/source class, policy revision (digest + activation generation), **resolved +jurisdiction** (so S2S never reads it from an eventually stale row — +the summary is self-sufficient for the recompute), `valid_until`, +provenance revision, evidence timestamp, and the and a **bounded replay history** whose slots are keyed by a **timestamp-independent `state_key`** — (source class, semantic result -digest _excluding_ `LastUpdated`) — distinct from the _evidence digest_ -(which for TCF includes `LastUpdated` for recency): keying slots on the +digest _excluding_ `LastUpdated`) — distinct from the _evidence +digest_ (which for TCF includes `LastUpdated` for recency), both with +**canonical wire construction**: `state_key` = SHA-256 `tsstk1|` over +`source-class-enum-byte | canonical-semantic-result` (enum bytes: tcf=1, +gpp=2, usp=3; timestamps integer epoch-ms where present), evidence +digest = SHA-256 `tsevd1|` over the same plus `LastUpdated`; vector: TCF +(P1 grant, P4 refuse) `tsstk1|tcf|p1=grant,p4=refuse` → +`a49148a0e3b486fd93a141404857868871319a7e1f2ef85b5499aed80c7e59df`: keying slots on the evidence digest would give every renewal a fresh key and make "updates its slot in place" impossible, the incompatibility an earlier draft shipped. A slot stores its `state_key`, the current evidence @@ -727,9 +747,14 @@ cannot hold independent timestamps for multiple overflow digests): when all slots hold distinct in-horizon states, the record sets a `saturation_epoch` with `saturated_until = now + consent TTL`, **fixed at entry and never extended by later overflow values**; while -saturated, novel values cannot grant (fail restrictive — overflow -evidence of either polarity affects the live request but is not -stored, so replaying it advances nothing), recovery is automatic at +saturated, novel values cannot grant (fail restrictive); a +**restrictive overflow** (timestamp-less opt-out or malformed) does not +mint fresh suppression at the current observation time — it is +recorded, if at all, under an **epoch-scoped restrictive marker whose +timestamp and `valid_until` are pinned to saturation-epoch entry**, so +replaying it later cannot advance its observation time or extend +suppression (the unpinned version let repetition renew denial forever), +recovery is automatic at epoch expiry as slots free, and saturation is a first-class metric — the cap and its denial behavior are **sign-off item 31**; record level: family ID, CAS version counter, schema version, stub marker (backfill, §5); unknown-field and range validation apply @@ -794,18 +819,18 @@ Requirements: **per-record-class consistency requirements**, because "has KV" says nothing about whether revocation is observable: - | Record class | Required semantics | - | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | - | Family revocation records | **Globally observable strong consistency** — every instance's read observes a committed revocation, not merely the writing session's own writes (writer-scoped read-your-writes is insufficient for a fleet). Cloudflare Workers KV is **not eligible** — "60 seconds or more" is an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | - | Authority-state records (suppression + positive summary) | **Globally observable strong reads AND linearizable per-key CAS** — CAS alone orders writes, but a stale successful _read_ on another instance would authorize egress after a committed suppression ("read failures fail closed" does not cover stale successes); both properties are adapter eligibility gates | - | Identity-row mutation | **Generation CAS** (conditional write on row generation) with reread/recompute on conflict — rows are heavily mutable (snapshots replaced, partner IDs merged, derived state refreshed), so unordered last-writer-wins loses newer evidence and mappings; _visibility_ may stay eventual, unordered _mutation_ may not. Fastly KV offers generation-marker conditional writes; Workers KV's documented concurrent last-write-wins is ineligible for mutation-bearing rows | - | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | - | Deployment metadata (schema floor) | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | - | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | - | Rowless prefix-withdrawal (`w`) records | **Globally strong reads + linearizable CAS** (same bar as authority-state), plus the **bounded listing-visibility window** the backfill scan depends on — all three are eligibility gates for the graphless migration; retention ≥ maximum cookie lifetime | - | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | - | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | + | Record class | Required semantics | + | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | + | Family revocation records | **Globally observable strong consistency** — every instance's read observes a committed revocation, not merely the writing session's own writes (writer-scoped read-your-writes is insufficient for a fleet). Cloudflare Workers KV is **not eligible** — "60 seconds or more" is an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | + | Authority-state records (suppression + positive summary) | **Globally observable strong reads AND linearizable per-key CAS** — CAS alone orders writes, but a stale successful _read_ on another instance would authorize egress after a committed suppression ("read failures fail closed" does not cover stale successes); both properties are adapter eligibility gates | + | Identity-row mutation | **Generation CAS** (conditional write on row generation) with reread/recompute on conflict — rows are heavily mutable (snapshots replaced, partner IDs merged, derived state refreshed), so unordered last-writer-wins loses newer evidence and mappings; _visibility_ may stay eventual, unordered _mutation_ may not. Fastly KV offers generation-marker conditional writes; Workers KV's documented concurrent last-write-wins is ineligible for mutation-bearing rows | + | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | + | Deployment metadata — schema floor (write-once/CAS); **graphless flag additionally requires globally observable strong reads** (N+2 lease revalidation and N+1's `not_before` barrier both need globally current reads, not just write-once) | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | + | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | + | Rowless prefix-withdrawal (`w`) records | **Globally strong reads + linearizable CAS** (same bar as authority-state), plus the **bounded listing-visibility window** the backfill scan depends on — all three are eligibility gates for the graphless migration; retention ≥ maximum cookie lifetime | + | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | + | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | Every record class additionally declares **durability and maximum retention**: a store passing the consistency check but capping TTLs @@ -826,19 +851,19 @@ Requirements: them is how a "yes" cell hides an unusable feature. Feature eligibility requires wired, not merely available: - | Capability | Fastly | Axum (dev) | Cloudflare | Spin | - | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | - | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | - | Prefix listing (cluster) | Used today, but **ratification requires a cited platform completeness bound, pagination behavior, and failure semantics** — the graphless scan's settle window cannot be derived from "works in practice" | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | - | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | - | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | - | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | - | Device host evidence (JA4/H2) | Yes | No | No | No | - | Authority-state: global strong reads + CAS | Conditional writes available (generation marker); **globally current read semantics to verify** — both required, wiring to verify | **Unavailable** (no store wired) | Workers KV: **ineligible**; Durable Objects: feasible, not wired | **Unavailable** | - | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | - | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | - | Deployment metadata (write-once/CAS) | **Not wired** — needs a primitive distinct from the config store | **Unavailable** | DO: feasible, not wired | **Unavailable** | - | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | + | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | + | Prefix listing (cluster) | Used today, but **ratification requires a cited platform completeness bound, pagination behavior, and failure semantics** — the graphless scan's settle window cannot be derived from "works in practice" | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | + | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | + | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | + | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | + | Device host evidence (JA4/H2) | Yes | No | No | No | + | Authority-state: global strong reads + CAS | Conditional writes available (generation marker); **globally current read semantics to verify** — both required, wiring to verify | **Unavailable** (no store wired) | Workers KV: **ineligible**; Durable Objects: feasible, not wired | **Unavailable** | + | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Deployment metadata — floor (write-once/CAS) **and graphless flag (globally strong reads + CAS)** | **Not wired** — needs a primitive distinct from the config store | **Unavailable** | DO: feasible, not wired | **Unavailable** | + | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 2b8bd1147..e77809e86 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -128,17 +128,21 @@ Requirements: treat revoked identities as live (aliases are reserved-future with the rewrite deferral, providers spec §6.1). N+1 must also **write family revocation records** — a withdrawal arriving on a - rolled-back N+1 fleet must still revoke. **Authority-state - (suppression) is different: N+1 neither creates nor clears it.** - Creating would be safe, but clearing now requires the - `AuthorityRefresh` provenance protocol over revision-bearing rows - that N+1 (a v1 writer) cannot produce — so an N+1 clearing without - the fence would expose stale positive snapshots, and one clearing - with it would need the whole N+2 write model. Instead: N+1 - **reads** authority-state fully and fails closed on suppressed - permissions; suppression created by N+2 stays in force during a - rollback, and **recovery (clearing) waits for roll-forward** — a - protective, declared limitation, not an undefined one. + rolled-back N+1 fleet must still revoke. **Authority-state: N+1 may create + stubs and write negative entries, but never positive commits or + clears.** The observed-row admission sequences (providers spec §5) + need an s-class stub before revoking an untouched v1 row and a + suppression CAS for a non-destructive signal — both are + **negative/stub writes N+1 is permitted**, so a first post-upgrade + GPC or SharingOptOut executes fully on N+1 (revocation, or a + persisted suppression, not a deny-and-forget). What N+1 must **not** + do is commit or clear **positive** authority (that needs the + `AuthorityRefresh` revision protocol a v1 writer cannot produce): + suppression created under N+2 stays in force through a rollback and + its **clearing waits for roll-forward** — a protective, declared + limitation. This is one contract, resolving the earlier + "neither creates nor clears" wording that made first-upgrade + withdrawal unexecutable. **N+1's identity-write behavior is v1, explicitly** — this resolves what was an impossible trilemma (write rows without provenance, @@ -481,37 +485,37 @@ the deciders, the date). The Decision-record column holds the link (`—` while open); an unratified row reverts to open, not to silently implemented. -| # | Decision | Where | Decision record | Status | -| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------- | ------------------------------------------- | -| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | — | open | -| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | — | open | -| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | — | open | -| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | — | open | -| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | — | open | -| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | — | open | -| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | — | open | -| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | — | open | -| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | — | superseded by descope (ratify the deferral) | -| 10 | Session-cookie exemption question | hook §3 | — | deferred with item 9 | -| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | — | open | -| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | — | open | -| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | — | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | — | open | -| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | — | open | -| 16 | Timestamp-less opt-out suppression is **TTL-sticky**: within its consent-TTL lifetime only a newer timestamped grant clears it; at `valid_until` it goes inert automatically (not user-sticky-forever, not administratively sticky — administrative clear is an optional early exit) | permission §4.3 | — | open | -| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | — | open | -| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | — | open | -| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | — | open | -| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | — | open | -| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | — | open | -| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | — | open | -| 23 | **Open question, not ratified**: may DataDome's security identifier (tag injection, `datadome` cookie, `X-DataDome-ClientID` read and vendor egress) operate outside the permission model? The decision must enumerate exactly which consumers may observe the cookie/ClientID — the enumerated observers are the security channel **and the publisher origin, which receives `X-DataDome-ClientID` via the upstream overlay** — "owner-scoped overlay" names the mechanism, this row names the recipient — plus retention and whether TS withdrawal expires it | hook §4a; permission §7 | — | open | -| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | — | open | -| 25 | Batch-sync stored-jurisdiction maximum age (consent-TTL horizon): a mover into GDPR stops old-rule egress at the horizon; a mover out is denied until a live visit | permission §7 | — | open | -| 26 | Embedded GPP GPC maps to the destructive global opt-out (header-OR-embedded aggregation) | permission §4.5 | — | open | -| 27 | Proxy-mode minimal opt-out extraction (decode only §4.5-mapped opt-out fields; no grants) | permission §4.4 | — | open | -| 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance | hook §4a; `datadome-header-allowlist.md` | — | open | -| 29 | Unverifiable roaming rowless cookies receive best-effort cookie-only expiry (admission rules forbid durable records for unverified values); a lost response can leave the cookie usable on the old network until re-presented | providers §5 | — | open | -| 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, observability, **and the overflow residual: a real row beyond the listed hashes permanently loses promotion and could resurface if `w` retention or the flag lifecycle were mishandled** — all in scope | providers §5 | — | open | -| 31 | Replay-history capacity (16 semantic-state slots + rolling restrictive slot): while saturated, novel values cannot grant — fresh consent in unusual multi-CMP setups can be rejected until slots expire | permission §4.3; providers wire schema | — | open | -| 32 | GPP sections 24–27 (MD/IN/KY/RI) are reserved with **national-section-only** handling until an official binary layout can be vendored — a state-specific opt-out expressed only in an undecodable state section is not honored | permission §4.5; `gpp-registry-snapshot.md` | — | open | +| # | Decision | Where | Decision record | Status | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------- | --------------------------------------------------------------------------------------- | +| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | — | open | +| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | — | open | +| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | — | open | +| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | — | open | +| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | — | open | +| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | — | open | +| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | — | open | +| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | — | open | +| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | — | open (descope ratification still required; record-less ⇒ open per the decisions README) | +| 10 | Session-cookie exemption question | hook §3 | — | open (deferred, but record-less ⇒ open per the decisions README) | +| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | — | open | +| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | — | open | +| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | — | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | — | open | +| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | — | open | +| 16 | Timestamp-less opt-out suppression is **TTL-sticky**: within its consent-TTL lifetime only a newer timestamped grant clears it; at `valid_until` it goes inert automatically (not user-sticky-forever, not administratively sticky — administrative clear is an optional early exit) | permission §4.3 | — | open | +| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | — | open | +| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | — | open | +| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | — | open | +| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | — | open | +| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | — | open | +| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | — | open | +| 23 | **Open question, not ratified**: may DataDome's security identifier (tag injection, `datadome` cookie, `X-DataDome-ClientID` read and vendor egress) operate outside the permission model? The decision must enumerate exactly which consumers may observe the cookie/ClientID — the enumerated observers are the security channel **and the publisher origin, which receives `X-DataDome-ClientID` via the upstream overlay** — "owner-scoped overlay" names the mechanism, this row names the recipient — plus retention and whether TS withdrawal expires it | hook §4a; permission §7 | — | open | +| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | — | open | +| 25 | Batch-sync stored-jurisdiction maximum age (consent-TTL horizon): a mover into GDPR stops old-rule egress at the horizon; a mover out is denied until a live visit | permission §7 | — | open | +| 26 | Embedded GPP GPC maps to the destructive global opt-out (header-OR-embedded aggregation) | permission §4.5 | — | open | +| 27 | Proxy-mode minimal opt-out extraction (decode only §4.5-mapped opt-out fields; no grants) | permission §4.4 | — | open | +| 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance | hook §4a; `datadome-header-allowlist.md` | — | open | +| 29 | Unverifiable roaming rowless cookies receive best-effort cookie-only expiry (admission rules forbid durable records for unverified values); a lost response can leave the cookie usable on the old network until re-presented | providers §5 | — | open | +| 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, observability, **and the saturation collateral: under a saturated prefix, any real row (listed or overflow) is denied and revoked immediately on surfacing — a non-abuser NAT-cohort row can be revoked; `w` is retained through the max(cookie, row, S2S) horizon and consulted by `valid_until`, not the flag, so this is deterministic, not a retention accident** — all in scope | providers §5 | — | open | +| 31 | Replay-history capacity (16 semantic-state slots + a fixed saturation epoch with an epoch-pinned restrictive marker): while saturated, novel values cannot grant — fresh consent in unusual multi-CMP setups can be rejected until the epoch expires | permission §4.3; providers wire schema | — | open | +| 32 | GPP sections 24–27 (MD/IN/KY/RI) are reserved with **national-section-only** handling until an official binary layout can be vendored — a state-specific opt-out expressed only in an undecodable state section is not honored | permission §4.5; `gpp-registry-snapshot.md` | — | open | diff --git a/docs/superpowers/specs/datadome-header-allowlist.md b/docs/superpowers/specs/datadome-header-allowlist.md index 1f8c24395..93e846df0 100644 --- a/docs/superpowers/specs/datadome-header-allowlist.md +++ b/docs/superpowers/specs/datadome-header-allowlist.md @@ -24,3 +24,19 @@ set mean: nothing else is accepted until a reviewed commit adds it. Note: the vendor's `X-Set-Cookie` response field is **not** a forwardable header — it lowers into the typed `datadome` cookie operation (hook spec §4a) and never reaches the browser as a header. + +## Pointer outcome table (every documented pointer, exactly one outcome) + +| Pointer | Outcome | +| ---------------------------- | ---------------------------------------------------- | +| `Set-Cookie`, `X-Set-Cookie` | typed `datadome` cookie operation (§4a) | +| `Location` | forward (Respond, 3xx only) | +| `Content-Type` | forward (Respond only) | +| `Cache-Control` | restricted merge | +| `Pragma` | drop-individually (logged), never batch-invalidating | +| `X-DataDome`, `X-DD-*` | forward as owner-scoped typed telemetry headers | +| anything unclassified | invalidate the batch → Continue | + +The documented vendor response (`Set-Cookie`, `Pragma`, `X-DataDome`, +`Cache-Control`) is a verbatim fixture asserting the decision stays +**Respond** and the mapped fields are emitted exactly. diff --git a/docs/superpowers/specs/gpp-registry-snapshot.md b/docs/superpowers/specs/gpp-registry-snapshot.md index 8565e6bd2..38fe85ebb 100644 --- a/docs/superpowers/specs/gpp-registry-snapshot.md +++ b/docs/superpowers/specs/gpp-registry-snapshot.md @@ -25,10 +25,6 @@ here is treated as malformed-present (permission spec §4.4). | 21 | usnj | 1 | | 22 | ustn | 1 | | 23 | usmn | 1 | -| 24 | usmd | 1 | -| 25 | usin | 1 | -| 26 | usky | 1 | -| 27 | usri | 1 | Version values for sections 6–23 were captured from the IAB registry at the time of writing and are re-verified against the official registry as @@ -36,3 +32,27 @@ part of ratification review. Sections 24–27 have assigned IDs but no reproducibly published binary layouts in the official sources as of this snapshot; they are reserved and inert until an official layout can be vendored here. Any change is a reviewed change to this file. + +## Reserved sections — NOT accepted, no version + +These state sections have assigned IDs but no reproducibly published +official binary layout as of this snapshot. They are **not** in the +accepted-version table above: an implementation MUST NOT decode them, +and a request carrying one behaves national-section-only (permission +spec §4.5, sign-off 32). A reserved ID is _expected-inert_; an unknown +ID (outside both tables) is _flagged for snapshot review_ — the only +observable difference is logging. + +| GPP section ID | State | Status | +| -------------- | --------- | ----------------------------- | +| 24 | usmd (MD) | reserved — no official layout | +| 25 | usin (IN) | reserved — no official layout | +| 26 | usky (KY) | reserved — no official layout | +| 27 | usri (RI) | reserved — no official layout | + +## Provenance and vectors + +Supported sections (6–23) pin to the official IAB GPP registry revision +recorded by the implementation PR (immutable upstream commit hash), with +per-section encoded conformance vectors vendored alongside. A date is not +a revision; the commit hash is the reproducible authority. diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index bff46332a..a3fb54046 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -287,36 +287,77 @@ All rows text-added per the R13 vocabulary. | P3 batch (§5 activation lead fixed to the commit point; §4 device wording; typo/punctuation/metric residue; client resv notation; old DataDome spec carries a supersession banner now) | text-added | | Sign-off table restructured **decision-centric** (Owner column → Decision-record link; deciders live in the records) per maintainer direction | text-added | -## Round 15 — review at f3eacf59 - -All rows text-added per the R13 vocabulary. - -| Finding | Status | -| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | -| P1 N+1 rollback invalidates rowless proof | text-added: any N+1 startup CASes the flag to suspended; re-activation needs full re-attestation; N+2→N+1-mint→N+2 named test | -| P1 admission rule blocks first-upgrade withdrawal / promotion | text-added: observed-row sequence adopted verbatim (row read → stub create-if-absent, no positive authority → family revocation → use denied between → cookie expiry); permission spec §4.3 references it | -| P1 `w` protocol absent from matrices/runtime/tests | text-added: abstract row (global reads + CAS + listing bound as migration gates), four runtime-failure rows, migration-window consult rule, saturation promotes listed hashes only (blanket row revocation excluded; overflow residual declared) | -| P1 replay saturation unimplementable | text-added: semantic-state slots with in-place TCF renewal updates (ordinary renewals never consume capacity), rolling restrictive slot outside the 16, serialized `saturated`/`saturated_until`, automatic recovery, metric; sign-off 31 | -| P1 TTL-sticky contradiction in migration | text-added: irreversible-artifact list corrected — suppression is TTL-bounded, administrative clear optional | -| P1 GPP 24–27 not reproducibly official | text-added: demoted to reserved-pending-official-schema in map and snapshot; those states behave national-only; the official-coverage claim retracted | -| P1 X-Set-Cookie unmapped | text-added: both cookie-return forms lower into the typed operation; the field itself never forwarded; allowlist file noted | -| P1 response allowlist vs representation rule | text-added: Respond may describe its body with `Content-Type` only; encoding/validators stay reserved; vendor needs arrive as allowlist-file review | -| P1 cache merge weakens origin policy | text-added: unknown snapshot directives preserved verbatim (mutations still drop unknowns); `Expires` in the freshness bound; no introduced `max-age`/`s-maxage` absent a snapshot upper bound (RFC 9111 §5.2.3/§5.3 cited) | -| P2 batch (HEAD serves the persisted GET artifact; fleet-stable revision tuple — registry content hash, global push version, build constant; tie winner's complete tuple survives (no timestamp ratchet); grammar consistency — suffix ≤123, `m` + registry index instead of padded names, full tag enumeration incl. `w`/`m`, registry-file note fixed, family-ID derivation with domain tag `tsfam1 | ` and vectors; graphless runbook with correct pointer, abort/re-attestation/clearing; policy digest defined (`tspol1 | `, canonical JSON, vectors); pointer parser total contract with fail-open semantics and cookie-source priority; `Pragma` moved to the drop-individually list — **the R15 ledger falsely said the allowlist file was updated; it was not until R16**) | text-added | -| P3 batch (wire-schema sentence restructured with the saturation rewrite; row 3e phrase completed; sign-off rows renumbered into order) | text-added | - -## Round 16 — review at 9a8596ee - -All rows text-added per the R13 vocabulary. - -| Finding | Status | -| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | -| P1 dual suppression admission contracts | text-added: two observed-row sequences (destructive: stub → revocation → expiry; non-destructive: stub → suppression CAS → deny-while-incomplete, family and cookie retained); "unconditional" clarified as independent-of-prior-authority, never of family admission | -| P1 flag suspension not fleet-linearizable | text-added: globally-strong state machine with epochs; N+2 validates active on a bounded lease; N+1 reads back suspension then waits one lease window before minting — no unstubbed cookie can coexist with a live active lease; `w` enforcement continues under suspended | -| P1 GPP 24–27 accept/inert/decode contradiction | text-added: removed from decoder work items and accepted versions; national-only limitation is sign-off 32; "Maryland must not vanish" withdrawn | -| P1 replay keys incompatible / rolling slot unrepresentable | text-added: timestamp-independent `state_key` (semantic result sans LastUpdated) keys slots; recency, not per-value history, is the replay defense; rolling slot replaced by a fixed `saturation_epoch` never extended by overflow values | -| P1 must-understand weakens no-store | text-added: mutation-introduced `must-understand` rejected (RFC 9111 §5.2.2.3); snapshot-present preserved | -| P1 Pragma fork / fail-open | text-added: drop-individually list for known-harmless standard fields (Pragma enumerated) vs batch-invalidating unknowns; the documented vendor response is a verbatim fixture; allowlist file corrected and retitled; consequence in sign-off 28 | -| P1 nondeterministic identifiers | text-added: record-kind byte assigned (class-tag ASCII), `tswsx1 | `16-byte truncated suffix hashes, **computed known-answer vectors embedded** for family ID and suffix hash; stale`fam:v0` example replaced | -| P2 batch (registration version bump contract + invariant-revision bump; head-only artifacts never satisfy GET, validator-mismatch invalidates; `w` retention ≥ max(cookie, row/S2S horizon) + overflow residual added to sign-off 30; Fastly listing cell requires a cited platform bound; suffix 123 everywhere + boundary fixtures; Expires via RFC 9111 §4.2.1 with conservative invalid-date handling; pointer-list tokenization grammar) | text-added | -| P3 batch (ledger R15 pipes/falsity corrected; wire-schema grammar; providers stale §4.2 pointer; allowlist title; migration tombstone fragment repaired) | text-added | +## Rounds 15–17 — note on table hygiene and closure honesty + +Earlier round tables here used raw `|` inside cells, which Markdown reads +as column breaks — the R15/R16 tables rendered malformed, and several +rows over-claimed closure (a phrase existing in the text is not proof the +cross-document contract is coherent, exactly the failure mode the R13 +"text-added vs verified-closed" vocabulary was introduced for and which +R15/16/17 each then demonstrated again). These three rounds are recorded +as prose to avoid both problems; the authoritative per-finding status is +the spec text itself, greppable by the anchor phrases below. + +**Round 15 (f3eacf59):** rollback-safe rowless proof (flag suspension), +observed-row admission, `w` capability/runtime rows, replay saturation +state machine, GPP 24–27 reserved (first pass), X-Set-Cookie typed, +Content-Type-only representation, cache-merge origin-directive +preservation. All text-added; several reopened in R16/R17. + +**Round 16 (9a8596ee):** dual destructive/non-destructive admission +sequences, fleet-linearizable suspension (first pass — reopened R17), +timestamp-independent `state_key`, fixed saturation epoch, deterministic +family/suffix vectors, Pragma drop-list (allowlist file not yet aligned — +fixed R17), reserved GPP table (snapshot not yet split — fixed R17). The +R15/R16 ledger's own "Pragma removed from allowlist file" claim was +**false at the time** and is corrected here. + +**Round 17 (this commit) — second-order interactions the prior repairs +left open:** + +- N+1 one contract: stubs and negative suppression writes permitted, + positive commits/clears forbidden until N+2 — first-upgrade withdrawal + now executes (was unexecutable under "neither creates nor clears"). +- Suspension is crash/concurrency safe: a fleet-stable `not_before` = + suspend + L (L = 120 s, stored) every N+1 observer honors, cyclic + transitions, globally-strong reads as a capability cell (not just + write-once/CAS). +- One `w` saturation outcome: any real row surfacing under a saturated + prefix is denied-then-revoked (resurrection closed), and `w` + consultation keys on the record `valid_until`, not the flag. +- Restrictive replay overflow uses an epoch-pinned timestamp — replay + can no longer renew denial indefinitely; sign-off 31 text updated. +- Two-record use requires `row.revision == summary.revision` (fails + closed both ways) and the strong summary now carries jurisdiction, so + a stale row's jurisdiction can never pair with a newer summary. +- Cache invariant folds in request-side `Authorization` (RFC 9111 §3.5): + no integration-introduced `public`/`must-revalidate`/`s-maxage` on an + authenticated response the origin did not mark shared-cacheable. +- 304 split: origin-revalidation 304 updates the stored base first (RFC + 9111 §4.3.4); local conditional hit re-emits. +- sessionByHeader unsupported by default (never requested); its observer + is an explicit sign-off-23 opt-in. +- Every documented DataDome pointer has one assigned outcome (table in + the allowlist file); the vendor example is a Respond-asserting fixture; + every Respond ends `private, no-store` with CDN fields stripped. +- GPP 24–27 fully reserved: removed from the accepted-version table into + a separate reserved table; presence-vs-unknown defined; supported + sections pin to an immutable registry commit with vendored vectors. +- P2/P3: HEAD-only distinct artifact type with validator/Content-Length + update rules; Vary→cache-key ordering; canonical `state_key`/evidence + digest construction (`tsstk1|`/`tsevd1|`) with a vector; non-HMAC + family vector; portable `[A-Za-z0-9._~-]` suffix alphabet; domain- + separated adapter-independent revision hashes (`tspol1|`/`tsreg1|`/ + `tscfg1|`); must-understand clause scoped to the six sticky directives; + Respond transport bounds; sign-off rows 9/10/31 corrected to open; + the `and the and a` fragment, the providers/hook §4.2 pointers, and the + malformed ledger tables fixed. + +Vectors embedded in-spec (recomputable): +family HMAC `tsfam1|i|hmac|{64×a}.AbC123` → +`e90616c381f64965b8326f17108c3c481cee932b2d7f8af783c7bdc2e21591ef`; +family non-HMAC `tsfam1|i|vend|abcdef` → +`278e67d721babaee94690cd246ee567d6ce709c43f8737c2e9dce1e1119c6be1`; +suffix `tswsx1|AbC123` → `08cb55acf42929772862e82b0960c134`; +state_key `tsstk1|tcf|p1=grant,p4=refuse` → +`a49148a0e3b486fd93a141404857868871319a7e1f2ef85b5499aed80c7e59df`. From 1a15575a4841acbfba6283f64a9ea7b8cdd22241 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 4 Aug 2026 12:55:17 -0700 Subject: [PATCH 19/24] Address eighteenth review: one N+1 test contract, total graphless state table, and single-source S2S authority P1 fixes: - The N+1 rollback tests mirror the one contract exactly: stub creation and negative suppression entries are tested read AND write (a rolled-back N+1 receiving SharingOptOut persists the suppression), positive-authority commits and clears are asserted forbidden - the required-and-forbidden contradiction between contract and tests is gone. - The graphless migration has a total state table (semantics x flag x strong record x row read x w -> one outcome each, with an indeterminate default): the v1 exception is declared, not hidden (v1 semantics use recognized cookies row-lessly until the new model activates - the pre-epic posture persisting is matrix row 14, not an undeclared outage); strong-record-present + successful stale not-found is visibility lag -> indeterminate (a stale replica must not fork a just-minted identity); and the 'while the flag is active' w-consultation leftover is replaced by valid_until keying. - The 120 s suspension barrier is skew-safe: not_before comes from store-issued time where available, else committer time + L + S (S = the normative 300 s bound), and both not_before and L are serialized fields of the metadata value with clock-skew and suspender-restart tests named. - S2S authority has one source: the strong summary is the sole decision input (jurisdiction included); the permission spec's authority paragraph now recomputes from the summary alone, and the row-schema provenance row is demoted to 'audit mirror only' with identity/ partner data released only behind the revision fence. - The saturation restrictive marker pins to the FIRST restrictive overflow's own timestamp with a full consent TTL - epoch-entry pinning back-dated genuinely new opt-outs and expired them early; later overflows inherit the marker (bounded shortening), and sign-off 31 is rewritten to ratify exactly that plus fresh-consent denial. - Revision identity is one pair everywhere - (tspol1| content digest, deployment-metadata activation ordinal): digest for identity (A->B->A yields A's digest at a new ordinal), ordinal for fleet-wide order CAS-incremented in the metadata primitive; the hook's push-version and the wire schema's digest-plus-generation variants are superseded; section 5.5 was rebuilt in the process (also repairing a paragraph a scripted edit had corrupted). - Cache stickiness is form-preserving: an unqualified private/no-cache never becomes field-qualified and qualified field sets never shrink (private="Set-Cookie" authorizes shared storage of everything but one field); the contradictory must-understand tail is removed. - Overlay cache identity: matching uses the exact final publisher request (redacted-view keying collapses personalized variants), sensitive Vary-nominated values persist only as keyed digests, and responses derived from identity-bearing TS overlays force private, no-store. - sessionByHeader is startup-rejected in v1 - one state: never requested, X-Set-Cookie unclassified, incoming header ClientIDs not forwarded to the vendor; the R16 translation into an ordinary cookie is retracted as not equivalent (header-session JS expects X-Set-Cookie/X-DD-B and outranks cookies); the old DataDome spec's banner supersedes its header-mode requirement; full header-mode support is the enumerated sign-off-23 opt-in. - One pointer contract in one place: the decision x session-mode x pointer matrix lives in datadome-header-allowlist.md (hook duplicates deleted), X-DD-B is enumerated (drop-individually in cookie mode), no wildcard remains, and both documented vendor responses (challenge and Set-Cookie X-DD-B allow) are decision-asserting fixtures. - The browser trust boundary enters the sign-offs: the cookie cannot be HttpOnly per vendor guidance so every same-origin script observes it, Respond serves vendor HTML with publisher-origin access, CSP cuts both ways - sign-offs 23/28 rewritten to ratify these observers, vendor code, CSP behavior, and challenge redirects. - HEAD/304 preserve transformed-artifact integrity: origin-side and processed-side metadata are stored separately, updates touching byte-coupled representation fields (Content-Encoding, Content-Type, validators, digests) invalidate and refetch a full 200 per RFC 9111 3.2, and a changed Vary evicts or rekeys. P2/P3: w entries carry per-entry horizons (a single record lifetime either shortchanged late entries or rolled forever); the replay hash prose is aligned to its own vector - ASCII source token, not enum bytes, slots per source - with the evidence-digest vector embedded (tsevd1|...|lu=1690000000000 -> 67259c02...); the schema-floor value is encoded (writer version + minimum-reader, numeric order) making N+1-after-floor decidable; the DataDome complete-response deadline is 3000 ms monotonic with 1500 ms first-byte retained and encoded responses batch-invalid; the cookie parser is total (repeated Cookie joined, Set-Cookie never combined, strict attribute/Expires rejection, 512-byte serialized measure); adapter header ceilings are enumerated capability cells validated against core's budget at startup; the ledger's GPP/PSL 'pinned' claim is corrected to placeholder-until-ratification; the section 5.5 grammar dangler, the remaining 'and the and a' fragment, and the duplicated Set-Cookie-reserved sentence are fixed. --- ...-datadome-server-side-protection-design.md | 6 +- ...integration-response-header-hook-design.md | 141 +++++++++++------- .../2026-07-30-permission-model-design.md | 77 ++++++---- .../2026-07-30-pluggable-providers-design.md | 109 +++++++++----- ...07-30-provider-migration-rollout-design.md | 82 +++++----- .../specs/datadome-header-allowlist.md | 44 ++++-- docs/superpowers/specs/pr986-review-ledger.md | 68 ++++++++- 7 files changed, 356 insertions(+), 171 deletions(-) diff --git a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md index 5d59e2ef6..c1b33d460 100644 --- a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md +++ b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md @@ -9,7 +9,11 @@ > effects → final cache/privacy invariant pass, unconditionally last), > with typed cookie/header operations, enumerated allowlists > (`datadome-header-allowlist.md`), and owner-only identifier -> boundaries. Where this document conflicts, the hook spec governs. +> boundaries. Where this document conflicts, the hook spec governs. Additionally, this document's sessionByHeader requirement ("always +> send `X-DataDome-X-Set-Cookie` when the header ID is used") is +> **superseded for v1**: header-session mode is startup-rejected (hook +> spec §4a); TS never requests it and does not forward incoming header +> ClientIDs to the vendor. **Issue:** #317 **Date:** 2026-06-11 diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 0af57dfc2..46afe5a6e 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -70,13 +70,20 @@ mutators to the outbound response for HTML document responses it processed. personalized response shared-storable. The merge: each of the **six sticky directives** — `no-store`, `no-cache`, `private`, `must-revalidate`, `proxy-revalidate`, `no-transform` — is present-in-snapshot-or-mutation ⇒ present-in-final (this - "snapshot or mutation ⇒ final" rule scopes to exactly these six) — and `must-understand` is the deliberate + "snapshot or mutation ⇒ final" rule scopes to exactly these six) — with two refinements. First, `must-understand` is the deliberate exception: **mutation-introduced `must-understand` is rejected** (snapshot-present survives untouched), because under RFC 9111 §5.2.2.3 a cache that understands the status may then ignore an - accompanying `no-store` — "adding" it can _weaken_ a stored `no-store` - response, so it is not an additive restriction at all — present in the snapshot or the - mutation ⇒ present in the final response, independently; `public` is + accompanying `no-store` — "adding" it weakens a stored `no-store`, so + it is not additive. Second, stickiness is **form-preserving, not + name-presence-only**: an **unqualified directive never becomes + field-qualified, and a qualified field set never shrinks** — + `private` → `private="Set-Cookie"` keeps the directive name while + authorizing shared storage of everything but one field (RFC 9111: + qualified `private`/`no-cache` have materially weaker semantics), so + a mutation supplying a qualified form where the snapshot is + unqualified keeps the snapshot's bare form, and qualified snapshot + sets may only grow; `public` is dropped whenever any restriction is present; **request-side authority is part of the invariant** — if the request carried `Authorization` and the origin did not itself authorize shared reuse (no `public`, @@ -100,10 +107,17 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may origin-supplied members included, not only core-required ones — and the mutation. Ordering is normative so the final `Vary` reaches TS's own cache key, not just the wire: **mutation/invariant → final `Vary` - computation → cache-key construction → body/metadata commit**, and the - nominated request values are stored with the artifact (emitting `Vary` - downstream is useless if the body was already keyed less specifically - internally). Parsing itself is a **shared core parser with + computation → cache-key construction → body/metadata commit** — with + three identity rules. Cache matching uses the **exact final publisher + request** (post-overlay view; keying from the redacted view would + collapse personalized variants). Nominated request values are stored + **only as keyed digests** when sensitive (cookies, identity overlays, + bearer tokens named by `Vary` must never be persisted literally). And + a response derived from a request carrying an **identity-bearing TS + overlay** (the DataDome ClientID overlay) is forced `private, + no-store` unless an explicit per-overlay contract says otherwise — + the `Authorization` rule protects origin credentials, and this rule + protects the identity TS itself injected. Parsing itself is a **shared core parser with fail-closed normalization**, not four adapter interpretations: invalid `Cache-Control` syntax normalizes to the most restrictive reading; duplicate directives keep the strongest; quoted and unquoted @@ -198,7 +212,7 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may merge), `Content-Language` (append), `X-Robots-Tag` (append), `Retry-After` (replace-only), `Content-Location` (replace-only); everything else known is classified reserved or rejected by the rules - above, and growing the admitted set is a spec change to this list (`Set-Cookie` is fully reserved in v1 — neither append nor replace). Replacing a + above, and growing the admitted set is a spec change to this list (cookies: see the §3 deferral above). Replacing a header the origin set is a deliberate act, visible in the mutator's code. - Later registrations see earlier mutations (order = registration order, which is deterministic). @@ -210,7 +224,11 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may operations (≤ 32), added headers (≤ 16), and added bytes (≤ 8 KiB), and a **cumulative final-response budget** (≤ 128 headers / ≤ 32 KiB total, counting `name: value` plus separators, within any lower adapter - ceiling) bounds the sum across integrations — enforced in registration + ceiling — **and those ceilings are enumerated capability cells per + adapter, with the counting rule fixed as serialized `name: value` + bytes plus separators**, validated against core's budget at startup + so a batch that passes core can never fail only on one adapter) + bounds the sum across integrations — enforced in registration order, so which operations are rejected when a budget trips is deterministic. Each mutator receives an **immutable, redacted snapshot of the response head** (status and headers as of its turn, prior integrations' @@ -249,17 +267,17 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by a fleet-stable tuple** — the integration-registry revision is a content hash over the ordered (integration ID, version) list; the config revision is the config store's globally assigned push version; the invariant revision is a build-time spec-version constant — local counters would collide across instances and binaries; there are **two distinct 304 cases**. A **locally generated conditional hit** (TS answers the client's `If-*` from its own fresh stored artifact) re-emits the persisted finals when all three revisions match, else cache-miss — as before. An **origin-revalidation 304** (TS revalidated upstream and the origin returned 304 with possibly new `Cache-Control`/`Vary`/`Expires`/validators) is different: RFC 9111 §4.3.4 requires the stored response to be **updated from the current 304 before serving**, so TS updates the stored base with the 304's metadata first; if any admitted/cache-relevant field changed, it **reruns the relevant processing or refetches a full 200** rather than re-emitting stale finals (a cached `public` followed by an origin `304 Cache-Control: private, no-store` must not keep serving the old public policy). Artifact absence or a revision mismatch strips internal preconditions before obtaining that full response. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | -| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals stored as a **distinct head-only artifact type that never satisfies a later GET** — when a stored GET artifact exists a HEAD may **update** it only if validators and `Content-Length` match (RFC 9111 §4.3.5), a mismatch **invalidating** the stored GET artifact rather than updating it | -| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | +| Response | Hook runs? | +| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by a fleet-stable tuple** — the integration-registry revision is a content hash over the ordered (integration ID, version) list; the config revision is the config store's globally assigned push version; the invariant revision is a build-time spec-version constant — local counters would collide across instances and binaries; there are **two distinct 304 cases**. A **locally generated conditional hit** (TS answers the client's `If-*` from its own fresh stored artifact) re-emits the persisted finals when all three revisions match, else cache-miss — as before. An **origin-revalidation 304** (TS revalidated upstream and the origin returned 304 with possibly new `Cache-Control`/`Vary`/`Expires`/validators) is different: RFC 9111 §4.3.4 requires the stored response to be **updated from the current 304 before serving**, so TS updates the stored base with the 304's metadata first; if any admitted/cache-relevant field changed, it **reruns the relevant processing or refetches a full 200** rather than re-emitting stale finals (a cached `public` followed by an origin `304 Cache-Control: private, no-store` must not keep serving the old public policy). Artifact absence or a revision mismatch strips internal preconditions before obtaining that full response. **Origin-side and processed-side metadata are stored separately** — the origin's validators/`Content-Length` describe origin bytes, not the rewritten HTML artifact — and any update touching **byte-coupled representation fields** (`Content-Encoding`, `Content-Type`, validators, digests) **invalidates and refetches a full 200 rather than updating** (RFC 9111 §3.2 excludes `Content-Length` from stored-response updates and warns against updating transformed artifacts with incompatible representation metadata); a changed `Vary` evicts or rekeys the stored index entry. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | +| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals stored as a **distinct head-only artifact type that never satisfies a later GET** — when a stored GET artifact exists a HEAD may **update** it only if validators and `Content-Length` match (RFC 9111 §4.3.5), a mismatch **invalidating** the stored GET artifact rather than updating it | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). @@ -286,7 +304,13 @@ degree of freedom is closed: are separate requirements) and a vendor cookie using `Expires` is normalized to its Max-Age equivalent (both present → `Max-Age` wins, per RFC 6265); a normalized lifetime exceeding the ceiling rejects - the whole operation batch; + the whole operation batch; and the parser is total: repeated `Cookie` + request fields are tolerated and joined (per the current cookie RFC), + `Set-Cookie` fields are **never combined**, duplicate attributes, + duplicate cookies in one field, an unparseable `Expires`, or unknown + attributes each reject the operation (the vendor cookie is + well-formed; strictness is safe), and the 512-byte limit measures the + serialized `name=value` plus attributes in bytes; `Max-Age` at most **31,536,000 seconds** (the vendor's one-year cap — the earlier 396-day figure exceeded it); size ≤ **512 bytes** (DataDome's current Fastly-module limit; 4 KiB was ours, not theirs). @@ -295,19 +319,30 @@ degree of freedom is closed: DataDome's mandatory response-directed mapping set — that reduction needs explicit product **and vendor** acceptance: sign-off item 28; a violating operation is rejected whole (the batch rule). **Both - sessionByHeader is unsupported by default and never requested** — TS - does not send `X-DataDome-X-Set-Cookie: true`, so the vendor uses - ordinary `Set-Cookie`, which lowers into the typed `datadome` - operation. Header-session mode exists for clients that cannot use a - cookie and expects JavaScript to receive `X-Set-Cookie` and `X-DD-B` - into local storage; supporting it means forwarding those as typed, - owner-scoped headers **and** accepting a JavaScript/local-storage - identifier observer — an explicit opt-in under **sign-off 23** (which - now enumerates that observer), not a silent cookie substitution. - Without the opt-in, an incoming `X-Set-Cookie` is unclassified → - Continue. Every `ts-*` - name is rejected. **Read is owner-only, and the strip inventory is exhaustive, not - integration-scoped** — the browser sends `datadome` in the ordinary + sessionByHeader is startup-rejected in v1 — one state, not three**: + TS never sends `X-DataDome-X-Set-Cookie: true`; a vendor + `X-Set-Cookie` is unclassified (→ batch handling); and an incoming + browser `X-DataDome-ClientID` is **not forwarded to the vendor** + (cookie-only session identity — an earlier revision translated + `X-Set-Cookie` into an ordinary cookie, which is not equivalent: + header-session clients expect JavaScript to receive `X-Set-Cookie` + and `X-DD-B`, and the higher-priority header session would never see + a cookie update; the older DataDome spec's "always send + X-DataDome-X-Set-Cookie when the header ID is used" is superseded for + v1 by its banner). Supporting header mode later means the full vendor + protocol — typed owner-scoped `X-Set-Cookie`/`X-DD-B` forwarding, + CORS exposure, and a JavaScript/local-storage identifier observer — + as an explicit opt-in under **sign-off 23**. Every `ts-*` + name is rejected. **Read is owner-only across every _server-side_ surface — and the + browser side is explicitly not ownable**: DataDome requires the + cookie to be readable by its JavaScript and warns against `HttpOnly`, + so **every same-origin page script can observe it**, and a Respond + serves vendor-owned HTML under the publisher origin (vendor scripts + with same-origin access to cookies, storage, and APIs; publisher CSP + can conversely break the challenge). Those browser-side observers, + same-origin vendor code, CSP interaction, and challenge redirects are + ratified in **sign-offs 23/28**, not implied. The server-side strip + inventory is exhaustive, not integration-scoped — the browser sends `datadome` in the ordinary `Cookie` header, so it is removed from **every non-DataDome surface**: other integrations' request views, publisher-origin proxy forwarding, proxy/click/Testlight upstreams, auction/page-bids request @@ -390,23 +425,27 @@ no-cache` has no standardized meaning, RFC 9111 §5.4) are **dropped allowlist-file addition. A _Continue_ decision may not touch representation metadata of publisher bytes. - **Respond transport is bounded.** The challenge body has a maximum - size (64 KiB) and a complete-response deadline; TS sends - `Accept-Encoding: identity` on challenge fetches so there is no - encoded body to reframe (`Content-Encoding` is reserved anyway), and + size (64 KiB) and a **complete-response deadline of 3000 ms on the + instance's monotonic clock** (the older spec's 1500 ms is first-byte + only and stays as the first-byte bound); TS sends `Accept-Encoding: +identity`, and because that does not _guarantee_ identity coding, a + response arriving with any `Content-Encoding` is itself batch-invalid; `Content-Length` is recomputed from the actual bytes before Respond - commits. Exceeding the size or deadline fails the batch → Continue. -- **Every documented DataDome pointer has exactly one assigned - outcome, and the vendor's documented response is a passing fixture.** - The outcome table (also in `datadome-header-allowlist.md`): - `Set-Cookie` / `X-Set-Cookie` → typed `datadome` cookie operation; - `Location`, `Content-Type` → forward (Respond only); `Cache-Control` → - restricted merge; `Pragma` → drop-individually (logged); `X-DataDome`, - `X-DD-*` → **forward as owner-scoped typed headers** (they are DataDome - telemetry, not publisher policy); anything unclassified → invalidate - (→ Continue). The fixture asserts DataDome's documented example - (`Set-Cookie`, `Pragma`, `X-DataDome`, `Cache-Control`) stays - **Respond** and emits exactly the mapped fields — none of it drops to - Continue. + commits. Exceeding size, first-byte, or total deadline fails the + batch → Continue. +- **One pointer contract, one place.** The single normative + decision × session-mode × pointer matrix lives in + **`datadome-header-allowlist.md`** — this spec's earlier inline + decision-scoped list and outcome list are deleted in its favor + (duplicated lists disagreed about `X-Set-Cookie`, `X-DataDome`, + `X-DD-*`, and `Pragma`, letting one conforming implementation accept + the vendor's documented `Set-Cookie X-DD-B` allow-example while + another invalidated the whole batch). No `X-DD-*` wildcard exists: + every name is enumerated, `X-DD-B` included (drop-individually in + cookie mode — dropping it does not break cookie sessions). The + documented vendor responses (both the challenge example and the + `Set-Cookie X-DD-B` allow example) are **verbatim fixtures asserting + the decision survives** and exactly the mapped fields emit. - **Every security Respond ends uncacheable, unconditionally.** After the decision's fields are applied, the invariant pass forces `Cache-Control: private, no-store` and strips all CDN cache fields on diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 6744912e7..1a6422376 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -835,34 +835,45 @@ migration story unresolvable (migration spec §2, rows 5 and 7). ### 5.5 Policy revision activation -A **policy revision** has a defined identity: the canonical content -digest of the `[permissions]` section — **defined**: SHA-256 with domain -tag `tspol1|` over the canonical JSON of the parsed policy (keys sorted -lexicographically by UTF-8 code unit, numbers shortest round-trip, -defaults materialized, no insignificant whitespace), cross-language -vectors required. **The other cache-tuple inputs are domain-separated -hashes of effective configuration, adapter-independent**: +A **policy revision** has one identity used everywhere: the pair +**(content digest, activation ordinal)**. The digest is SHA-256 with +domain tag `tspol1|` over the canonical JSON of the parsed policy (keys +sorted lexicographically by UTF-8 code unit, numbers shortest +round-trip, defaults materialized, no insignificant whitespace; +cross-language vectors required) — identity, so an A→B→A rollback +yields A's digest again. The ordinal is a **globally ordered activation +counter, CAS-incremented in the deployment-metadata primitive on each +config activation** — order, adapter-independent (not every adapter has +a native push version, and per-instance counters ordered nothing across +a fleet). Authority wire records, the S2S recompute, and the hook cache +tuple all use this same pair; the hook's earlier "config-store push +version" and any digest-only usage are superseded. The other cache-tuple +inputs are likewise domain-separated hashes of effective configuration: integration-registry revision = `tsreg1|` over the canonical-JSON `(id, version)` list; config revision = `tscfg1|` over the effective -config blob — so adapters without a native push version still derive -identical revisions from identical configuration by -§7); the mixing window is bounded by config propagation and observable via -the config-version metric; and mixed-revision irreversibility is bounded and **accepted, not -denied** (sign-off 19): destructive withdrawal triggers are user -signals, never policy (§4.2 trigger 3) — the one revision-sensitive destructive case -(trigger 2 under a now-`denied` baseline) requires an affirmative user -refusal at the evaluating instance, which is safe under either revision. -S2S recomputation always evaluates against the instance's current -revision and records it. One divergence is explicitly accepted rather -than fenced: during convergence, a live refusal under a -`granted`-revision instance suppresses while the same refusal under a -tightened-revision instance destroys (trigger 2) — the destructive -outcome is the target revision's intended behavior arriving early on -part of the fleet, coordinated activation fencing is not worth its -machinery, and the acceptance is sign-off item 19. Rolling a policy -revision back restores acquisition rules but **cannot resurrect -tombstoned identities**; the migration guide says so where operators -will read it. +config blob — so adapters derive identical revisions from identical +configuration. + +A policy edit propagates through the config store, so a fleet briefly +mixes revisions. The contract: instances stamp every resolution and +every provenance write with the (digest, ordinal) they used (already +required by §7); the mixing window is bounded by config propagation and +observable via the activation-ordinal metric; and mixed-revision +irreversibility is bounded and **accepted, not denied** (sign-off 19): +destructive withdrawal triggers are user signals, never policy (§4.2 +trigger 3) — the one revision-sensitive destructive case (trigger 2 +under a now-`denied` baseline) requires an affirmative user refusal at +the evaluating instance, which is safe under either revision. S2S +recomputation always evaluates against the instance's current revision +and records it. One divergence is explicitly accepted rather than +fenced: during convergence, a live refusal under a `granted`-revision +instance suppresses while the same refusal under a tightened-revision +instance destroys (trigger 2) — the destructive outcome is the target +revision's intended behavior arriving early on part of the fleet, +coordinated activation fencing is not worth its machinery, and the +acceptance is sign-off item 19. Rolling a policy revision back restores +acquisition rules but **cannot resurrect tombstoned identities**; the +migration guide says so where operators will read it. ## 6. Failure-mode matrix — normative @@ -935,8 +946,12 @@ Consumers of the resolved set in this epic: **S2S authority (batch sync).** A context-free server-to-server request carries no user signals, geo, or `EcContext`. Its authority is the - identity's **stored provenance**: per-permission, time-bounded evidence - written at mint and replaced on later live requests — grant basis + **strong authority-state summary — the sole decision input** (the + identity row's provenance fields are an audit mirror; reading + decision inputs from an eventually consistent row was the two-source + bug the revision fence exists to prevent): per-permission, + time-bounded evidence written at mint and replaced on later live + requests — grant basis (which signal class granted, per permission), the evidence's **authoritative timestamp and `valid_until`** (per evidence class), resolved jurisdiction, and policy revision — **not** provider/version, @@ -972,8 +987,10 @@ Consumers of the resolved set in this epic: resolution **atomically replaces the complete per-permission snapshot**, never merges — a refusal, opt-out, malformed or absent state in the fresh resolution clears prior positive authority for its - scope, so an old P4 grant cannot survive a later P4 refusal. A sync request performs a **full recompute of both - permissions** from that stored evidence against the _current_ policy: + scope, so an old P4 grant cannot survive a later P4 refusal. A sync request performs a **full recompute of both permissions from + the strong summary alone** against the _current_ policy — the row + contributes identity and partner data only after the + `row.provenance_revision == summary_revision` fence passes: it fails closed when the stored jurisdiction's rule is now `denied`, when a `granted` baseline tightened to `requires_signal` and the stored evidence contains no accepted grant for that permission, when the diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index cee6c23ed..d8f9c4a98 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -318,9 +318,14 @@ variant**. Therefore: observable strong reads _in addition to_ CAS — a capability cell, since CAS alone says nothing about what other instances currently see). The suspension transition stamps a **fleet-stable `not_before` - deadline** = commit time + L into the suspended epoch, where **L is an - assigned constant (120 s) stored in the metadata**, not a per-instance - timer: **every** N+1 instance — the one that suspended, a second that + deadline** into the suspended epoch — computed from **store-issued + time where the backend provides one, else committer time + L + S** + (S = the normative 300 s skew bound; without folding S in, a + slow-clocked suspender could write a deadline already passed on + another instance while an N+2 lease survived) — where **L is an + assigned constant (120 s)**; **both `not_before` and L are serialized + fields of the metadata value** (the schema names them), and + clock-skew and suspender-restart schedules are named tests: **every** N+1 instance — the one that suspended, a second that starts and reads an already-suspended state, or one recovering after the suspender crashed — refuses to mint until `now ≥ not_before` (globally strong read of the epoch). N+2 instances prove `active` at @@ -347,11 +352,29 @@ variant**. Therefore: every row-backed identity has an authority-state record under its derivable family ID, in the globally-strong class — so _rowless_ = flag set AND the strong read finds **no record** for the cookie's - derived family ID — and, while the flag is active, **every HMAC row - discovery consults the prefix's `w` state before any live or S2S - use** (a pending or saturated entry means promotion-then-denial per + derived family ID — and **every HMAC row discovery consults the prefix's `w` state + before any live or S2S use whenever a live `w` record exists** + (keyed on the record's `valid_until`, never on the flag — an earlier + "while the flag is active" scope contradicted the valid_until rule + one sentence later) (a pending or saturated entry means promotion-then-denial per the runtime matrix, §6.2), so a withdrawn suffix cannot slip into use - through the row path. **`w` consultation is keyed on the record's `valid_until`, not the rowless-classification flag** — the flag may clear after one cookie lifetime while `w` is retained through the longer max(cookie, row, S2S) horizon, and a late row must still find a live `w`; enforcement ends only when the `w` record itself expires. Graphless-era cookies never got a stub because + through the row path. **`w` consultation is keyed on the record's `valid_until`, not the rowless-classification flag** — the flag may clear after one cookie lifetime while `w` is retained through the longer max(cookie, row, S2S) horizon, and a late row must still find a live `w`; enforcement ends only when the `w` record itself expires. + + **The total state table** — every (semantics, flag, strong record, + row read, `w`) combination has exactly one outcome; anything not + listed falls to the bolded default: + + | Semantics | Flag | Strong record | Row read | Live `w` entry | Outcome | + | ----------------------------------- | -------------- | ------------- | ------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- | + | v1 (N+1, or N+2 + old-shape config) | any | any | any | any | **v1 behavior: recognized cookies are used per pre-epic rules, rows not required** — the declared v1 exception, not an outage: the pre-epic privacy posture persists until the new model activates (matrix row 14) | + | new model | absent/cleared | present | found, revisions match | none | normal use | + | new model | absent/cleared | present | **not-found (successful read)** | none | **visibility lag, not absence**: the strong record proves the row committed, so the identity is indeterminate this request — no use, no mint, no expiry (a stale replica must not fork a just-minted identity) | + | new model | absent/cleared | absent | any | none | indeterminate (no rowless classification without the flag) | + | new model | active | absent | authoritative not-found | none | rowless: expire-and-re-mint / withdrawal per §5 | + | new model | suspended | absent | any | none | indeterminate (classification off; re-attestation pending) | + | new model | any | any | found | **matching entry or saturated** | denied, then promoted to family revocation (§6.2) | + | new model | any | any | error / `w` read error | — | **default: indeterminate — no use, no mint, no expiry, no negative writes** | Graphless-era cookies never got a stub because | + they have no row for the scan to find; no eventual read participates. The flag itself is specified: a named deployment-metadata key (write-once/CAS class), set by the §4.2 migration runbook step (migration spec §4) only on deployments that actually ran graphless @@ -362,6 +385,7 @@ variant**. Therefore: or on any read error, the state is **indeterminate**: no identity use, no mint, no cookie expiry — "treated as absent" was the wrong contract, since absence feeds the fresh-mint path. + - A verified rowless cookie (`verify → VerifiedIdentity`, carrying the matched version) is **expired and replaced by a fresh mint through the ordinary graph-backed path** when permissions allow; continuity is @@ -638,17 +662,23 @@ solves. **Physical key grammar.** Core constructs every key; providers supply only the bounded suffix: -| Record class | Key | Notes | -| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ **123** bytes (128-byte total key cap minus tag and provider code — an earlier 128 here contradicted the constructor; 123/124 boundary fixtures required), portable alphabet **`[A-Za-z0-9._~-]`** (not "KV-safe per host", which would break the single cross-adapter grammar; providers with native identifiers outside it emit base64url suffixes); ≤ **123** bytes with 123/124 boundary fixtures. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | -| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | -| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | -| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | -| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | -| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8), saturation flag, CAS version, created-at, `valid_until` ≥ **max(cookie lifetime, row/S2S authority horizon)** — a `w` expiring before the row horizon would let an overflow row's identity resurface (the overflow non-promotion residual is sign-off 30); readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | -| Deployment metadata | `m` + fixed metadata name (fixed-width grammar; schema floor, graphless-migration flag) | Write-once/CAS class; value carries schema version, state, epoch, set-at; the graphless flag's lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded | -| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | -| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | +| Record class | Key | Notes | +| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ **123** bytes (128-byte total key cap minus tag and provider code — an earlier 128 here contradicted the constructor; 123/124 boundary fixtures required), portable alphabet **`[A-Za-z0-9._~-]`** (not "KV-safe per host", which would break the single cross-adapter grammar; providers with native identifiers outside it emit base64url suffixes); ≤ **123** bytes with 123/124 boundary fixtures. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | +| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | +| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | +| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | +| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | +| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8) where **each entry carries its own `valid_until`** = its withdrawal time + max(cookie lifetime, row/S2S authority horizon) — one record-level lifetime either shortchanged late entries or, rolling, let an attacker keep a saturated NAT cohort withdrawn forever; the record expires when its last entry (or the saturation flag's own pinned horizon) expires; saturation flag with its own entry-time-pinned horizon; CAS version; created-at — a `w` expiring before the row horizon would let an overflow row's identity resurface (the overflow non-promotion residual is sign-off 30); readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | +| Deployment metadata | `m` + fixed metadata name (fixed-width grammar; schema floor, graphless-migration flag — the | + +**floor value is encoded**: integer writer-activation schema version + +minimum-reader version, ordered numerically; a binary starts only if its +declared reader capability ≥ the floor's minimum-reader, which is what +makes "is N+1 permitted after N+2 activates" decidable: N+1 declares +N+2-reader capability, so yes) | Write-once/CAS class; value carries schema version, state, epoch, set-at, and — for the graphless flag — **`not_before` and `L`** (serialized, so every observer reads the same deadline) plus the attestation; the graphless flag's lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded | +| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | +| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | Grammars are pairwise non-intersecting by their literal prefixes (plus the reserved hmac grammar), and every record value carries a `kind` @@ -722,17 +752,24 @@ reproduce them**): kind (user evidence vs policy baseline), grant basis/source class, policy revision (digest + activation generation), **resolved jurisdiction** (so S2S never reads it from an eventually stale row — the summary is self-sufficient for the recompute), `valid_until`, -provenance revision, evidence timestamp, and the -and a **bounded replay history** whose slots are keyed by a +provenance revision, evidence timestamp, and a **bounded replay history** whose slots are keyed by a **timestamp-independent `state_key`** — (source class, semantic result digest _excluding_ `LastUpdated`) — distinct from the _evidence digest_ (which for TCF includes `LastUpdated` for recency), both with -**canonical wire construction**: `state_key` = SHA-256 `tsstk1|` over -`source-class-enum-byte | canonical-semantic-result` (enum bytes: tcf=1, -gpp=2, usp=3; timestamps integer epoch-ms where present), evidence -digest = SHA-256 `tsevd1|` over the same plus `LastUpdated`; vector: TCF -(P1 grant, P4 refuse) `tsstk1|tcf|p1=grant,p4=refuse` → -`a49148a0e3b486fd93a141404857868871319a7e1f2ef85b5499aed80c7e59df`: keying slots on the +**canonical wire construction — the hash input is the lowercase ASCII +source token, not an enum byte** (an earlier draft said "enum bytes: +tcf=1…" while its own vector hashed the ASCII token; the vector was +right, the prose wrong — enum bytes exist only in storage, never in hash +input): `state_key` = SHA-256 `tsstk1|` + `` (`tcf` / +`gpp` / `usp`) + `|` + canonical semantic result covering **all enforced +permissions for that source** (slots are **per source**, not per +permission·source — the result string carries every permission, as the +vector shows; timestamps integer epoch-ms where present); evidence +digest = SHA-256 `tsevd1|` over the same plus `|lu=`. +Vectors: `tsstk1|tcf|p1=grant,p4=refuse` → +`a49148a0e3b486fd93a141404857868871319a7e1f2ef85b5499aed80c7e59df`; +`tsevd1|tcf|p1=grant,p4=refuse|lu=1690000000000` → +`67259c0247ae2b33c52d9f18193bcd622f48ad4754ffbab6998d7c293b0143b4`: keying slots on the evidence digest would give every renewal a fresh key and make "updates its slot in place" impossible, the incompatibility an earlier draft shipped. A slot stores its `state_key`, the current evidence @@ -741,19 +778,23 @@ timestamp observed; a TCF renewal (same `state_key`, newer `LastUpdated`) updates the slot in place, while a replay (not newer) changes nothing — replay protection derives from recency comparison, not per-value history, so no per-digest sublists are needed. 16 slots -per permission·source; entries live to the evidence/suppression +per source; entries live to the evidence/suppression horizon. **Saturation is a fixed epoch, not a rolling slot** (one slot cannot hold independent timestamps for multiple overflow digests): when all slots hold distinct in-horizon states, the record sets a `saturation_epoch` with `saturated_until = now + consent TTL`, **fixed at entry and never extended by later overflow values**; while saturated, novel values cannot grant (fail restrictive); a -**restrictive overflow** (timestamp-less opt-out or malformed) does not -mint fresh suppression at the current observation time — it is -recorded, if at all, under an **epoch-scoped restrictive marker whose -timestamp and `valid_until` are pinned to saturation-epoch entry**, so -replaying it later cannot advance its observation time or extend -suppression (the unpinned version let repetition renew denial forever), +**restrictive overflow** (timestamp-less opt-out or malformed) is +recorded under an **epoch-scoped restrictive marker pinned at first +restrictive overflow**: the marker takes that _first_ overflow's own +observation timestamp and a full consent-TTL `valid_until` from it — +replays and later overflow values neither advance nor extend it (the +unpinned version let repetition renew denial forever), and pinning to +the epoch's entry instead would have back-dated a genuinely new opt-out +and expired it early. Later restrictive overflows within the epoch +inherit the marker — a **bounded shortening** (at most the gap between +first and later overflow) declared in sign-off 31, recovery is automatic at epoch expiry as slots free, and saturation is a first-class metric — the cap and its denial behavior are **sign-off item 31**; record level: family ID, @@ -787,7 +828,7 @@ achievable through a structured serializer). | `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | | New: **immutable mint tag** (`mint_provider`, `mint_version`) | Credential retirement and audit — write-once at mint (legacy backfill may populate a missing tag once); **never part of the replaceable snapshot**, or a v1 identity revisited after rotation would be restamped v2 | Mint (or one-time backfill) | — | Immutable | — | Retained | | New: **provenance revision** (application-level monotonic counter, u64, initialized at 1, incremented by every provenance-bearing write, CAS'd with the row generation, serialized as an integer; overflow is a hard error, not a wrap) | Orders clears vs. snapshots (permission spec §4.3) | Core | Read by S2S/clears | Monotonic | — | Retained | -| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision) | S2S authority | Live resolution only | Read by S2S recompute | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | +| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision) | **Audit mirror only — never the S2S decision input**: the strong summary carries every decision field (jurisdiction included); the row supplies identity/partner data only after the exact revision fence | Live resolution only | Read by S2S recompute | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | | New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | | `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | | `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index e77809e86..a1b0aad54 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -165,11 +165,15 @@ Requirements: later than the model it protects. Live-request paths keep v1 semantics until N+2. - Rollback tests therefore run: family-revocation read **and - write**; authority-state/suppression **read-and-fail-closed only** - (N+1 writes none — the earlier read-and-write test requirement - contradicted this contract); and v1-minting behavior — all on N+1 - against N+2-written data. **Rollback is binaries-first too, in the other direction** — + Rollback tests therefore mirror the one contract exactly: + family-revocation read **and write**; authority-state **stub + creation and negative suppression entries, read and write** (the + earlier "read-and-fail-closed only / N+1 writes none" test text + contradicted the required observed-row sequences — a rolled-back + N+1 receiving SharingOptOut must persist the suppression, not deny + once and forget); **positive-authority commits and clears asserted + forbidden**; and v1-minting behavior — all on N+1 against + N+2-written data. **Rollback is binaries-first too, in the other direction** — N+2 → N+1 binaries roll back keeping the new config (N+1 reads it fully; reverting config first would hand the old shape to N+2 binaries that reject it) — **with one structural rule that makes it possible at all**: @@ -485,37 +489,37 @@ the deciders, the date). The Decision-record column holds the link (`—` while open); an unratified row reverts to open, not to silently implemented. -| # | Decision | Where | Decision record | Status | -| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------- | --------------------------------------------------------------------------------------- | -| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | — | open | -| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | — | open | -| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | — | open | -| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | — | open | -| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | — | open | -| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | — | open | -| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | — | open | -| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | — | open | -| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | — | open (descope ratification still required; record-less ⇒ open per the decisions README) | -| 10 | Session-cookie exemption question | hook §3 | — | open (deferred, but record-less ⇒ open per the decisions README) | -| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | — | open | -| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | — | open | -| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | — | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | — | open | -| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | — | open | -| 16 | Timestamp-less opt-out suppression is **TTL-sticky**: within its consent-TTL lifetime only a newer timestamped grant clears it; at `valid_until` it goes inert automatically (not user-sticky-forever, not administratively sticky — administrative clear is an optional early exit) | permission §4.3 | — | open | -| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | — | open | -| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | — | open | -| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | — | open | -| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | — | open | -| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | — | open | -| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | — | open | -| 23 | **Open question, not ratified**: may DataDome's security identifier (tag injection, `datadome` cookie, `X-DataDome-ClientID` read and vendor egress) operate outside the permission model? The decision must enumerate exactly which consumers may observe the cookie/ClientID — the enumerated observers are the security channel **and the publisher origin, which receives `X-DataDome-ClientID` via the upstream overlay** — "owner-scoped overlay" names the mechanism, this row names the recipient — plus retention and whether TS withdrawal expires it | hook §4a; permission §7 | — | open | -| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | — | open | -| 25 | Batch-sync stored-jurisdiction maximum age (consent-TTL horizon): a mover into GDPR stops old-rule egress at the horizon; a mover out is denied until a live visit | permission §7 | — | open | -| 26 | Embedded GPP GPC maps to the destructive global opt-out (header-OR-embedded aggregation) | permission §4.5 | — | open | -| 27 | Proxy-mode minimal opt-out extraction (decode only §4.5-mapped opt-out fields; no grants) | permission §4.4 | — | open | -| 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance | hook §4a; `datadome-header-allowlist.md` | — | open | -| 29 | Unverifiable roaming rowless cookies receive best-effort cookie-only expiry (admission rules forbid durable records for unverified values); a lost response can leave the cookie usable on the old network until re-presented | providers §5 | — | open | -| 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, observability, **and the saturation collateral: under a saturated prefix, any real row (listed or overflow) is denied and revoked immediately on surfacing — a non-abuser NAT-cohort row can be revoked; `w` is retained through the max(cookie, row, S2S) horizon and consulted by `valid_until`, not the flag, so this is deterministic, not a retention accident** — all in scope | providers §5 | — | open | -| 31 | Replay-history capacity (16 semantic-state slots + a fixed saturation epoch with an epoch-pinned restrictive marker): while saturated, novel values cannot grant — fresh consent in unusual multi-CMP setups can be rejected until the epoch expires | permission §4.3; providers wire schema | — | open | -| 32 | GPP sections 24–27 (MD/IN/KY/RI) are reserved with **national-section-only** handling until an official binary layout can be vendored — a state-specific opt-out expressed only in an undecodable state section is not honored | permission §4.5; `gpp-registry-snapshot.md` | — | open | +| # | Decision | Where | Decision record | Status | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------- | --------------------------------------------------------------------------------------- | +| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | — | open | +| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | — | open | +| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | — | open | +| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | — | open | +| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | — | open | +| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | — | open | +| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | — | open | +| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | — | open | +| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | — | open (descope ratification still required; record-less ⇒ open per the decisions README) | +| 10 | Session-cookie exemption question | hook §3 | — | open (deferred, but record-less ⇒ open per the decisions README) | +| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | — | open | +| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | — | open | +| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | — | open | +| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | — | open | +| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | — | open | +| 16 | Timestamp-less opt-out suppression is **TTL-sticky**: within its consent-TTL lifetime only a newer timestamped grant clears it; at `valid_until` it goes inert automatically (not user-sticky-forever, not administratively sticky — administrative clear is an optional early exit) | permission §4.3 | — | open | +| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | — | open | +| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | — | open | +| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | — | open | +| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | — | open | +| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | — | open | +| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | — | open | +| 23 | **Open question, not ratified**: may DataDome's security identifier (tag injection, `datadome` cookie, `X-DataDome-ClientID` read and vendor egress) operate outside the permission model? The decision must enumerate exactly which consumers may observe the cookie/ClientID — the enumerated observers are the security channel **and the publisher origin, which receives `X-DataDome-ClientID` via the upstream overlay** — "owner-scoped overlay" names the mechanism, this row names the recipient — plus retention, whether TS withdrawal expires it, **the browser-side observers the vendor's design implies — every same-origin page script (the cookie must not be HttpOnly per vendor guidance), vendor challenge pages executing with publisher-origin access, and (if header mode is ever opted into) the JavaScript/local-storage observer** — and challenge redirect targets | hook §4a; permission §7 | — | open | +| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | — | open | +| 25 | Batch-sync stored-jurisdiction maximum age (consent-TTL horizon): a mover into GDPR stops old-rule egress at the horizon; a mover out is denied until a live visit | permission §7 | — | open | +| 26 | Embedded GPP GPC maps to the destructive global opt-out (header-OR-embedded aggregation) | permission §4.5 | — | open | +| 27 | Proxy-mode minimal opt-out extraction (decode only §4.5-mapped opt-out fields; no grants) | permission §4.4 | — | open | +| 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance — including CSP interaction with vendor challenge pages, same-origin vendor code on the publisher origin (or an origin-isolation/sandboxing requirement), and the fail-open consequence of batch invalidation | hook §4a; `datadome-header-allowlist.md` | — | open | +| 29 | Unverifiable roaming rowless cookies receive best-effort cookie-only expiry (admission rules forbid durable records for unverified values); a lost response can leave the cookie usable on the old network until re-presented | providers §5 | — | open | +| 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, observability, **and the saturation collateral: under a saturated prefix, any real row (listed or overflow) is denied and revoked immediately on surfacing — a non-abuser NAT-cohort row can be revoked; `w` is retained through the max(cookie, row, S2S) horizon and consulted by `valid_until`, not the flag, so this is deterministic, not a retention accident** — all in scope | providers §5 | — | open | +| 31 | Replay-history capacity (16 per-source semantic-state slots + a saturation epoch whose restrictive marker is pinned at the **first** restrictive overflow with its own full TTL): while saturated, fresh consent cannot grant until the epoch expires, and **later restrictive overflows inherit the first marker — a bounded shortening of their lifetime** | permission §4.3; providers wire schema | — | open | +| 32 | GPP sections 24–27 (MD/IN/KY/RI) are reserved with **national-section-only** handling until an official binary layout can be vendored — a state-specific opt-out expressed only in an undecodable state section is not honored | permission §4.5; `gpp-registry-snapshot.md` | — | open | diff --git a/docs/superpowers/specs/datadome-header-allowlist.md b/docs/superpowers/specs/datadome-header-allowlist.md index 93e846df0..63824bbd2 100644 --- a/docs/superpowers/specs/datadome-header-allowlist.md +++ b/docs/superpowers/specs/datadome-header-allowlist.md @@ -25,18 +25,32 @@ Note: the vendor's `X-Set-Cookie` response field is **not** a forwardable header — it lowers into the typed `datadome` cookie operation (hook spec §4a) and never reaches the browser as a header. -## Pointer outcome table (every documented pointer, exactly one outcome) - -| Pointer | Outcome | -| ---------------------------- | ---------------------------------------------------- | -| `Set-Cookie`, `X-Set-Cookie` | typed `datadome` cookie operation (§4a) | -| `Location` | forward (Respond, 3xx only) | -| `Content-Type` | forward (Respond only) | -| `Cache-Control` | restricted merge | -| `Pragma` | drop-individually (logged), never batch-invalidating | -| `X-DataDome`, `X-DD-*` | forward as owner-scoped typed telemetry headers | -| anything unclassified | invalidate the batch → Continue | - -The documented vendor response (`Set-Cookie`, `Pragma`, `X-DataDome`, -`Cache-Control`) is a verbatim fixture asserting the decision stays -**Respond** and the mapped fields are emitted exactly. +## The single pointer matrix (normative — decision × session mode × pointer) + +This is the one authoritative contract; the hook spec §4a references it +and carries no duplicate lists. Session mode is **cookie** in v1 +(sessionByHeader is startup-rejected; a header-mode column is added by +the sign-off-23 opt-in, never implicitly). No wildcard rows exist — +every accepted name is enumerated. + +| Pointer | Respond (cookie mode) | Continue (cookie mode) | +| ------------------- | ---------------------------------------------------------------------------- | ------------------------------------- | +| `Set-Cookie` | typed `datadome` cookie operation | typed `datadome` cookie operation | +| `X-Set-Cookie` | unclassified → batch handling (v1: sessionByHeader rejected) | unclassified → batch handling | +| `Location` | forward (3xx only), replace | rejected | +| `Content-Type` | forward (owns its body), replace | rejected | +| `Cache-Control` | restricted merge; invariant pass last | rejected | +| `Pragma` | drop-individually, logged | drop-individually, logged | +| `X-DataDome` | forward, owner-scoped typed telemetry | forward, owner-scoped typed telemetry | +| `X-DD-B` | drop-individually, logged (header-session artifact; harmless in cookie mode) | drop-individually, logged | +| anything not listed | invalidate the batch → Continue | invalidate → effects dropped | + +**Fixtures**: DataDome's documented challenge response (`Set-Cookie`, +`Pragma`, `X-DataDome`, `Cache-Control`) asserts the decision stays +**Respond** with exactly the mapped fields; the documented allow example +(`Set-Cookie X-DD-B`) asserts Continue proceeds with the cookie applied +and `X-DD-B` dropped-and-logged — neither fixture may fail open. + +Note: the vendor's `X-Set-Cookie` response field is never forwarded as a +header; in v1 it is unclassified because sessionByHeader is rejected at +startup (hook spec §4a). diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index a3fb54046..fd9848b40 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -342,7 +342,10 @@ left open:** every Respond ends `private, no-store` with CDN fields stripped. - GPP 24–27 fully reserved: removed from the accepted-version table into a separate reserved table; presence-vs-unknown defined; supported - sections pin to an immutable registry commit with vendored vectors. + sections **must** pin to an immutable registry commit with vendored + vectors — the snapshot and PSL files remain placeholders until + ratification records the commits (a named ratification gate, not a + closed item). - P2/P3: HEAD-only distinct artifact type with validator/Content-Length update rules; Vary→cache-key ordering; canonical `state_key`/evidence digest construction (`tsstk1|`/`tsevd1|`) with a vector; non-HMAC @@ -361,3 +364,66 @@ family non-HMAC `tsfam1|i|vend|abcdef` → suffix `tswsx1|AbC123` → `08cb55acf42929772862e82b0960c134`; state_key `tsstk1|tcf|p1=grant,p4=refuse` → `a49148a0e3b486fd93a141404857868871319a7e1f2ef85b5499aed80c7e59df`. + +## Round 18 — review at 184c9d9b (prose; text-added pending next review) + +- N+1's rollback **tests** now mirror the one contract (stubs + negative + writes read-and-write; positive commits/clears asserted forbidden) — + the contract/test contradiction is gone. +- The graphless migration has a **total state table** (semantics × flag × + strong record × row read × `w` → outcome), with the declared v1 + exception (v1 semantics use recognized cookies row-lessly until the + new model activates — matrix row 14), strong-record-present + + successful stale not-found defined as visibility lag → indeterminate, + and the "while the flag is active" `w` scope leftover replaced by + valid_until keying. +- The suspension barrier is skew-safe: `not_before` from store-issued + time or committer + L + S (S = 300 s), with `not_before` and L + serialized in the metadata schema; clock-skew and suspender-restart + tests named. +- The strong summary is the **sole S2S decision source** — the §7 + authority paragraph and the row-schema provenance row (now "audit + mirror only") no longer describe a second source. +- The saturation restrictive marker pins to the **first restrictive + overflow's own timestamp with a full TTL** (later overflows inherit — + bounded shortening, ratified in the rewritten sign-off 31), replacing + the epoch-entry pinning that back-dated genuinely new opt-outs. +- Revision identity is one pair everywhere: (tspol1| content digest, + deployment-metadata activation ordinal); the hook's push-version and + digest-only variants are superseded; §5.5 rebuilt (also repairing a + paragraph corrupted by an earlier scripted edit). +- Cache stickiness is form-preserving: unqualified directives never + become qualified, qualified field sets never shrink; the contradictory + must-understand tail is gone. +- Overlay cache identity: matching on the exact final publisher request, + digest-only storage of sensitive Vary values, and identity-bearing TS + overlays force private, no-store. +- sessionByHeader is startup-rejected in v1 (never requested, + X-Set-Cookie unclassified, incoming header ClientIDs not forwarded); + the R16 cookie-translation is retracted as not equivalent; the old + DataDome spec's banner now supersedes its header-mode requirement. +- One pointer matrix (decision × session mode × pointer) lives in the + allowlist file with X-DD-B enumerated and no wildcard; both documented + vendor responses are decision-asserting fixtures. +- Browser trust boundary ratified, not implied: non-HttpOnly cookie + readable by every same-origin script, vendor challenge HTML with + publisher-origin access, CSP interaction — sign-offs 23/28 rewritten. +- HEAD/304 store separate origin-side and processed-side metadata; + byte-coupled representation updates invalidate and refetch (RFC 9111 + §3.2); changed Vary evicts/rekeys. +- P2s: per-entry `w` horizons (record expires with its last entry); + state_key/evidence digests hash the ASCII source token (prose aligned + to the published vector; slots are per source; evidence-digest vector + embedded: `tsevd1|tcf|p1=grant,p4=refuse|lu=1690000000000` → + `67259c0247ae2b33c52d9f18193bcd622f48ad4754ffbab6998d7c293b0143b4`); + schema-floor value encoded (writer version + minimum-reader, numeric + order — N+1 with N+2-reader capability is permitted); DataDome + complete-response deadline 3000 ms monotonic (1500 ms stays + first-byte), encoded responses batch-invalid; total cookie parser + (repeated Cookie joined, Set-Cookie never combined, strict attribute + rejection, 512-byte serialized measure); adapter header ceilings are + enumerated capability cells validated against core's budget at + startup; GPP/PSL placeholder status stated honestly here and above. +- P3s: §5.5 grammar dangler repaired with the section rebuild; the + remaining "and the and a" occurrence fixed; the duplicated + Set-Cookie-reserved sentence deduplicated. From fe4d7bd3443c4a5bb80cc86ada0cf9b11f64569c Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 4 Aug 2026 15:29:54 -0700 Subject: [PATCH 20/24] Address nineteenth review: invariant negative gates, complete state coverage, and single-source activation identity P1 fixes: - Negative gates are release-, config-, and flag-invariant: the total state table opens with that rule and splits the v1 catch-all into three rows - positive relaxation only, revocation/suppression present -> denied, live w -> withdrawn - so an N+2-written rowless withdrawal binds a rolled-back N+1 (the reproducer lands on the w row, not the catch-all); the migration rollback contract states the same invariance. - The state table covers ordinary operation and legacy backfill: row-backed use with matching authority is normal use in every flag state (the flag governs rowless classification only), and a found row with a stub-only authority record routes to the live AuthorityRefresh backfill - which admits on the observed row plus live resolution, never a prior summary, so the legacy recovery path is reachable by construction (permission spec states it too). - The error default no longer suppresses admitted destructive withdrawal: 'no negative writes' became 'no writes whose admission depended on the failed read' - a successfully read strong family record proves admission, so live GPC commits the family revocation (and SharingOptOut its suppression CAS) despite a row read failure; only the cookie expiry waits for the commit. - The suspension deadline has a clock contract end to end: the store-time branch re-reads store current time at every not_before check (one clock domain); the fallback subtracts S_fleet - the maximum pairwise fleet clock skew, a declared and monitored infrastructure bound distinct from the evidence tolerance S - at comparison as well as at write; the clock domain is serialized in the metadata value; fastest-observer/slowest-committer and suspender-restart tests are named; a backend with neither store time nor a skew bound cannot host the migration. - The activation ordinal is a real register: deployment-metadata name 02 holds linearizable {source_version, policy_digest, ordinal} with idempotent same-activation reuse (the CAS winner assigns, everyone else adopts) and stale-source_version rejection; digest-only backends are declared unable to detect staleness (safe, visible in the metric); the hook's 'globally assigned push version' revision is superseded by the tscfg1| digest plus the section 5.5 pair. - The strong summary can enforce jurisdiction expiry: it gains jurisdiction_observed_at, written only by live geo resolution - evidence timestamps are disqualified as proxies (TCF LastUpdated predates lookups; policy baselines have no wall clock) - and decision 25's S2S age gate measures against it. - The saturation shortening is a declared product choice, not a buried contradiction: permission 4.3's TTL-sticky rule carries the exception (a restrictive overflow during a saturated epoch inherits the first-overflow marker, down to nearly zero lifetime), decisions 16 and 31 both state it with the rejected alternatives (per-overflow state: unbounded storage; marker refresh: replay extension), and epoch expiry is a complete transition - lazy slot GC, re-saturation opens a new epoch pinned to its own first overflow, no cross-epoch timestamp inheritance. - The pointer matrix has no predecessors: the allowlist file is rewritten with the matrix as the only browser-response contract (old response-direction table and cookie-translation notes deleted), the X-Set-Cookie cell terminates in one exact outcome (invalidate the batch -> Continue, mode mismatch), and the hook's inline decision list is physically replaced by a deferral - its 'unclassified -> batch handling' phrase aligned to the cell. - Incoming header ClientID has one v1 path: stripped from the shared request and never used for the vendor payload (ClientID derives only from the datadome cookie; forwarding a header ClientID would misdeclare the session mode per the vendor contract); the unreachable 'header form wins' priority rule is deleted; a both-sources fixture pins cookie-only derivation. - The 304 flow is staged then atomic: metadata diffs off-record against stored origin-side metadata; byte-coupled fields gate on changed, not present (a 304 routinely repeats the matching validator); safe changes publish in one atomic cache commit so a concurrent hit never sees old transformed bytes under new policy metadata; unchanged -> local-hit re-emit. - Artifact recovery is unconditional: absence or revision mismatch strips every conditional field - the client's and TS's own - fetches and processes the full 200 under current revisions, then evaluates the client's condition against the new processed validator. P2/P3: the N+1 interim is one release x config-shape matrix (policy resolution, live gating, negative gates, row writes, positive commits, batch egress - a new-shape denial on N+1 is telemetry, never gating); GPP/USP evidence digests omit |lu= entirely with an embedded vector (tsevd1|gpp|... -> 89b08580...), and replay slots carry a per-permission observed_at map refreshed only for changed tokens (a P4-only change cannot refresh P1's age); w capability retention is the per-entry max(cookie, row, S2S) horizon with a permanent strong-read obligation for HMAC row discovery; HEAD updates compare against origin-side metadata only and never touch processed-side headers; the sensitive-Vary rule became a reproducible contract (HMAC-SHA-256, tsvry1| grammar, every nominated value digested, secret-store key with versioned id, zero-key vector 60fdeb3a...); the adapter ceiling matrix now exists (Axum fixed at/above budget; Fastly/Cloudflare/Spin qualification-pending, fail-closed at startup) with the exact counted-bytes formula and over-budget-snapshot behavior; the DataDome cookie parser defines the '; ' join, duplicate-datadome ambiguity, foreign-name and HttpOnly rejection, floor-based Expires conversion, and normalized-form size; pointed-field multiplicity is closed (singleton duplication invalidates atomically, list fields join per RFC 9110 5.3); Respond on HEAD validates the body but emits none; the 3000 ms deadline has measurement points (pre-acquisition to final byte, async cancellation); malformed Cache-Control has an enumerated result (uncacheable + merging batch rejected); sign-off 28 names the exact X-DD-B divergence; the GPP snapshot joins PSL as a named ratification gate and states its own placeholder status; the state table's stray column and absorbed prose are gone; the metadata row is a proper table cell; the authority row reads 'not read for gating'; 'ratified in sign-offs 23/28' became 'enter sign-offs 23/28 for ratification'; the two R18 ledger overclaims (matrix single-sourcing, ceiling cells) are corrected as text-added-not-verified; and the old DataDome spec's superseded steps carry inline supersession notes at each site, not only the banner. --- ...-datadome-server-side-protection-design.md | 22 +- ...integration-response-header-hook-design.md | 194 +++++++++------ .../2026-07-30-permission-model-design.md | 49 +++- .../2026-07-30-pluggable-providers-design.md | 232 +++++++++++------- ...07-30-provider-migration-rollout-design.md | 37 ++- .../specs/datadome-header-allowlist.md | 74 +++--- .../specs/gpp-registry-snapshot.md | 6 + docs/superpowers/specs/pr986-review-ledger.md | 131 +++++++++- 8 files changed, 523 insertions(+), 222 deletions(-) diff --git a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md index c1b33d460..31a539527 100644 --- a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md +++ b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md @@ -79,6 +79,9 @@ JavaScript SDK. Store using configured store/name fields. Do not store the literal key in `trusted-server.toml`. 8. **Timeout:** use `1500ms` as the default Protection API timeout for v1. + _(Superseded: `1500 ms` is the **first-byte** bound only; the + complete-response deadline is 3000 ms with defined measurement + points — hook spec §4a.)_ 9. **Duplicate tag handling:** do not attempt automatic duplicate-tag detection in v1; operators can disable injection with `inject_client_side_tag = false`. @@ -215,6 +218,9 @@ Important behavior: - Response header mutations are accumulated and applied to the final response. - On `Respond`, routing short-circuits with that response while preserving any downstream response header effects that must be applied after finalization. + _(Superseded: one global order applies — core finalization → ordinary + mutators → security effects → invariant pass unconditionally last; + nothing applies after the invariant pass — hook spec §4a.)_ - DataDome transport/API failures should not bubble out as registry errors; DataDome should convert them to `Continue(Default::default())` to preserve fail-open behavior. @@ -447,7 +453,7 @@ Request headers: ```text Content-Type: application/x-www-form-urlencoded Content-Length: -X-DataDome-X-Set-Cookie: true # only when X-DataDome-ClientID is used +X-DataDome-X-Set-Cookie: true # only when X-DataDome-ClientID is used — SUPERSEDED for v1: never sent (hook spec §4a) ``` Payload fields should include the core fields from DataDome's official module: @@ -494,6 +500,9 @@ Payload fields should include the core fields from DataDome's official module: When `X-DataDome-ClientID` is used, send `X-DataDome-X-Set-Cookie: true` to the Protection API. +_(Superseded for v1: header-supplied ClientIDs are not forwarded at +all — the vendor payload's ClientID derives only from the `datadome` +cookie, so this header is never sent — hook spec §4a.)_ Encoding and size rules: @@ -722,7 +731,10 @@ Update after implementation to describe: - form encoding is correct - empty fields are omitted - `ClientID` comes from `X-DataDome-ClientID` before cookie + _(superseded for v1: cookie-only — the header is stripped and never + used for the vendor payload, hook spec §4a)_ - `X-DataDome-X-Set-Cookie` is sent when header-based ClientID is used + _(superseded for v1: never sent, hook spec §4a)_ - `datadome` cookie is parsed safely - long fields are truncated according to configured limits - request headers list is generated deterministically enough for tests @@ -788,7 +800,9 @@ passes. methods, including `HEAD`, are eligible when the URL is otherwise in scope. 2. The DataDome server-side key is loaded from runtime Secret Store in v1. The config contains only the secret store and secret name. -3. The default Protection API timeout is `1500ms` for v1. +3. The default Protection API timeout is `1500ms` for v1. _(Superseded: + first-byte bound only; 3000 ms complete-response deadline — hook + spec §4a.)_ 4. Auto-injection does not attempt duplicate-tag detection in v1. The explicit `inject_client_side_tag = false` escape hatch is sufficient. @@ -796,7 +810,9 @@ passes. 1. **Timeout semantics:** `timeout_ms = 1500` is the v1 default and maps to the dynamic backend first-byte timeout. It is not a full end-to-end response-body - deadline in v1. + deadline in v1. _(The hook spec §4a now adds the 3000 ms + complete-response deadline on the monotonic clock with defined + measurement points; both bounds apply.)_ 2. **Client metadata scope:** JA4 and H2 fingerprint values are sent only in the form-encoded Protection API payload to DataDome. They are not forwarded to the publisher origin or returned to the browser unless DataDome independently diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 46afe5a6e..8f54ae87a 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -110,17 +110,32 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may computation → cache-key construction → body/metadata commit** — with three identity rules. Cache matching uses the **exact final publisher request** (post-overlay view; keying from the redacted view would - collapse personalized variants). Nominated request values are stored - **only as keyed digests** when sensitive (cookies, identity overlays, - bearer tokens named by `Vary` must never be persisted literally). And + collapse personalized variants). **Every** `Vary`-nominated request value is stored **only as a keyed + digest** — every value, not a sensitivity classification an unknown + credential field could slip past: HMAC-SHA-256 with domain tag + `tsvry1|`, input = `tsvry1|` + lowercased field name + `|` + the + value octets (repeated members of one field joined with a single + comma in received order before hashing; an absent field hashes the + fixed empty-value form `tsvry1||`), keyed by a fleet-stable key + from the platform secret store with the key id versioned into the + cache entry (rotation = introduce a new id; entries under old ids + simply miss and refill), output lowercase hex (64 chars). + Known-answer vector under the all-zero 32-byte test key: + `tsvry1|authorization|Bearer abc` → + `60fdeb3a933d038ba9dc29a860dc4b2f8c200a0a82f4c3842fc68af62f37589b`. And a response derived from a request carrying an **identity-bearing TS overlay** (the DataDome ClientID overlay) is forced `private, no-store` unless an explicit per-overlay contract says otherwise — the `Authorization` rule protects origin credentials, and this rule protects the identity TS itself injected. Parsing itself is a **shared core parser with fail-closed normalization**, not four adapter interpretations: - invalid `Cache-Control` syntax normalizes to the most restrictive - reading; duplicate directives keep the strongest; quoted and unquoted + a `Cache-Control` value that fails the shared grammar has an + **enumerated result, not a "most restrictive reading"** (restrictions + are independent axes, so no single most-restrictive point exists): + the response is treated as **uncacheable for the storage decision** + (`no-store`-equivalent in the invariant) and any mutation batch + merging against the malformed value is rejected whole; among + well-formed values, duplicate directives keep the strongest; quoted and unquoted forms are equivalent; conflicting `max-age` values keep the smallest; unknown extension directives are dropped **from mutations only — unknown directives already in the snapshot are preserved verbatim** (a @@ -224,13 +239,28 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may operations (≤ 32), added headers (≤ 16), and added bytes (≤ 8 KiB), and a **cumulative final-response budget** (≤ 128 headers / ≤ 32 KiB total, counting `name: value` plus separators, within any lower adapter - ceiling — **and those ceilings are enumerated capability cells per - adapter, with the counting rule fixed as serialized `name: value` - bytes plus separators**, validated against core's budget at startup - so a batch that passes core can never fail only on one adapter) + ceiling — and those ceilings are the enumerated capability cells **below**, + with the counting rule fixed exactly: counted bytes = Σ over emitted + fields of `len(name) + 2 + len(value) + 2` (the `": "` and CRLF + separators), validated against core's budget at startup so a batch + that passes core can never fail only on one adapter; a snapshot + already **over** the core budget before any mutation rejects every + mutation batch — the budget bounds additions and never bricks an + over-budget origin response, which passes through and is counted) bounds the sum across integrations — enforced in registration order, so which operations are rejected when a budget trips is - deterministic. Each mutator receives an **immutable, redacted snapshot of the + deterministic. + + | Adapter | Header-count / total-bytes ceiling (capability cell) | + | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | + | Axum | no platform ceiling below the core budget (native HTTP stack) — cell fixed at ≥ 128 headers / ≥ 32 KiB | + | Fastly | **qualification-pending**: the measured platform ceiling is recorded in this cell by the adapter-qualification commit; unrecorded ⇒ hook startup fails | + | Cloudflare | **qualification-pending**: same rule | + | Spin | **qualification-pending**: same rule | + + A recorded cell below core's 128-header / 32 KiB budget is a startup + error (shrink the core budget or raise the ceiling — never a silent + per-adapter divergence). Each mutator receives an **immutable, redacted snapshot of the response head** (status and headers as of its turn, prior integrations' accepted operations applied) as its read context; it never holds a mutable reference (§2). Redaction is a security boundary, not @@ -267,17 +297,17 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by a fleet-stable tuple** — the integration-registry revision is a content hash over the ordered (integration ID, version) list; the config revision is the config store's globally assigned push version; the invariant revision is a build-time spec-version constant — local counters would collide across instances and binaries; there are **two distinct 304 cases**. A **locally generated conditional hit** (TS answers the client's `If-*` from its own fresh stored artifact) re-emits the persisted finals when all three revisions match, else cache-miss — as before. An **origin-revalidation 304** (TS revalidated upstream and the origin returned 304 with possibly new `Cache-Control`/`Vary`/`Expires`/validators) is different: RFC 9111 §4.3.4 requires the stored response to be **updated from the current 304 before serving**, so TS updates the stored base with the 304's metadata first; if any admitted/cache-relevant field changed, it **reruns the relevant processing or refetches a full 200** rather than re-emitting stale finals (a cached `public` followed by an origin `304 Cache-Control: private, no-store` must not keep serving the old public policy). Artifact absence or a revision mismatch strips internal preconditions before obtaining that full response. **Origin-side and processed-side metadata are stored separately** — the origin's validators/`Content-Length` describe origin bytes, not the rewritten HTML artifact — and any update touching **byte-coupled representation fields** (`Content-Encoding`, `Content-Type`, validators, digests) **invalidates and refetches a full 200 rather than updating** (RFC 9111 §3.2 excludes `Content-Length` from stored-response updates and warns against updating transformed artifacts with incompatible representation metadata); a changed `Vary` evicts or rekeys the stored index entry. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | -| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals stored as a **distinct head-only artifact type that never satisfies a later GET** — when a stored GET artifact exists a HEAD may **update** it only if validators and `Content-Length` match (RFC 9111 §4.3.5), a mismatch **invalidating** the stored GET artifact rather than updating it | -| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | +| Response | Hook runs? | +| ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by a fleet-stable tuple** — the integration-registry revision is a content hash over the ordered (integration ID, version) list; the config revision is the `tscfg1 | `effective-config digest, and the policy revision is the (digest, activation-ordinal) pair from the permission spec's §5.5 register — the earlier "config store's globally assigned push version" definition is superseded (not every adapter has one; the register orders activations for all of them); the invariant revision is a build-time spec-version constant — local counters would collide across instances and binaries; there are **two distinct 304 cases**. A **locally generated conditional hit** (TS answers the client's`If-\*`from its own fresh stored artifact) re-emits the persisted finals when all three revisions match, else cache-miss — as before. An **origin-revalidation 304** (TS revalidated upstream and the origin returned 304 with possibly new`Cache-Control`/`Vary`/`Expires`/validators) is different — RFC 9111 §4.3.4's update-then-serve is implemented **staged, then atomic, never in place**: TS stages the 304's metadata off-record and diffs it against the separately stored **origin-side** metadata (origin validators/`Content-Length`describe origin bytes, not the rewritten HTML artifact — origin-side and processed-side metadata are stored separately). (a) If any **byte-coupled representation field changed** —`Content-Encoding`, `Content-Type`, validators, digests; **changed, not merely present**, since an ordinary 304 routinely repeats the matching validator — nothing is published: the stored entry is invalidated and a full 200 is fetched and processed before any serve (RFC 9111 §3.2 excludes `Content-Length`from stored-response updates and warns against updating transformed artifacts with incompatible representation metadata). (b) If only safe cache-relevant fields changed, the origin-side metadata and the re-derived persisted finals publish in **one atomic cache commit** — a concurrent hit observes the complete old entry or the complete new one, never old transformed bytes under newly published policy metadata (a cached`public`followed by an origin`304 Cache-Control: private, no-store`must not keep serving the old public policy); a changed`Vary` evicts or rekeys the stored index entry inside that same commit. (c) Nothing changed → the persisted finals re-emit as in the local-hit case. Artifact absence or a revision mismatch makes the recovery fetch **unconditional — every conditional field is stripped, the client's (`If-None-Match`, `If-Modified-Since`) and TS's own alike** (forwarding the client's condition could return another 304 TS holds no usable bytes for); TS obtains and processes the full 200 under current revisions, then separately evaluates the client's original condition against the **new processed validator**, answering the client 304 only on a match. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | +| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals stored as a **distinct head-only artifact type that never satisfies a later GET** — when a stored GET artifact exists a HEAD may **update** it only when the comparison — made against the separately stored **origin-side** metadata, never the processed artifact's (origin and rewritten lengths differ by construction, so the processed length is never the comparand and HEAD metadata never touches processed-side headers) — finds validators and origin `Content-Length` matching and no byte-coupled representation field changed (RFC 9111 §4.3.5); qualifying updates land origin-side under the same atomic-commit discipline as the 304 rules, and any mismatch **invalidates** the stored GET artifact rather than updating it | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). @@ -305,11 +335,22 @@ degree of freedom is closed: normalized to its Max-Age equivalent (both present → `Max-Age` wins, per RFC 6265); a normalized lifetime exceeding the ceiling rejects the whole operation batch; and the parser is total: repeated `Cookie` - request fields are tolerated and joined (per the current cookie RFC), - `Set-Cookie` fields are **never combined**, duplicate attributes, - duplicate cookies in one field, an unparseable `Expires`, or unknown - attributes each reject the operation (the vendor cookie is - well-formed; strictness is safe), and the 512-byte limit measures the + request fields are joined with `"; "` (semicolon-space — the + order-preserving join of the current cookie RFC) before parsing, and + **duplicate `datadome` pairs after the join make the request-side + identity ambiguous: treated as cookie-absent for the vendor call and + counted**, while cookies under other names pass through untouched; + `Set-Cookie` fields are **never combined**, and a vendor response + carrying more than one `datadome` `Set-Cookie` field, a `Set-Cookie` + for any other name (the typed operation pins the name exactly), an + **`HttpOnly` attribute** (the vendor requires script readability — + its presence is a protocol anomaly), duplicate attributes, duplicate + cookies in one field, an unparseable `Expires`, or unknown attributes + each reject the operation batch (the vendor cookie is well-formed; + strictness is safe). `Expires` normalizes as + `Max-Age = max(0, floor(expires − now))` whole seconds on the server + wall clock at parse time (the shared skew-bounded basis; a result of + 0 is a deletion), and the 512-byte limit measures the **normalized** serialized `name=value` plus attributes in bytes; `Max-Age` at most **31,536,000 seconds** (the vendor's one-year cap — the earlier 396-day figure exceeded it); size ≤ **512 bytes** @@ -321,7 +362,8 @@ degree of freedom is closed: a violating operation is rejected whole (the batch rule). **Both sessionByHeader is startup-rejected in v1 — one state, not three**: TS never sends `X-DataDome-X-Set-Cookie: true`; a vendor - `X-Set-Cookie` is unclassified (→ batch handling); and an incoming + `X-Set-Cookie` **invalidates the batch → Continue** (its matrix cell — + a session mode the fleet never requested must not half-apply); and an incoming browser `X-DataDome-ClientID` is **not forwarded to the vendor** (cookie-only session identity — an earlier revision translated `X-Set-Cookie` into an ordinary cookie, which is not equivalent: @@ -340,8 +382,9 @@ degree of freedom is closed: serves vendor-owned HTML under the publisher origin (vendor scripts with same-origin access to cookies, storage, and APIs; publisher CSP can conversely break the challenge). Those browser-side observers, - same-origin vendor code, CSP interaction, and challenge redirects are - ratified in **sign-offs 23/28**, not implied. The server-side strip + same-origin vendor code, CSP interaction, and challenge redirects + enter **sign-offs 23/28** for ratification (both decision records + still open), not implied. The server-side strip inventory is exhaustive, not integration-scoped — the browser sends `datadome` in the ordinary `Cookie` header, so it is removed from **every non-DataDome surface**: other integrations' request views, publisher-origin proxy forwarding, @@ -357,11 +400,19 @@ degree of freedom is closed: like the cookie.** DataDome prioritizes the header over the cookie, so leaving it in the shared request would hand other integrations and upstream routing the same identifier the cookie boundary strips: core - **extracts it into the DataDome-only view and removes it from the - shared request** before integrations and upstream routing run — - it joins `RedactedRequestView`'s enumerated strip set (providers - spec) — and only DataDome-returned overlay data reaches the - publisher, never the raw browser-supplied header. + **removes it from the shared request** before integrations and + upstream routing run — it joins `RedactedRequestView`'s enumerated + strip set (providers spec) — **and in v1 it is stripped for the + vendor too: the Protection API request's ClientID derives only from + the `datadome` cookie, never from the incoming header** (DataDome's + contract requires `X-DataDome-X-Set-Cookie: true` whenever a + header-supplied ClientID is forwarded, so forwarding the header under + cookie-only mode would misdeclare the session mode); observed header + occurrences are counted, and a **fixture pins the path**: a request + carrying both cookie and header produces a vendor payload whose + ClientID equals the cookie value, with no header-derived identity + sent. Only DataDome-returned overlay data reaches the publisher, + never the raw browser-supplied header. - **The pointer protocol has a total parser contract** — adapters cannot differ where malformed batches fail open: the pointer list is tokenized by the vendor's documented space separation — repeated @@ -370,9 +421,17 @@ degree of freedom is closed: then names are ASCII-lowercased before duplicate detection; duplicate names after normalization, invalid names, more than 16 pointers, or more than 4 KiB of pointer payload render the batch invalid - (→ Continue, the vendor's fail-open); when both cookie sources arrive - (header form and `Set-Cookie`), the header form wins, matching the - vendor's documented priority. + (→ Continue, the vendor's fail-open). **Pointed-field multiplicity is + closed**: for singleton fields (`Location`, `Content-Type`, + `X-DataDome`, `X-DD-B`, `X-Set-Cookie`) more than one instance in the + vendor response invalidates the batch atomically — never a + first/last/join choice an adapter makes; list-valued fields + (`Cache-Control`, `Pragma`) are joined per RFC 9110 §5.3 before their + matrix outcome applies; `Set-Cookie` multiplicity follows the typed + cookie rule (exactly one `datadome` field, above). No both-source + priority rule exists in v1: the header session form (`X-Set-Cookie`) + is matrix-governed as batch-invalid, so "header form wins" is + unreachable and deleted. - **Request-header pointers are a positive, enumerated allowlist.** "Documented enrichment headers" is not enforceable; the registration enumerates the exact names from the **checked-in allowlist file @@ -391,29 +450,16 @@ degree of freedom is closed: routing-authority fields — is rejected by name and by class: a compromised endpoint must not replace origin credentials, inject `ts-ec`, or spoof client location. -- **Browser-response headers are a decision-scoped positive allowlist - too** — request pointers were enumerated, response headers were not, - leaving either the six-field ordinary registry (which would reject a - challenge's `Location`) or an open door. Normatively, per decision: - a _Respond_ (challenge/deny) may set exactly `Location` (replace; - 3xx only), `Content-Type` (its own body, per the representation rule - below), `Cache-Control` (through the restricted merge; the invariant pass - still runs last). **`Pragma` and its kin get a defined middle path**: - allowlist-absent fields that are known-harmless standard cache - metadata (`Pragma` is the enumerated case — response `Pragma: -no-cache` has no standardized meaning, RFC 9111 §5.4) are **dropped - individually and logged**, never batch-invalidating; genuinely - unknown or active fields still invalidate the batch (→ Continue). - Without this split, DataDome's own documented response — which points - at `Set-Cookie`, `Pragma`, `X-DataDome`, and `Cache-Control` — would - fail every challenge open; that documented vendor response is a - **verbatim test fixture**, and the fail-open consequence of - batch-invalidation is explicitly within sign-off 28's scope, the typed security cookie (above), - and the vendor response headers enumerated in the **response section - of `datadome-header-allowlist.md`**; a _Continue_ may set only the - typed cookie and those enumerated vendor headers. Everything else is - rejected — the atomic-302 example's `Location` is hereby admitted - rather than assumed. +- **Browser-response headers are decision-scoped through the single + matrix, and only there.** This spec carries **no per-decision field + list**: every pointer's outcome per decision and session mode — + `Location`'s Respond-only 3xx admission, `Pragma`'s + drop-individually middle path (response `Pragma: no-cache` has no + standardized meaning, RFC 9111 §5.4), and the batch-invalidation + default for unlisted names — is exactly one cell of the matrix in + `datadome-header-allowlist.md`. The fail-open consequence of batch + invalidation stays within sign-off 28's scope, and both documented + vendor responses are verbatim fixtures at the matrix. - **Representation rules are decision-scoped and narrow.** A _Respond_ decision (challenge/deny) owns its body but may describe it with **`Content-Type` only** — encoding and validator fields @@ -424,15 +470,25 @@ no-cache` has no standardized meaning, RFC 9111 §5.4) are **dropped Continue). If the vendor ever requires more, it arrives as a reviewed allowlist-file addition. A _Continue_ decision may not touch representation metadata of publisher bytes. -- **Respond transport is bounded.** The challenge body has a maximum - size (64 KiB) and a **complete-response deadline of 3000 ms on the - instance's monotonic clock** (the older spec's 1500 ms is first-byte - only and stays as the first-byte bound); TS sends `Accept-Encoding: -identity`, and because that does not _guarantee_ identity coding, a - response arriving with any `Content-Encoding` is itself batch-invalid; - `Content-Length` is recomputed from the actual bytes before Respond - commits. Exceeding size, first-byte, or total deadline fails the - batch → Continue. +- **Respond transport is bounded, with exact measurement points.** The + challenge body has a maximum size (64 KiB) and a **complete-response + deadline of 3000 ms on the instance's monotonic clock, measured from + immediately before vendor-backend acquisition/dispatch to the final + body byte** — connection setup and request send are inside the + window; the 1500 ms first-byte bound (the older spec's figure, now + first-byte only) runs from the same origin on the same clock. At + expiry the decision is final (batch fails → Continue) and the vendor + request is cancelled; cancellation and resource cleanup complete + asynchronously and never delay the response. TS sends + `Accept-Encoding: identity`, and because that does not _guarantee_ + identity coding, a response arriving with any `Content-Encoding` is + itself batch-invalid; `Content-Length` is recomputed from the actual + bytes before Respond commits. **On a HEAD request, Respond validates + the challenge body exactly as for GET (size, deadline, encoding) but + emits no body**: the outward response carries the validated bytes' + `Content-Length` and no content (RFC 9110 HEAD semantics — the older + DataDome spec's HEAD handling is superseded by this rule). Exceeding + size, first-byte, or total deadline fails the batch → Continue. - **One pointer contract, one place.** The single normative decision × session-mode × pointer matrix lives in **`datadome-header-allowlist.md`** — this spec's earlier inline diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 1a6422376..2d62325ee 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -485,7 +485,14 @@ and the fail-closed marker: circulated: not user-sticky-forever, and not the migration spec's former "irreversible artifact requiring administrative clear", which is superseded; administrative clear remains an optional early exit — - sign-off 16); **TCF refusal** — cleared by any regime-accepted grant with newer + sign-off 16 — **with one declared exception**: an opt-out arriving as + a restrictive _overflow_ while its source's replay history is + saturated inherits the saturation epoch's first-overflow marker and + may receive less than a full lifetime, down to nearly zero late in + the epoch (providers spec wire schema; the exception is carried by + sign-offs 16 and 31, and the alternatives — per-overflow state, + marker refresh — were rejected for unbounded storage and + replay-extension respectively); **TCF refusal** — cleared by any regime-accepted grant with newer authoritative evidence; **malformed-present / absence** — cleared by any regime-accepted valid grant with newer evidence, including a timestamp-less grant whose first-seen is newer (these causes are not @@ -841,11 +848,26 @@ domain tag `tspol1|` over the canonical JSON of the parsed policy (keys sorted lexicographically by UTF-8 code unit, numbers shortest round-trip, defaults materialized, no insignificant whitespace; cross-language vectors required) — identity, so an A→B→A rollback -yields A's digest again. The ordinal is a **globally ordered activation -counter, CAS-incremented in the deployment-metadata primitive on each -config activation** — order, adapter-independent (not every adapter has -a native push version, and per-instance counters ordered nothing across -a fleet). Authority wire records, the S2S recompute, and the hook cache +yields A's digest again. The ordinal comes from the **policy-activation register** — +deployment-metadata name `02` (providers spec §6.3), a linearizable +`{source_version, policy_digest, ordinal}` value with three transition +rules that make it idempotent and single-source, not merely a counter: +an activation presenting the register's stored +`(source_version, policy_digest)` pair **adopts the stored ordinal +without incrementing** (every instance activating the same push +converges on one ordinal — the CAS winner assigns, everyone else +reuses, so assignment is effectively single-actor); a novel pair +CAS-increments; and an activation whose `source_version` is **older** +than the stored one is rejected as stale (an instance restarting on old +config can neither mint a new ordinal nor regress the register). +`source_version` is the config store's push version where the backend +assigns an ordered one; a backend without one uses the `tscfg1|` +config-revision digest as `source_version`, which cannot detect +staleness — there, a laggard re-activating an older digest mints a new +ordinal, which is **safe** (revision identity is the pair; fleet order +stays correct) but visible in the activation-ordinal metric. Order is +adapter-independent either way (per-instance counters ordered nothing +across a fleet). Authority wire records, the S2S recompute, and the hook cache tuple all use this same pair; the hook's earlier "config-store push version" and any digest-only usage are superseded. The other cache-tuple inputs are likewise domain-separated hashes of effective configuration: @@ -1003,7 +1025,12 @@ Consumers of the resolved set in this epic: visit — and a visitor who moved from a permissive into a GDPR jurisdiction would otherwise keep old-rule egress for up to the row lifetime. A stored jurisdiction older than the **consent-TTL - horizon** fails closed pending a live refresh (the inverse — denying + horizon** — age measured as now − `jurisdiction_observed_at`, the + summary's dedicated field written **only by live geo resolution** + (providers spec §6.3; evidence timestamps are not a proxy: TCF + `LastUpdated` can predate the live lookup, and a policy-baseline + grant has no wall-clock evidence timestamp at all) — fails closed + pending a live refresh (the inverse — denying a visitor who moved the other way — is the accepted cost); the horizon choice and its legal trade-off are **sign-off item 25**. @@ -1011,7 +1038,13 @@ Consumers of the resolved set in this epic: as reserved `hmac-v0` provenance with **no stored grant evidence**, so they **fail closed for partner egress and batch updates** until a live browser request lazily backfills provenance from a fresh resolution. - Failing open here would grandfather every pre-epic identity past the + That path is reachable by construction: a found legacy/v1 row whose + authority record is a stub (or absent) is **denied egress but not + indeterminate** — the permission-exempt `AuthorityRefresh` admits on + the observed row plus the live resolution, never on a prior positive + summary or matching revision, commits the summary, and the revision + fence then opens use (providers spec §5 total state table). Failing + open here would grandfather every pre-epic identity past the permission model indefinitely. 4. **Server-side auction dispatch** — gated on the policy `regime` class, diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index d8f9c4a98..844cd4752 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -318,14 +318,30 @@ variant**. Therefore: observable strong reads _in addition to_ CAS — a capability cell, since CAS alone says nothing about what other instances currently see). The suspension transition stamps a **fleet-stable `not_before` - deadline** into the suspended epoch — computed from **store-issued - time where the backend provides one, else committer time + L + S** - (S = the normative 300 s skew bound; without folding S in, a - slow-clocked suspender could write a deadline already passed on - another instance while an N+2 lease survived) — where **L is an - assigned constant (120 s)**; **both `not_before` and L are serialized - fields of the metadata value** (the schema names them), and - clock-skew and suspender-restart schedules are named tests: **every** N+1 instance — the one that suspended, a second that + deadline** into the suspended epoch — **and the deadline comparison + has a clock contract, not just the write**: where the backend issues + store time, `not_before` = store time + L and every instance's + `now ≥ not_before` check **re-reads store current time on the same + strong primitive** — one clock domain end to end, because a fast + local clock compared against a store-issued deadline could cross it + while another instance's N+2 lease was still valid. Where the + backend has no store clock, `not_before` = committer time + L + + S*fleet and every local comparison subtracts S_fleet again + (mint only when `now − S_fleet ≥ not_before`) — **S_fleet is the + maximum pairwise fleet clock skew, a declared and monitored + infrastructure bound, a separate qualification from the + evidence-timestamp tolerance S even though both are assigned 300 s** + (an adapter capability cell states which branch the backend + qualifies for; a deployment that can guarantee neither store time + nor a fleet-skew bound cannot host the graphless migration). Without + the bound, a slow-clocked suspender could write a deadline already + passed on another instance while an N+2 lease survived, and a + fast-clocked N+1 could mint before the fleet had quiesced — where + **L is an assigned constant (120 s)**; **`not_before`, its clock + domain (store or committer), and L are serialized fields of the + metadata value** (the schema names them), and clock-skew + (fastest-observer and slowest-committer schedules) and + suspender-restart schedules are named tests: **every** N+1 instance — the one that suspended, a second that starts and reads an already-suspended state, or one recovering after the suspender crashed — refuses to mint until `now ≥ not_before` (globally strong read of the epoch). N+2 instances prove `active` at @@ -338,7 +354,7 @@ variant**. Therefore: absent → active → suspended → re-attested-active → **suspended** (a second rollback) → … ; CAS losers on any transition re-read and retry against the winner's epoch. Under `suspended`, rowless - _classification_ stops but **`w` consultation and enforcement + \_classification* stops but **`w` consultation and enforcement continue** (withdrawn stays withdrawn). Re-activation after roll-forward requires complete re-attestation over the gap window. The N+2 → rollback-to-N+1 → mint → roll-forward-to-N+2 schedule is a @@ -362,20 +378,31 @@ variant**. Therefore: **The total state table** — every (semantics, flag, strong record, row read, `w`) combination has exactly one outcome; anything not - listed falls to the bolded default: - - | Semantics | Flag | Strong record | Row read | Live `w` entry | Outcome | - | ----------------------------------- | -------------- | ------------- | ------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- | - | v1 (N+1, or N+2 + old-shape config) | any | any | any | any | **v1 behavior: recognized cookies are used per pre-epic rules, rows not required** — the declared v1 exception, not an outage: the pre-epic privacy posture persists until the new model activates (matrix row 14) | - | new model | absent/cleared | present | found, revisions match | none | normal use | - | new model | absent/cleared | present | **not-found (successful read)** | none | **visibility lag, not absence**: the strong record proves the row committed, so the identity is indeterminate this request — no use, no mint, no expiry (a stale replica must not fork a just-minted identity) | - | new model | absent/cleared | absent | any | none | indeterminate (no rowless classification without the flag) | - | new model | active | absent | authoritative not-found | none | rowless: expire-and-re-mint / withdrawal per §5 | - | new model | suspended | absent | any | none | indeterminate (classification off; re-attestation pending) | - | new model | any | any | found | **matching entry or saturated** | denied, then promoted to family revocation (§6.2) | - | new model | any | any | error / `w` read error | — | **default: indeterminate — no use, no mint, no expiry, no negative writes** | Graphless-era cookies never got a stub because | - - they have no row for the scan to find; no eventual read participates. The flag itself is specified: a named + listed falls to the bolded default. **Negative gates are release-, + config-, and flag-invariant**: family revocations, suppression + entries, and live `w` records are read and enforced in every row of + this table, v1 semantics included — the v1 exception relaxes only the + _positive_ side (provenance, row presence, the new gating model), + never the negative one, or an N+2-written rowless withdrawal would + stop binding the moment the fleet rolled back to N+1 (the rollback + contract's N+1 obligations, migration spec §4.4): + + | Semantics | Flag | Strong records (`r`/`s`) | Row read | Live `w` entry | Outcome | + | ----------------------------------- | -------------- | ----------------------------------- | ------------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | v1 (N+1, or N+2 + old-shape config) | any | no revocation, no live suppression | any | none | **v1 positive behavior: recognized cookies are used per pre-epic rules, rows not required** — the declared v1 exception, not an outage: the pre-epic privacy posture persists until the new model activates (matrix row 14) | + | v1 (N+1, or N+2 + old-shape config) | any | revocation or live suppression | any | any | denied exactly as under the new model — negative gates do not wait for the new model | + | v1 (N+1, or N+2 + old-shape config) | any | any | any | **matching entry or saturated** | withdrawn: denied and promoted per §6.2 — a rolled-back N+1 enforces N+2-written `w` records | + | new model | any | present | found, revisions match | none | normal use — in **every** flag state, active and suspended included: the flag governs rowless _classification_ only, never row-backed operation | + | new model | any | **stub only** (no positive summary) | found (legacy or v1 row) | none | no egress, but **not** a dead end: the live-backfill path applies — a live request resolving a regime-accepted grant runs the permission-exempt `AuthorityRefresh` (admission is the observed row + live resolution, never a prior summary or matching revision), commits the positive summary, and the revision fence then opens use (permission spec §7 legacy rule) | + | new model | absent/cleared | present | **not-found (successful read)** | none | **visibility lag, not absence**: the strong record proves the row committed, so the identity is indeterminate this request — no use, no mint, no expiry (a stale replica must not fork a just-minted identity) | + | new model | absent/cleared | absent | any | none | indeterminate (no rowless classification without the flag) | + | new model | active | absent | authoritative not-found | none | rowless: expire-and-re-mint / withdrawal per §5 | + | new model | suspended | absent | any | none | indeterminate (rowless classification off; re-attestation pending) | + | new model | any | any | found | **matching entry or saturated** | denied, then promoted to family revocation (§6.2) | + | any | any | any | error / `w` read error | — | **default: indeterminate — no use, no mint, no expiry — and no writes whose admission depended on the failed read.** Admitted negative writes still proceed: a successfully read strong authority record proves family admission (§5), so a live destructive signal commits its family revocation (and a non-destructive one its suppression CAS) even when the eventual row read or classification failed — only the browser-cookie expiry waits for the revocation commit. A row read failure must never leave S2S authority live against an already-provable withdrawal | + + Graphless-era cookies never got a stub because they have no row for + the scan to find; no eventual read participates. The flag itself is specified: a named deployment-metadata key (write-once/CAS class), set by the §4.2 migration runbook step (migration spec §4) only on deployments that actually ran graphless (requires the deployment-metadata capability), surviving binary @@ -384,7 +411,8 @@ variant**. Therefore: clearing ends rowless classification permanently. Outside the flag, or on any read error, the state is **indeterminate**: no identity use, no mint, no cookie expiry — "treated as absent" was the wrong - contract, since absence feeds the fresh-mint path. + contract, since absence feeds the fresh-mint path (admitted negative + writes still proceed, per the default row above). - A verified rowless cookie (`verify → VerifiedIdentity`, carrying the matched version) is **expired and replaced by a fresh mint through the @@ -619,20 +647,20 @@ Startup validation (§6) covers configuration; this covers what happens when a healthy configuration meets an unhealthy runtime. Every row logs at `error` with a metric; none is silent: -| Failure | Behavior | -| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | -| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | -| Authority-state commit fails after the row commit at mint | No cookie, no eligibility (the strong record never reported the revision); the orphan row expires by TTL; counted — **no recovery claim**, nothing can find it | -| `w` read fails (rowless path or migration-window row discovery) | Fail closed: the cookie/row is **indeterminate** — no use, no mint, no expiry this request | -| `w` write fails at rowless withdrawal | Cookie retained (withdrawal did not durably occur); durable client signal retries next presentation | -| `w` saturation encountered, a real row surfaces under the prefix | **Denied, then promoted to a family revocation** — a saturated prefix means the safe assumption is "withdrawn", so any real row under it is denied all use and its family revoked, listed-hash or overflow alike. The earlier "overflow loses promotion, never blanket-denies" rule left a completed rowless withdrawal usable the instant its row surfaced; that resurrection is closed here, not left to retention. The collateral — a non-abuser row under a saturated NAT prefix is revoked — is sign-off 30 | -| Promotion (listed suffix-hash matches a discovered row) fails | The row is denied all use until the promoted family revocation commits; retried on next observation | -| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | -| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | -| Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | -| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | -| Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | +| Failure | Behavior | +| ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | +| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | +| Authority-state commit fails after the row commit at mint | No cookie, no eligibility (the strong record never reported the revision); the orphan row expires by TTL; counted — **no recovery claim**, nothing can find it | +| `w` read fails (rowless path, or any HMAC row discovery while a live `w` record exists — enforcement outlives the migration window) | Fail closed: the cookie/row is **indeterminate** — no use, no mint, no expiry this request | +| `w` write fails at rowless withdrawal | Cookie retained (withdrawal did not durably occur); durable client signal retries next presentation | +| `w` saturation encountered, a real row surfaces under the prefix | **Denied, then promoted to a family revocation** — a saturated prefix means the safe assumption is "withdrawn", so any real row under it is denied all use and its family revoked, listed-hash or overflow alike. The earlier "overflow loses promotion, never blanket-denies" rule left a completed rowless withdrawal usable the instant its row surfaced; that resurrection is closed here, not left to retention. The collateral — a non-abuser row under a saturated NAT prefix is revoked — is sign-off 30 | +| Promotion (listed suffix-hash matches a discovered row) fails | The row is denied all use until the promoted family revocation commits; retried on next observation | +| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | +| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | +| Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | +| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | +| Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | The **degraded-graph health signal** referenced above and by the withdrawal contract is a defined state machine, not a vibe: it is @@ -662,23 +690,17 @@ solves. **Physical key grammar.** Core constructs every key; providers supply only the bounded suffix: -| Record class | Key | Notes | -| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ **123** bytes (128-byte total key cap minus tag and provider code — an earlier 128 here contradicted the constructor; 123/124 boundary fixtures required), portable alphabet **`[A-Za-z0-9._~-]`** (not "KV-safe per host", which would break the single cross-adapter grammar; providers with native identifiers outside it emit base64url suffixes); ≤ **123** bytes with 123/124 boundary fixtures. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | -| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | -| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | -| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | -| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | -| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8) where **each entry carries its own `valid_until`** = its withdrawal time + max(cookie lifetime, row/S2S authority horizon) — one record-level lifetime either shortchanged late entries or, rolling, let an attacker keep a saturated NAT cohort withdrawn forever; the record expires when its last entry (or the saturation flag's own pinned horizon) expires; saturation flag with its own entry-time-pinned horizon; CAS version; created-at — a `w` expiring before the row horizon would let an overflow row's identity resurface (the overflow non-promotion residual is sign-off 30); readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | -| Deployment metadata | `m` + fixed metadata name (fixed-width grammar; schema floor, graphless-migration flag — the | - -**floor value is encoded**: integer writer-activation schema version + -minimum-reader version, ordered numerically; a binary starts only if its -declared reader capability ≥ the floor's minimum-reader, which is what -makes "is N+1 permitted after N+2 activates" decidable: N+1 declares -N+2-reader capability, so yes) | Write-once/CAS class; value carries schema version, state, epoch, set-at, and — for the graphless flag — **`not_before` and `L`** (serialized, so every observer reads the same deadline) plus the attestation; the graphless flag's lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded | -| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | -| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | +| Record class | Key | Notes | +| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ **123** bytes (128-byte total key cap minus tag and provider code — an earlier 128 here contradicted the constructor; 123/124 boundary fixtures required), portable alphabet **`[A-Za-z0-9._~-]`** (not "KV-safe per host", which would break the single cross-adapter grammar; providers with native identifiers outside it emit base64url suffixes); ≤ **123** bytes with 123/124 boundary fixtures. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | +| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | +| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | +| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | +| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | +| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8) where **each entry carries its own `valid_until`** = its withdrawal time + max(cookie lifetime, row/S2S authority horizon) — one record-level lifetime either shortchanged late entries or, rolling, let an attacker keep a saturated NAT cohort withdrawn forever; the record expires when its last entry (or the saturation flag's own pinned horizon) expires; saturation flag with its own entry-time-pinned horizon; CAS version; created-at — a `w` expiring before the row horizon would let an overflow row's identity resurface (the overflow non-promotion residual is sign-off 30); readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | +| Deployment metadata | `m` + fixed metadata name (fixed-width grammar): schema floor, graphless-migration flag, policy-activation register | Write-once/CAS class. The **floor value is encoded**: integer writer-activation schema version + minimum-reader version, ordered numerically; a binary starts only if its declared reader capability ≥ the floor's minimum-reader, which is what makes "is N+1 permitted after N+2 activates" decidable (N+1 declares N+2-reader capability, so yes). The **graphless flag's** value carries schema version, state, epoch, set-at, **`not_before`, its clock domain, and `L`** (serialized, so every observer reads the same deadline) plus the attestation; lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded. The **policy-activation register** holds the linearizable `{source_version, policy_digest, ordinal, activated_at}` value with the transition rules of permission spec §5.5 (idempotent same-activation reuse; stale `source_version` rejected) | +| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | +| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | Grammars are pairwise non-intersecting by their literal prefixes (plus the reserved hmac grammar), and every record value carries a `kind` @@ -727,7 +749,7 @@ already encodes derivation); `w` + provider-code(4) + prefix(64 hex) for rowless withdrawal; `x` + family-id(64) for transactions; `m` + a **2-digit registry-assigned index** for deployment metadata (a closed name registry in this spec: `00` schema floor, `01` graphless-migration -flag — padding-based names aliased `foo` and `foo-`, so names are not +flag, `02` policy-activation register — padding-based names aliased `foo` and `foo-`, so names are not encoded in keys at all). Maximum physical key length **128 bytes**; every class has a total parser, and segment boundaries are positional, so no segment can contain or escape a @@ -749,10 +771,15 @@ expired entries are inert), and the provenance revision a clear references; positive side (the summary, **every field the permission spec's absence/replay decisions consume — a reduced schema cannot reproduce them**): kind (user evidence vs policy baseline), grant -basis/source class, policy revision (digest + activation generation), **resolved -jurisdiction** (so S2S never reads it from an eventually stale row — -the summary is self-sufficient for the recompute), `valid_until`, -provenance revision, evidence timestamp, and a **bounded replay history** whose slots are keyed by a +basis/source class, policy revision (the §5.5 pair: digest + activation +ordinal), **resolved jurisdiction with its own `jurisdiction_observed_at`** +— set **only by live geo resolution**, never derived from evidence +timestamps (TCF `LastUpdated` can predate the live lookup, and a +policy-baseline grant has no wall-clock evidence time at all), so S2S +never reads jurisdiction from an eventually stale row and decision 25's +stored-jurisdiction age gate has a field that actually measures +jurisdiction age (the summary is self-sufficient for the recompute) — +`valid_until`, provenance revision, evidence timestamp, and a **bounded replay history** whose slots are keyed by a **timestamp-independent `state_key`** — (source class, semantic result digest _excluding_ `LastUpdated`) — distinct from the _evidence digest_ (which for TCF includes `LastUpdated` for recency), both with @@ -765,16 +792,27 @@ input): `state_key` = SHA-256 `tsstk1|` + `` (`tcf` / permissions for that source** (slots are **per source**, not per permission·source — the result string carries every permission, as the vector shows; timestamps integer epoch-ms where present); evidence -digest = SHA-256 `tsevd1|` over the same plus `|lu=`. +digest = SHA-256 `tsevd1|` over the same, **plus `|lu=` +only for sources with an intrinsic authoritative timestamp (TCF); +timestamp-less sources (GPP, USP) omit the `|lu=` field entirely — +omission is the canonical form, no sentinel value exists**. Vectors: `tsstk1|tcf|p1=grant,p4=refuse` → `a49148a0e3b486fd93a141404857868871319a7e1f2ef85b5499aed80c7e59df`; `tsevd1|tcf|p1=grant,p4=refuse|lu=1690000000000` → -`67259c0247ae2b33c52d9f18193bcd622f48ad4754ffbab6998d7c293b0143b4`: keying slots on the +`67259c0247ae2b33c52d9f18193bcd622f48ad4754ffbab6998d7c293b0143b4`; +timestamp-less form `tsevd1|gpp|p1=grant,p4=refuse` → +`89b08580c214070b6d1d58ad57c12bd585134f324c718cbc0802b4d82a0d72e6`: keying slots on the evidence digest would give every renewal a fresh key and make "updates its slot in place" impossible, the incompatibility an earlier draft shipped. A slot stores its `state_key`, the current evidence -digest, that digest's pinned first-seen, and the newest authoritative -timestamp observed; a TCF renewal (same `state_key`, newer +digest, that digest's pinned first-seen, the newest authoritative +timestamp observed, **and a per-permission `observed_at` map** (bounded +by the enforced permission set): a slot update refreshes `observed_at` +only for permissions whose §4.5 semantic token actually changed against +the stored vector — unchanged permissions keep theirs, which is what +makes the permission spec's per-permission equality digests real (a +P4-only change must never refresh P1's age) without per-permission +slots; a TCF renewal (same `state_key`, newer `LastUpdated`) updates the slot in place, while a replay (not newer) changes nothing — replay protection derives from recency comparison, not per-value history, so no per-digest sublists are needed. 16 slots @@ -793,10 +831,26 @@ replays and later overflow values neither advance nor extend it (the unpinned version let repetition renew denial forever), and pinning to the epoch's entry instead would have back-dated a genuinely new opt-out and expired it early. Later restrictive overflows within the epoch -inherit the marker — a **bounded shortening** (at most the gap between -first and later overflow) declared in sign-off 31, -recovery is automatic at -epoch expiry as slots free, and saturation is a first-class metric — +inherit the marker — the **declared saturation exception** (sign-offs +16 and 31 both carry it): a genuine opt-out arriving as a restrictive +overflow late in the epoch receives **less than its §4.3 full-TTL +lifetime, down to nearly zero at the epoch's end**. This is a product +choice, not an accident: per-overflow state is exactly what saturation +exists to avoid storing (unbounded slots), and refreshing the marker on +later overflows would let replays of evicted values extend denial +indefinitely (the anti-replay bound) — so the shortening is accepted, +bounded to sources presenting ≥ 16 distinct in-horizon states, and +ratified explicitly. **Epoch expiry is a complete transition, not a +hope**: at `saturated_until` the epoch ends; expired slots are lazily +garbage-collected on the next write; if a slot is then free, the record +leaves saturation and novel values occupy slots normally, each with its +own full TTL; if every slot still holds an unexpired state — TCF +renewals legitimately extend slots past `saturated_until`, since slots +are individually lived and the epoch bounds only untracked overflow — +the next novel value opens a **new epoch** whose restrictive marker +pins to its own first restrictive overflow: epochs never chain or +inherit timestamps across their boundary. Saturation is a first-class +metric — the cap and its denial behavior are **sign-off item 31**; record level: family ID, CAS version counter, schema version, stub marker (backfill, §5); unknown-field and range validation apply like every class (strong class, permission-exempt writes per the @@ -819,26 +873,26 @@ readers round-trip unknown keys **semantically** (values preserved through read-modify-write; byte-identical output is not required and not achievable through a structured serializer). -| Field | Purpose | Source | Gating permission (egress) | TTL / refresh | Rewrite | On revocation | -| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | -| key (v1: identifier verbatim; v2: core-constructed, §4) | Row identity | Provider/core | — | Row TTL (1 y today) | New canonical row; old key becomes alias | Family record governs; member tombstone as cleanup | -| `v` | Schema discriminator | Core | — | — | Written at current version | Retained | -| `created` / **`expires_at`** | Row age and the **absolute retention deadline, pinned at mint** — every update writes with the _remaining_ lifetime, never a fresh full TTL (today's full-TTL rewrite lets a frequently visited identity live forever; refreshable derived state must never rejuvenate the identity) | Core | P1 (first-party ops) | Never extended | Preserved (no rejuvenation) | Retained in tombstone | -| `consent.tcf` / `consent.gpp` | Raw signal snapshot for audit; superseded as authority by provenance | Request | Never egressed to partners | Replaced on live resolution (§7 snapshot rule, permission spec) | Fresh live values | Scrubbed | -| `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | -| New: **immutable mint tag** (`mint_provider`, `mint_version`) | Credential retirement and audit — write-once at mint (legacy backfill may populate a missing tag once); **never part of the replaceable snapshot**, or a v1 identity revisited after rotation would be restamped v2 | Mint (or one-time backfill) | — | Immutable | — | Retained | -| New: **provenance revision** (application-level monotonic counter, u64, initialized at 1, incremented by every provenance-bearing write, CAS'd with the row generation, serialized as an integer; overflow is a hard error, not a wrap) | Orders clears vs. snapshots (permission spec §4.3) | Core | Read by S2S/clears | Monotonic | — | Retained | -| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision) | **Audit mirror only — never the S2S decision input**: the strong summary carries every decision field (jurisdiction included); the row supplies identity/partner data only after the exact revision fence | Live resolution only | Read by S2S recompute | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | -| New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | -| `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | -| `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | -| `pub_properties` (origin/seen domains) | Creation context | Core at mint | P1 | Write-once | Preserved | Scrubbed | -| `device.*` (JA4 class, H2 hash, quality metadata) | **Discontinued for new rows** (§5): fingerprint-derived, buyer-facing — beyond security-classification authorization. v1 rows retain them read-only; they are never egressed post-epic and are dropped at rewrite | Fastly device provider | None grants egress | Write-once (v1) | **Dropped** | Scrubbed | -| New: security classification outcome (boolean) | Bot-gate result | Device provider | — (never egressed) | Written at mint | Fresh | Scrubbed | -| `network.*` (immutable evidence: ASN etc.) | Cluster disambiguation | Platform at mint | P1 | Write-once | Fresh | Scrubbed | -| Derived cluster state (`cluster_size`, computed-at) | Trust gating | Computed | — | **Refreshable, short validity; generation-CAS update; never touches `expires_at`** | Recomputed | Scrubbed | -| `ids` (partner → UID map) | Partner identity graph | Pixel/pull/batch sync | P1 ∧ P4 (partner egress) | Per-mapping timestamps; bounded count/length | Copied **with original timestamps/expiry** | Scrubbed | -| New: alias record kind | Rewrite indirection (§6.1) | Core | — | Retirement deadline | Is the mechanism | Family-revoked like any member | +| Field | Purpose | Source | Gating permission (egress) | TTL / refresh | Rewrite | On revocation | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | +| key (v1: identifier verbatim; v2: core-constructed, §4) | Row identity | Provider/core | — | Row TTL (1 y today) | New canonical row; old key becomes alias | Family record governs; member tombstone as cleanup | +| `v` | Schema discriminator | Core | — | — | Written at current version | Retained | +| `created` / **`expires_at`** | Row age and the **absolute retention deadline, pinned at mint** — every update writes with the _remaining_ lifetime, never a fresh full TTL (today's full-TTL rewrite lets a frequently visited identity live forever; refreshable derived state must never rejuvenate the identity) | Core | P1 (first-party ops) | Never extended | Preserved (no rejuvenation) | Retained in tombstone | +| `consent.tcf` / `consent.gpp` | Raw signal snapshot for audit; superseded as authority by provenance | Request | Never egressed to partners | Replaced on live resolution (§7 snapshot rule, permission spec) | Fresh live values | Scrubbed | +| `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | +| New: **immutable mint tag** (`mint_provider`, `mint_version`) | Credential retirement and audit — write-once at mint (legacy backfill may populate a missing tag once); **never part of the replaceable snapshot**, or a v1 identity revisited after rotation would be restamped v2 | Mint (or one-time backfill) | — | Immutable | — | Retained | +| New: **provenance revision** (application-level monotonic counter, u64, initialized at 1, incremented by every provenance-bearing write, CAS'd with the row generation, serialized as an integer; overflow is a hard error, not a wrap) | Orders clears vs. snapshots (permission spec §4.3) | Core | Read by S2S/clears | Monotonic | — | Retained | +| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision) | **Audit mirror only — never the S2S decision input**: the strong summary carries every decision field (jurisdiction included); the row supplies identity/partner data only after the exact revision fence | Live resolution only | Not read for gating — audit and cleanup only (S2S reads the strong summary alone) | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | +| New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | +| `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | +| `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | +| `pub_properties` (origin/seen domains) | Creation context | Core at mint | P1 | Write-once | Preserved | Scrubbed | +| `device.*` (JA4 class, H2 hash, quality metadata) | **Discontinued for new rows** (§5): fingerprint-derived, buyer-facing — beyond security-classification authorization. v1 rows retain them read-only; they are never egressed post-epic and are dropped at rewrite | Fastly device provider | None grants egress | Write-once (v1) | **Dropped** | Scrubbed | +| New: security classification outcome (boolean) | Bot-gate result | Device provider | — (never egressed) | Written at mint | Fresh | Scrubbed | +| `network.*` (immutable evidence: ASN etc.) | Cluster disambiguation | Platform at mint | P1 | Write-once | Fresh | Scrubbed | +| Derived cluster state (`cluster_size`, computed-at) | Trust gating | Computed | — | **Refreshable, short validity; generation-CAS update; never touches `expires_at`** | Recomputed | Scrubbed | +| `ids` (partner → UID map) | Partner identity graph | Pixel/pull/batch sync | P1 ∧ P4 (partner egress) | Per-mapping timestamps; bounded count/length | Copied **with original timestamps/expiry** | Scrubbed | +| New: alias record kind | Rewrite indirection (§6.1) | Core | — | Retirement deadline | Is the mechanism | Family-revoked like any member | ## 7. Composition root and adapter parity @@ -869,7 +923,7 @@ Requirements: | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | | Deployment metadata — schema floor (write-once/CAS); **graphless flag additionally requires globally observable strong reads** (N+2 lease revalidation and N+1's `not_before` barrier both need globally current reads, not just write-once) | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | - | Rowless prefix-withdrawal (`w`) records | **Globally strong reads + linearizable CAS** (same bar as authority-state), plus the **bounded listing-visibility window** the backfill scan depends on — all three are eligibility gates for the graphless migration; retention ≥ maximum cookie lifetime | + | Rowless prefix-withdrawal (`w`) records | **Globally strong reads + linearizable CAS** (same bar as authority-state), plus the **bounded listing-visibility window** the backfill scan depends on — all three are eligibility gates for the graphless migration. The strong-read obligation is **permanent for HMAC row discovery** — enforcement runs to the last entry's `valid_until`, long after the flag clears — and retention runs through the **max(cookie, row, S2S) horizon of each entry** (an earlier "cookie lifetime" cell contradicted §6.3's per-entry horizons) | | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index a1b0aad54..07992a0b3 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -128,7 +128,11 @@ Requirements: treat revoked identities as live (aliases are reserved-future with the rewrite deferral, providers spec §6.1). N+1 must also **write family revocation records** — a withdrawal arriving on a - rolled-back N+1 fleet must still revoke. **Authority-state: N+1 may create + rolled-back N+1 fleet must still revoke. These negative gates are + **config-shape- and graphless-flag-invariant** (providers spec §5 + total state table): no config shape, flag state, or v1-semantics + row disables reading and enforcing family revocations, suppression + entries, or live `w` records. **Authority-state: N+1 may create stubs and write negative entries, but never positive commits or clears.** The observed-row admission sequences (providers spec §5) need an s-class stub before revoking an untouched v1 row and a @@ -154,8 +158,10 @@ Requirements: **old-shape config on N+1 runs the pre-epic consent gate unchanged** — dual-read means dual-behavior — so the compiled protective fallback cannot flip behavior mid-convergence before the - operator pushes the new-shape policy; the new model engages only - with new-shape config. The interim is declared as sign-off item 20 — with one + operator pushes the new-shape policy; the new model's **live gating** + engages only with the N+2 writer _and_ new-shape config together — + on N+1, new-shape config engages validation, telemetry, and the + batch fail-closed boundary below, never live-request gating. The interim is declared as sign-off item 20 — with one boundary that does **not** wait for N+2: once new-shape config is active, **context-free partner egress (batch sync) on N+1 fails closed for rows without provenance**, exactly as the permission @@ -165,6 +171,19 @@ Requirements: later than the model it protects. Live-request paths keep v1 semantics until N+2. + The interim is **one matrix, not competing prose** — per release × + config shape, each dimension separately (a new-shape P1 denial on + N+1 has exactly one meaning: telemetry, never gating): + + | Dimension | N+1, old shape | N+1, new shape | N+2, new shape | + | ----------------------------------------------------------- | ---------------------- | --------------------------------------------------------- | --------------- | + | Policy resolution | not parsed | parsed, validated, logged — **never gates live requests** | gates | + | Live-request permission gating | pre-epic gate | pre-epic gate (a new-shape denial is telemetry only) | new model | + | Negative gates (revocation, suppression, `w`), read + write | **active** (invariant) | **active** (invariant) | active | + | Identity-row writes | v1 rows | v1 rows | v2 + provenance | + | Positive authority commits / clears | forbidden | forbidden | N+2 writer | + | Context-free batch (S2S) egress | pre-epic row check | **fails closed for provenance-less rows** | full recompute | + Rollback tests therefore mirror the one contract exactly: family-revocation read **and write**; authority-state **stub creation and negative suppression entries, read and write** (the @@ -215,7 +234,11 @@ Requirements: together with **filling the PSL snapshot reference** (`psl-snapshot-ref.md` is a placeholder; ratification cannot reproduce the cookie-domain computation it approves until the - vendored commit is recorded) and creating the §8 decision records. + vendored commit is recorded), **filling the GPP registry snapshot** + (`gpp-registry-snapshot.md` equally lacks its immutable registry + commit and per-section conformance vectors; ratifying §4.5 field + mappings that cannot be reproduced against a pinned registry is the + same defect), and creating the §8 decision records. 3. **Revocation-eligible storage is a per-adapter gate, and ungated adapters migrate stateless.** Identity features require the adapter's strong-consistency rows in the capability matrix (providers spec §7) @@ -506,7 +529,7 @@ implemented. | 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | — | open | | 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | — | open | | 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | — | open | -| 16 | Timestamp-less opt-out suppression is **TTL-sticky**: within its consent-TTL lifetime only a newer timestamped grant clears it; at `valid_until` it goes inert automatically (not user-sticky-forever, not administratively sticky — administrative clear is an optional early exit) | permission §4.3 | — | open | +| 16 | Timestamp-less opt-out suppression is **TTL-sticky**: within its consent-TTL lifetime only a newer timestamped grant clears it; at `valid_until` it goes inert automatically (not user-sticky-forever, not administratively sticky — administrative clear is an optional early exit) — **with the declared saturation exception**: an opt-out arriving as a restrictive overflow during a saturation epoch inherits the epoch marker's earlier `valid_until` and may get less than a full lifetime (providers wire schema; ratified here and in row 31) | permission §4.3 | — | open | | 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | — | open | | 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | — | open | | 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | — | open | @@ -518,8 +541,8 @@ implemented. | 25 | Batch-sync stored-jurisdiction maximum age (consent-TTL horizon): a mover into GDPR stops old-rule egress at the horizon; a mover out is denied until a live visit | permission §7 | — | open | | 26 | Embedded GPP GPC maps to the destructive global opt-out (header-OR-embedded aggregation) | permission §4.5 | — | open | | 27 | Proxy-mode minimal opt-out extraction (decode only §4.5-mapped opt-out fields; no grants) | permission §4.4 | — | open | -| 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance — including CSP interaction with vendor challenge pages, same-origin vendor code on the publisher origin (or an origin-isolation/sandboxing requirement), and the fail-open consequence of batch invalidation | hook §4a; `datadome-header-allowlist.md` | — | open | +| 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance — including CSP interaction with vendor challenge pages, same-origin vendor code on the publisher origin (or an origin-isolation/sandboxing requirement), the **specific `X-DD-B` divergence** (DataDome's documented cookie-mode allow example directs `Set-Cookie` **and** `X-DD-B` to the client, while TS drops `X-DD-B` per the allowlist matrix — vendor acceptance must name that exact field), and the fail-open consequence of batch invalidation | hook §4a; `datadome-header-allowlist.md` | — | open | | 29 | Unverifiable roaming rowless cookies receive best-effort cookie-only expiry (admission rules forbid durable records for unverified values); a lost response can leave the cookie usable on the old network until re-presented | providers §5 | — | open | | 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, observability, **and the saturation collateral: under a saturated prefix, any real row (listed or overflow) is denied and revoked immediately on surfacing — a non-abuser NAT-cohort row can be revoked; `w` is retained through the max(cookie, row, S2S) horizon and consulted by `valid_until`, not the flag, so this is deterministic, not a retention accident** — all in scope | providers §5 | — | open | -| 31 | Replay-history capacity (16 per-source semantic-state slots + a saturation epoch whose restrictive marker is pinned at the **first** restrictive overflow with its own full TTL): while saturated, fresh consent cannot grant until the epoch expires, and **later restrictive overflows inherit the first marker — a bounded shortening of their lifetime** | permission §4.3; providers wire schema | — | open | +| 31 | Replay-history capacity (16 per-source semantic-state slots + a saturation epoch whose restrictive marker is pinned at the **first** restrictive overflow with its own full TTL): while saturated, fresh consent cannot grant until the epoch expires, and **later restrictive overflows inherit the first marker — a shortening of their lifetime, down to nearly zero late in the epoch**; ratification chooses this knowingly — the alternatives (per-overflow state; refreshing the marker on later overflows) were rejected for unbounded storage and replay-extension respectively | permission §4.3; providers wire schema | — | open | | 32 | GPP sections 24–27 (MD/IN/KY/RI) are reserved with **national-section-only** handling until an official binary layout can be vendored — a state-specific opt-out expressed only in an undecodable state section is not honored | permission §4.5; `gpp-registry-snapshot.md` | — | open | diff --git a/docs/superpowers/specs/datadome-header-allowlist.md b/docs/superpowers/specs/datadome-header-allowlist.md index 63824bbd2..2cc9db018 100644 --- a/docs/superpowers/specs/datadome-header-allowlist.md +++ b/docs/superpowers/specs/datadome-header-allowlist.md @@ -1,56 +1,48 @@ # DataDome header allowlist (normative, checked-in — request and response directions) -The complete set of response-named header pointers the security channel -(hook spec §4a) may copy into the owner-scoped publisher-upstream -overlay. Every `X-DataDome-*` name not listed here is rejected. Adding a -name is a reviewed commit to this file and a spec change. +Adding or changing any name here is a reviewed commit to this file and +a spec change. This file holds the **only** normative pointer lists; +the hook spec §4a references it and carries no duplicate or +per-decision lists of its own. + +## Request-direction pointer (vendor response → publisher-upstream overlay) + +The complete set of vendor-response header pointers the security +channel (hook spec §4a) may copy into the owner-scoped +publisher-upstream overlay. Every `X-DataDome-*` name not listed here +is rejected. | Header | Direction | Scope | | --------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------- | | `X-DataDome-ClientID` | response → upstream overlay | Owner-scoped overlay only; never the shared request view; vendor egress governed by sign-off item 23 | -## Response-direction allowlist (browser-response pointers) - -Headers a DataDome decision may set on the outgoing response, beyond -the typed security cookie (hook spec §4a). Empty rows below the base -set mean: nothing else is accepted until a reviewed commit adds it. - -| Header | Decision | Semantics | -| ------------------------- | ---------------------------- | ------------------------------------------- | -| `Location` | Respond (3xx) only | replace | -| `Content-Type` | Respond only (owns its body) | replace | -| `Cache-Control`, `Pragma` | Respond only | restricted merge; invariant pass still last | - -Note: the vendor's `X-Set-Cookie` response field is **not** a -forwardable header — it lowers into the typed `datadome` cookie -operation (hook spec §4a) and never reaches the browser as a header. - ## The single pointer matrix (normative — decision × session mode × pointer) -This is the one authoritative contract; the hook spec §4a references it -and carries no duplicate lists. Session mode is **cookie** in v1 -(sessionByHeader is startup-rejected; a header-mode column is added by -the sign-off-23 opt-in, never implicitly). No wildcard rows exist — -every accepted name is enumerated. - -| Pointer | Respond (cookie mode) | Continue (cookie mode) | -| ------------------- | ---------------------------------------------------------------------------- | ------------------------------------- | -| `Set-Cookie` | typed `datadome` cookie operation | typed `datadome` cookie operation | -| `X-Set-Cookie` | unclassified → batch handling (v1: sessionByHeader rejected) | unclassified → batch handling | -| `Location` | forward (3xx only), replace | rejected | -| `Content-Type` | forward (owns its body), replace | rejected | -| `Cache-Control` | restricted merge; invariant pass last | rejected | -| `Pragma` | drop-individually, logged | drop-individually, logged | -| `X-DataDome` | forward, owner-scoped typed telemetry | forward, owner-scoped typed telemetry | -| `X-DD-B` | drop-individually, logged (header-session artifact; harmless in cookie mode) | drop-individually, logged | -| anything not listed | invalidate the batch → Continue | invalidate → effects dropped | +This is the one authoritative browser-response contract. Session mode +is **cookie** in v1 (sessionByHeader is startup-rejected; a header-mode +column is added by the sign-off-23 opt-in, never implicitly). No +wildcard rows exist — every accepted name is enumerated, and **every +cell terminates in exactly one outcome**. + +| Pointer | Respond (cookie mode) | Continue (cookie mode) | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ | +| `Set-Cookie` | typed `datadome` cookie operation (hook spec §4a parser; exactly one `datadome` field) | typed `datadome` cookie operation | +| `X-Set-Cookie` | **invalidate the batch → Continue** (mode mismatch: TS never requests header sessions in v1; a vendor response asserting one must not half-apply) | **invalidate the batch** → effects dropped | +| `Location` | forward (3xx only), replace | rejected | +| `Content-Type` | forward (owns its body), replace | rejected | +| `Cache-Control` | restricted merge; invariant pass last | rejected | +| `Pragma` | drop-individually, logged (response `Pragma` has no standardized meaning, RFC 9111 §5.4) | drop-individually, logged | +| `X-DataDome` | forward, owner-scoped typed telemetry | forward, owner-scoped typed telemetry | +| `X-DD-B` | drop-individually, logged (header-session artifact; the vendor's cookie-mode allow example emits it — the drop is the named divergence in sign-off 28) | drop-individually, logged | +| anything not listed | invalidate the batch → Continue | invalidate → effects dropped | + +Singleton-field multiplicity (`Location`, `Content-Type`, `X-DataDome`, +`X-DD-B`, `X-Set-Cookie` appearing more than once) invalidates the +batch atomically; list-valued fields (`Cache-Control`, `Pragma`) join +per RFC 9110 §5.3 before their cell applies (hook spec §4a). **Fixtures**: DataDome's documented challenge response (`Set-Cookie`, `Pragma`, `X-DataDome`, `Cache-Control`) asserts the decision stays **Respond** with exactly the mapped fields; the documented allow example (`Set-Cookie X-DD-B`) asserts Continue proceeds with the cookie applied and `X-DD-B` dropped-and-logged — neither fixture may fail open. - -Note: the vendor's `X-Set-Cookie` response field is never forwarded as a -header; in v1 it is unclassified because sessionByHeader is rejected at -startup (hook spec §4a). diff --git a/docs/superpowers/specs/gpp-registry-snapshot.md b/docs/superpowers/specs/gpp-registry-snapshot.md index 38fe85ebb..2a9eff551 100644 --- a/docs/superpowers/specs/gpp-registry-snapshot.md +++ b/docs/superpowers/specs/gpp-registry-snapshot.md @@ -56,3 +56,9 @@ Supported sections (6–23) pin to the official IAB GPP registry revision recorded by the implementation PR (immutable upstream commit hash), with per-section encoded conformance vectors vendored alongside. A date is not a revision; the commit hash is the reproducible authority. + +**Status: placeholder until ratification.** Neither the immutable +registry commit nor the conformance vectors are recorded yet; like the +PSL snapshot, filling them is a pre-ratification prerequisite +(migration spec §7) — the §4.5 field mappings cannot be reproduced +against a pinned registry until they land. diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index fd9848b40..9c6c62c66 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -402,9 +402,13 @@ state_key `tsstk1|tcf|p1=grant,p4=refuse` → X-Set-Cookie unclassified, incoming header ClientIDs not forwarded); the R16 cookie-translation is retracted as not equivalent; the old DataDome spec's banner now supersedes its header-mode requirement. -- One pointer matrix (decision × session mode × pointer) lives in the +- One pointer matrix (decision × session mode × pointer) added to the allowlist file with X-DD-B enumerated and no wildcard; both documented - vendor responses are decision-asserting fixtures. + vendor responses made decision-asserting fixtures. **Correction + (R19): text-added, not verified-closed** — R19 found the predecessor + tables (request-note, response-direction table) and the hook's inline + decision list still present and contradictory; they were physically + removed in R19. - Browser trust boundary ratified, not implied: non-HttpOnly cookie readable by every same-origin script, vendor challenge HTML with publisher-origin access, CSP interaction — sign-offs 23/28 rewritten. @@ -421,9 +425,126 @@ state_key `tsstk1|tcf|p1=grant,p4=refuse` → complete-response deadline 3000 ms monotonic (1500 ms stays first-byte), encoded responses batch-invalid; total cookie parser (repeated Cookie joined, Set-Cookie never combined, strict attribute - rejection, 512-byte serialized measure); adapter header ceilings are - enumerated capability cells validated against core's budget at - startup; GPP/PSL placeholder status stated honestly here and above. + rejection, 512-byte serialized measure); adapter header ceilings were claimed as + enumerated capability cells validated at startup — **correction + (R19): overstated; the sentence existed, the matrix did not** (added + in R19 with per-adapter cells and the exact counting rule); GPP/PSL placeholder status stated honestly here and above. - P3s: §5.5 grammar dangler repaired with the section rebuild; the remaining "and the and a" occurrence fixed; the duplicated Set-Cookie-reserved sentence deduplicated. + +## Round 19 (at 1a15575a4) + +12 P1 (reviewer numbered 11 with one two-part item), 14 P2, 6 P3. All +addressed; dispositions below use the text-added vs verified-closed +vocabulary. Two R18 closure claims above were corrected as overstated +(pointer-matrix single-sourcing; ceiling cells). + +P1 dispositions: + +- v1/N+1 bypass of negative gates: the total state table now opens with + "negative gates are release-, config-, and flag-invariant" and splits + the v1 catch-all into three rows (positive relaxation; revocation or + suppression present → denied; live `w` → withdrawn) — the + rollback-to-N+1 reproducer lands on the `w` row, not the catch-all. + The migration rollback contract cross-references the invariance. +- Missing row-backed and legacy rows: the table adds "normal use in + every flag state" (row-backed operation unaffected by + active/suspended) and a stub-only row that routes to the live + `AuthorityRefresh` backfill; the permission spec's legacy paragraph + states that path is reachable because AuthorityRefresh admits on the + observed row + live resolution, never a prior summary. +- Error row suppressing admitted destructive withdrawal: the default is + now "no writes whose admission depended on the failed read" — with + strong family admission already proven, family revocation and + suppression CAS proceed; only the cookie expiry waits. +- Shared-clock suspension deadline: store-time branch re-reads store + current time at every check (one clock domain); the fallback defines + S_fleet as a declared, monitored maximum pairwise fleet skew — a + separate qualification from the evidence tolerance S — subtracted + again at comparison; clock domain serialized in the metadata value; + fastest-observer/slowest-committer tests named; a deployment with + neither store time nor a skew bound cannot host the migration. +- Activation ordinal: deployment-metadata name `02` is the + policy-activation register — linearizable {source_version, + policy_digest, ordinal} with idempotent same-activation reuse and + stale-source_version rejection; §5.5 rewritten around it; the hook's + "globally assigned push version" wording replaced with the + `tscfg1|` digest + §5.5 pair. +- Jurisdiction expiry: the summary gains `jurisdiction_observed_at`, + written only by live geo resolution; decision 25's age gate measures + against it (evidence timestamps explicitly disqualified as proxies). +- Saturation vs per-signal lifetime: product choice made and declared — + the bounded shortening stands (first-overflow-pinned marker + inherited by later overflows), now stated as a declared exception in + permission §4.3 and carried by decisions 16 AND 31, with the + rejected alternatives (per-overflow state; marker refresh) recorded + in both. +- Pointer-matrix predecessors: the allowlist file was rewritten — the + old response-direction table and both cookie-translation notes are + gone; the hook's inline decision list bullet was replaced by a + deferral to the matrix; the `X-Set-Cookie` cell now terminates in + one exact outcome (invalidate the batch → Continue, mode mismatch) + and the hook's "unclassified → batch handling" phrase was aligned. +- Incoming header ClientID: one v1 path — stripped from the shared + request AND never used for the vendor payload (ClientID derives only + from the `datadome` cookie; forwarding a header ClientID would + require declaring X-DataDome-X-Set-Cookie per the vendor contract); + the "header form wins" priority rule deleted as unreachable; a + both-sources fixture pins cookie-only derivation. +- 304 atomicity: staged off-record diff → (a) byte-coupled field + changed (changed, not present) → invalidate + full 200 before any + serve; (b) safe changes → one atomic cache commit (old-bytes-under- + new-metadata impossible); (c) no change → local-hit re-emit. +- Recovery conditionals: artifact-absence/revision-mismatch recovery + strips every conditional field (client's and internal), processes + the full 200, then evaluates the client condition against the new + processed validator. + +P2 dispositions: N+1 interim reduced to one release × config-shape +matrix (policy resolution / live gating / negative gates / row writes / +positive commits / batch egress); GPP-USP evidence digests omit `|lu=` +entirely with an embedded vector (`tsevd1|gpp|p1=grant,p4=refuse` → +`89b08580…`), and slots gain a per-permission `observed_at` map +refreshed only for changed tokens; saturation epoch expiry is a +complete transition (lazy GC, re-saturation opens a new epoch pinned to +its own first overflow, no cross-epoch inheritance); `w` capability +retention corrected to the per-entry max(cookie, row, S2S) horizon with +a permanent strong-read obligation for HMAC row discovery (and the §6.2 +"migration-window" label fixed); HEAD updates compare against +origin-side metadata only and never touch processed-side headers; the +Vary digest is a full contract (HMAC-SHA-256, `tsvry1|` grammar, comma +join, secret-store key with versioned id, all values digested, zero-key +vector `60fdeb3a…`); the adapter ceiling matrix now exists (Axum fixed +≥ budget; Fastly/Cloudflare/Spin qualification-pending with fail-closed +startup) with the exact counted-bytes formula and over-budget-snapshot +behavior; the cookie parser defines the "; " join, duplicate-datadome +ambiguity (request: cookie-absent + counted; response: batch reject), +non-`datadome` Set-Cookie rejection, HttpOnly rejection, floor-based +Expires→Max-Age conversion, and normalized-form size measurement; +pointed-field multiplicity is closed (singletons atomically +batch-invalid on duplication, list fields joined per RFC 9110 §5.3); +Respond on HEAD validates the body but emits none with recomputed +Content-Length; the 3000 ms deadline has measurement points (pre- +acquisition → final byte; cancellation async, never delaying); malformed +Cache-Control has an enumerated result (uncacheable + merging batch +rejected — "most restrictive reading" deleted); sign-off 28 names the +exact X-DD-B divergence; the GPP snapshot joins the PSL snapshot as a +named pre-ratification gate and the snapshot file itself states its +placeholder status. + +P3 dispositions: the state table's stray separator column and absorbed +prose are gone (table rebuilt); the deployment-metadata row is a single +proper table row with the floor encoding inside its cell; the authority +row's gating column now reads "not read for gating — audit and cleanup +only" and "activation generation" was renamed to the §5.5 ordinal; the +hook's "ratified in sign-offs 23/28" now reads "enter sign-offs 23/28 +for ratification (both records still open)"; the two R18 ledger +overclaims are corrected above; the old DataDome spec's superseded +steps (timeout, post-finalization ordering, X-DataDome-X-Set-Cookie +wire example and rules, implementation clarification 1) each carry an +inline supersession note in place, not only the banner. + +Ratification state is unchanged and user-side: all 32 decision rows +open, `decisions/` empty beyond the README, no adapter qualified, and +the PSL and GPP snapshots are placeholders — all named gates. From 607453ddc76609297a8e4681bd7513ef874c566c Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 4 Aug 2026 16:53:00 -0700 Subject: [PATCH 21/24] Address twentieth review: ordered classification procedure, historical activation register, and collision-free cache identity P1 fixes: - The classification contract is an ordered first-match-wins procedure replacing the five-column table that was neither total nor disjoint: negative gates evaluate first (w, then family revocation, then suppression - the w/revocation overlap has one outcome because promotion onto an already-revoked family is an idempotent no-op), the v1 exception runs after them, and the rowless step keys on the strong-class read's authoritative absence - the old row's 'authoritative not-found' eventual read is named as a state the store cannot produce. The missing states have steps: an absent (not just stub) authority record on a found row routes to the AuthorityRefresh backfill, and nonmatching revisions fail the fence (row data withheld, summary-only posture, live refresh realigns). - The policy-activation register keeps a bounded 16-entry history: exact (source_version, digest) pairs adopt their historical ordinal, closing the interleaved-eviction scenario that minted two ordinals for one activation; the same source_version with a different digest fails closed as mixed-binary parse divergence, never a novel pair; stale and window-expired versions are rejected; the digest-only fallback is deleted - source_version must be ordered and assigned exactly once upstream (backend push version, or the monotonic push sequence the ts config push envelope stamps); and the register is a named capability in both provider matrices. - One S2S authority source, verified this time: the provider-switching paragraph claiming S2S 'recomputes from' row provenance is deleted (audit-mirrored only), and the permission spec's summary enumeration declares itself a reference to the one normative schema (providers 6.3 wire record) with jurisdiction_observed_at included. - The restrictive marker is marker-scoped, not epoch-scoped, in lifetime: it lives to its own valid_until, outliving the epoch when the first overflow arrived late (discarding it at saturated_until would deny that overflow its promised TTL); the record holds at most one live marker per source so overlap never needs representing; later overflows inherit it across epoch boundaries (the anti-replay rule and the declared shortening); a fresh marker pins only after expiry; and 'novel values cannot grant' is narrowed to the epoch. - The Vary digest grammar is collision-free: presence, instance count, and length-prefixed member octets (absent no longer hashes like present-but-empty, per RFC 9111 4.1; ['a','b'] no longer collides with 'a,b'), with new zero-key vectors for the present and absent forms. - Origin-304 're-derived finals' has a defined derivation: replay the persisted mutation IR (the accepted operation batches, whose append/replace/merge semantics are deterministic core functions) over the updated origin metadata plus the invariant pass - mutators never re-run; IR-less entries force a full refetch; publication is one atomic entry commit, with changed-Vary rekeying ordered insert-new-entry-then-update-index so torn states miss; both are adapter capability cells. P2 fixes: the per-source replay slot has a defined transactional algorithm (current_state_key pointer; comparison against the current slot's vector, never the revisited slot's; per-permission copy-forward; full map replacement on A-B-A; named vectors); malformed/absence suppressions gain a narrowly scoped recovery-observation clearing rule (a valid grant presented after the suppression clears it without refreshing the grant's pinned age; opt-out stickiness untouched); the suspension named test states the branch-appropriate comparison and the store-clock/S_fleet branches are capability cells; every DataDome matrix cell is terminal (non-3xx Respond Location invalidates the batch; Continue Location/Content-Type/Cache-Control invalidate with effects dropped); Respond on HEAD omits Content-Length (RFC 9110 9.3.2 - HEAD bytes cannot establish the GET length absent a vendor equivalence guarantee); the fleet Vary key is a deployable contract (setting name, 32-byte CSPRNG floor, key-id grammar, fail-closed startup, overlap rotation, capability gate); and the 304 row counts four revisions, evaluates the complete RFC 9110 section 13 precondition set after recovery (If-Match/If-Unmodified-Since can yield 412), and states the safe-update field set once as the 'cache-relevant fields' definition. P3 fixes: the response-eligibility table is structurally valid again (the unescaped tscfg1 pipe had split the 304 row into a third column; the row is rebuilt with zero content pipes and the separator normalized); the GPP snapshot cites migration section 4; the ledger corrects three prior overclaims as text-added-not-verified (R18 S2S sole-source, R19 Vary full-contract, R19 no-cross-epoch-inheritance) and adds the round-20 section; the broken _classification_ emphasis is repaired; and cookie citations reference RFC 10025 (obsoleting RFC 6265). --- ...integration-response-header-hook-design.md | 73 ++++-- .../2026-07-30-permission-model-design.md | 84 ++++-- .../2026-07-30-pluggable-providers-design.md | 247 ++++++++++++------ ...07-30-provider-migration-rollout-design.md | 2 +- .../specs/datadome-header-allowlist.md | 22 +- .../specs/gpp-registry-snapshot.md | 2 +- docs/superpowers/specs/pr986-review-ledger.md | 91 ++++++- 7 files changed, 367 insertions(+), 154 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 8f54ae87a..6a0a5e94e 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -113,16 +113,30 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may collapse personalized variants). **Every** `Vary`-nominated request value is stored **only as a keyed digest** — every value, not a sensitivity classification an unknown credential field could slip past: HMAC-SHA-256 with domain tag - `tsvry1|`, input = `tsvry1|` + lowercased field name + `|` + the - value octets (repeated members of one field joined with a single - comma in received order before hashing; an absent field hashes the - fixed empty-value form `tsvry1||`), keyed by a fleet-stable key - from the platform secret store with the key id versioned into the - cache entry (rotation = introduce a new id; entries under old ids - simply miss and refill), output lowercase hex (64 chars). - Known-answer vector under the all-zero 32-byte test key: - `tsvry1|authorization|Bearer abc` → - `60fdeb3a933d038ba9dc29a860dc4b2f8c200a0a82f4c3842fc68af62f37589b`. And + `tsvry1|`, over an input that encodes **presence, instance count, and + length-prefixed octets** — the earlier comma-join grammar had + deterministic collisions (absent hashed identically to + present-but-empty, violating RFC 9111 §4.1's absence-matches-only- + absence, and two members `a`,`b` collided with one member `a,b`), + which could select a representation built under a different + credential or tenant. Absent field → `tsvry1||a`; present → + `tsvry1||p|` then, per member in received order, + `|:` with `len` the ASCII-decimal byte count. Output + lowercase hex (64 chars). **The key is a deployable contract, not an + implementation detail**: the setting + `[cache] vary_digest_key_secret_name` names a platform-secret-store + entry (≥ 32 CSPRNG bytes); key ids match `[a-z0-9-]{1,16}` and + version every cache entry; startup **fails** when response caching is + enabled and the key does not resolve (digests are never computed + unkeyed); rotation introduces a new id while the previous stays + resolvable — entries under any resolvable id keep matching, others + miss and refill; the whole requirement is an adapter capability and + startup gate, since every caching adapter needs it. Known-answer + vectors under the all-zero 32-byte test key: + `tsvry1|authorization|p|1|10:Bearer abc` → + `c880c5e8c36febc0b1581c92f1d598fded34391626e67372ed63b2857d8a7b6b`; + absent-field form `tsvry1|x-tenant|a` → + `a2ae26cf529a5843a25f1448acc4e90016d4c1dce0ffda5662e3ac459433e1ab`. And a response derived from a request carrying an **identity-bearing TS overlay** (the DataDome ClientID overlay) is forced `private, no-store` unless an explicit per-overlay contract says otherwise — @@ -297,17 +311,17 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set** (the cache-relevant fields) was persisted alongside the representation, **versioned by a fleet-stable tuple** — the integration-registry revision is a content hash over the ordered (integration ID, version) list; the config revision is the `tscfg1 | `effective-config digest, and the policy revision is the (digest, activation-ordinal) pair from the permission spec's §5.5 register — the earlier "config store's globally assigned push version" definition is superseded (not every adapter has one; the register orders activations for all of them); the invariant revision is a build-time spec-version constant — local counters would collide across instances and binaries; there are **two distinct 304 cases**. A **locally generated conditional hit** (TS answers the client's`If-\*`from its own fresh stored artifact) re-emits the persisted finals when all three revisions match, else cache-miss — as before. An **origin-revalidation 304** (TS revalidated upstream and the origin returned 304 with possibly new`Cache-Control`/`Vary`/`Expires`/validators) is different — RFC 9111 §4.3.4's update-then-serve is implemented **staged, then atomic, never in place**: TS stages the 304's metadata off-record and diffs it against the separately stored **origin-side** metadata (origin validators/`Content-Length`describe origin bytes, not the rewritten HTML artifact — origin-side and processed-side metadata are stored separately). (a) If any **byte-coupled representation field changed** —`Content-Encoding`, `Content-Type`, validators, digests; **changed, not merely present**, since an ordinary 304 routinely repeats the matching validator — nothing is published: the stored entry is invalidated and a full 200 is fetched and processed before any serve (RFC 9111 §3.2 excludes `Content-Length`from stored-response updates and warns against updating transformed artifacts with incompatible representation metadata). (b) If only safe cache-relevant fields changed, the origin-side metadata and the re-derived persisted finals publish in **one atomic cache commit** — a concurrent hit observes the complete old entry or the complete new one, never old transformed bytes under newly published policy metadata (a cached`public`followed by an origin`304 Cache-Control: private, no-store`must not keep serving the old public policy); a changed`Vary` evicts or rekeys the stored index entry inside that same commit. (c) Nothing changed → the persisted finals re-emit as in the local-hit case. Artifact absence or a revision mismatch makes the recovery fetch **unconditional — every conditional field is stripped, the client's (`If-None-Match`, `If-Modified-Since`) and TS's own alike** (forwarding the client's condition could return another 304 TS holds no usable bytes for); TS obtains and processes the full 200 under current revisions, then separately evaluates the client's original condition against the **new processed validator**, answering the client 304 only on a match. "Cache-relevant fields" is defined: the registry-admitted mutable set plus the Cache-Control family and `Vary`; `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | -| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals stored as a **distinct head-only artifact type that never satisfies a later GET** — when a stored GET artifact exists a HEAD may **update** it only when the comparison — made against the separately stored **origin-side** metadata, never the processed artifact's (origin and rewritten lengths differ by construction, so the processed length is never the comparand and HEAD metadata never touches processed-side headers) — finds validators and origin `Content-Length` matching and no byte-coupled representation field changed (RFC 9111 §4.3.5); qualifying updates land origin-side under the same atomic-commit discipline as the 304 rules, and any mismatch **invalidates** the stored GET artifact rather than updating it | -| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | +| Response | Hook runs? | +| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set and the accepted mutation-operation batches that produced it (the persisted mutation IR)** were stored alongside the representation, **versioned by a fleet-stable tuple of four revisions** — the integration-registry revision (content hash over the ordered (integration ID, version) list), the config revision (the `tscfg1`-tagged effective-config digest), the policy revision (the (digest, activation-ordinal) pair from the permission spec's §5.5 register — the earlier "config store's globally assigned push version" definition is superseded), and the build-time invariant revision; local counters would collide across instances and binaries. There are two distinct 304 cases. A **locally generated conditional hit** (TS answers the client's conditional from its own fresh stored artifact) re-emits the persisted finals when **all four revisions** match, else cache-miss. An **origin-revalidation 304** is handled staged, then atomic, never in place: TS stages the 304's metadata off-record and diffs it against the separately stored **origin-side** metadata (origin validators and origin `Content-Length` describe origin bytes, not the rewritten artifact). (a) Any **byte-coupled representation field changed** (`Content-Encoding`, `Content-Type`, validators, digests — **changed, not merely present**: an ordinary 304 repeats the matching validator) → nothing publishes; the entry is invalidated and a full 200 is fetched and processed before any serve (RFC 9111 §3.2). (b) Only fields of the **enumerated safe-update set** changed — exactly: `Cache-Control`, the four enumerated CDN cache fields (under their reserved rules), `Expires`, `Date`, `Age`, `Vary`, and the registry-admitted mutable fields; **this set is the one definition of "cache-relevant fields," stated once** — → the new finals are **derived deterministically without re-running mutators** (mutators may be nondeterministic and run only at fill time): replay the persisted mutation IR — whose append/replace/merge semantics are core-defined deterministic functions of the operations plus the base — over the updated origin metadata, then re-run the invariant pass; an entry lacking its IR (an older cache schema) is unsafe → full refetch. The updated origin-side metadata, re-derived finals, and IR publish in **one atomic entry commit**; a changed `Vary` rekeys by **insert-new-entry-then-update-index ordering**, so a torn state yields a miss, never a wrong hit — single-entry atomic commit and this rekey discipline are adapter capability cells. (c) Nothing changed → re-emit as in the local-hit case. Artifact absence or a revision mismatch makes the recovery fetch **unconditional — every conditional field is stripped, the client's and TS's own alike** (forwarding the client's condition could return another 304 TS holds no usable bytes for); TS processes the full 200 under current revisions, then evaluates the client's **complete precondition set per RFC 9110 §13 against the new processed validators** — a failing `If-Match` or `If-Unmodified-Since` yields `412`, a matching `If-None-Match` or `If-Modified-Since` yields `304`, anything else the full response. `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | +| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals stored as a **distinct head-only artifact type that never satisfies a later GET** — when a stored GET artifact exists a HEAD may **update** it only when the comparison — made against the separately stored **origin-side** metadata, never the processed artifact's (origin and rewritten lengths differ by construction, so the processed length is never the comparand and HEAD metadata never touches processed-side headers) — finds validators and origin `Content-Length` matching and no byte-coupled representation field changed (RFC 9111 §4.3.5); qualifying updates land origin-side under the same atomic-commit discipline as the 304 rules, and any mismatch **invalidates** the stored GET artifact rather than updating it | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). @@ -330,13 +344,13 @@ degree of freedom is closed: `SameSite` configurable `Lax` (default) / `Strict` / `None` (`None` requires `Secure`), matching the vendor's endpoint options; the returned `Domain` must additionally **domain-match the current - request host** per RFC 6265 (domain-match and the PSL boundary check + request host** per RFC 10025, the current cookie specification obsoleting RFC 6265 (domain-match and the PSL boundary check are separate requirements) and a vendor cookie using `Expires` is normalized to its Max-Age equivalent (both present → `Max-Age` wins, - per RFC 6265); a normalized lifetime exceeding the ceiling rejects + per RFC 10025); a normalized lifetime exceeding the ceiling rejects the whole operation batch; and the parser is total: repeated `Cookie` request fields are joined with `"; "` (semicolon-space — the - order-preserving join of the current cookie RFC) before parsing, and + order-preserving join of RFC 10025) before parsing, and **duplicate `datadome` pairs after the join make the request-side identity ambiguous: treated as cookie-absent for the vendor call and counted**, while cookies under other names pass through untouched; @@ -485,9 +499,14 @@ degree of freedom is closed: itself batch-invalid; `Content-Length` is recomputed from the actual bytes before Respond commits. **On a HEAD request, Respond validates the challenge body exactly as for GET (size, deadline, encoding) but - emits no body**: the outward response carries the validated bytes' - `Content-Length` and no content (RFC 9110 HEAD semantics — the older - DataDome spec's HEAD handling is superseded by this rule). Exceeding + emits no body**: the outward response **omits `Content-Length`** and + carries no content — RFC 9110 §9.3.2 permits `Content-Length` on + HEAD only when it equals the equivalent GET body's length, and the + vendor does not guarantee method-invariant challenge bodies, so the + validated HEAD bytes cannot establish the GET length (a + vendor-guaranteed equivalence, if ever ratified, may restore the + field as a reviewed change; the older DataDome spec's HEAD handling + remains superseded). Exceeding size, first-byte, or total deadline fails the batch → Continue. - **One pointer contract, one place.** The single normative decision × session-mode × pointer matrix lives in diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 2d62325ee..345288a03 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -487,17 +487,30 @@ and the fail-closed marker: is superseded; administrative clear remains an optional early exit — sign-off 16 — **with one declared exception**: an opt-out arriving as a restrictive _overflow_ while its source's replay history is - saturated inherits the saturation epoch's first-overflow marker and - may receive less than a full lifetime, down to nearly zero late in - the epoch (providers spec wire schema; the exception is carried by + saturated inherits the live restrictive marker (first-overflow-pinned; + it outlives its epoch and can span epoch boundaries, providers spec + wire schema) and may receive less than a full lifetime, down to + nearly zero near the marker's expiry (the exception is carried by sign-offs 16 and 31, and the alternatives — per-overflow state, marker refresh — were rejected for unbounded storage and replay-extension respectively); **TCF refusal** — cleared by any regime-accepted grant with newer authoritative evidence; **malformed-present / absence** — cleared by any regime-accepted valid grant with newer evidence, including a - timestamp-less grant whose first-seen is newer (these causes are not - user opt-outs, so stickiness does not apply — without this rule, one - truncated request would permanently deny a GPP-only user). Policy + timestamp-less grant whose first-seen is newer, **and — recovery + observation, scoped to these two causes only — by a regime-accepted + valid grant whose _presentation_ is observed after the suppression's + observation timestamp, even when its pinned first-seen is older**: + the common recovery is the unchanged grant re-presented after one + truncated request, whose first-seen predates the malformed event by + construction — first-seen comparison alone would deny until the + suppression's TTL. The scope is what keeps replay harmless: clearing + a malformed/absence suppression asserts only "the CMP currently + emits valid state", which any valid presentation proves; the grant's + own pinned first-seen and expiry are untouched (no age refresh), and + opt-out causes still clear only on strictly newer authoritative + evidence (these causes are not user opt-outs, so stickiness does not + apply — without recovery, one truncated request would deny a + GPP-only user for the suppression's full TTL). Policy changes never clear user-signal suppressions. **Anti-replay for timestamps.** A future-dated record is rejected as @@ -850,24 +863,42 @@ round-trip, defaults materialized, no insignificant whitespace; cross-language vectors required) — identity, so an A→B→A rollback yields A's digest again. The ordinal comes from the **policy-activation register** — deployment-metadata name `02` (providers spec §6.3), a linearizable -`{source_version, policy_digest, ordinal}` value with three transition -rules that make it idempotent and single-source, not merely a counter: -an activation presenting the register's stored -`(source_version, policy_digest)` pair **adopts the stored ordinal -without incrementing** (every instance activating the same push -converges on one ordinal — the CAS winner assigns, everyone else -reuses, so assignment is effectively single-actor); a novel pair -CAS-increments; and an activation whose `source_version` is **older** -than the stored one is rejected as stale (an instance restarting on old -config can neither mint a new ordinal nor regress the register). -`source_version` is the config store's push version where the backend -assigns an ordered one; a backend without one uses the `tscfg1|` -config-revision digest as `source_version`, which cannot detect -staleness — there, a laggard re-activating an older digest mints a new -ordinal, which is **safe** (revision identity is the pair; fleet order -stays correct) but visible in the activation-ordinal metric. Order is -adapter-independent either way (per-instance counters ordered nothing -across a fleet). Authority wire records, the S2S recompute, and the hook cache +register holding the current `{source_version, policy_digest, ordinal}` +**plus a bounded history of the last 16 activations**. Current-value-only +was shown to break activation identity: an interleaved registration +evicted the pair, and a same-push latecomer then minted a second +ordinal for one activation — two `(digest, ordinal)` identities for one +push. Transition rules, evaluated on a strong read + CAS: + +- `(source_version, policy_digest)` **found in the history** → adopt + that entry's ordinal, no increment — idempotent for every instance + of the same activation however late it arrives within the window, so + one activation has exactly one `(digest, ordinal)` fleet-wide. +- `source_version` found in the history with a **different digest** → + **fail closed at startup**: one pushed configuration parsing to two + canonical-policy digests is mixed-binary parse divergence — a hard + incompatibility, never a novel pair, never a new ordinal. +- `source_version` **newer** than every history entry → a new + activation: CAS-append `(source_version, digest, max_ordinal + 1)`, + evicting the oldest history entry. +- `source_version` older than the newest and absent from the history → + **stale, rejected** (an instance restarting on old config can + neither mint an ordinal nor regress the register; a laggard older + than the 16-entry window is also rejected — it must fetch current + config, not activate). + +`source_version` must be an **ordered identifier assigned exactly once +upstream**: the config store's push version where the backend has one, +else the **monotonic push sequence the `ts config push` envelope stamps +into the blob**. The earlier digest-only fallback is **deleted** — a +digest cannot distinguish a deliberate rollback from a stale-instance +restart, and it re-minted ordinals for a single activation. A +deployment with neither ordered identifier is not eligible for +multi-instance policy activation (an adapter capability cell, providers +spec §7). A→B→A remains a **third activation** — new `source_version`, +A's digest, a new ordinal — digest for identity, ordinal for order, +exactly as revision identity requires. Order is adapter-independent +(per-instance counters ordered nothing across a fleet). Authority wire records, the S2S recompute, and the hook cache tuple all use this same pair; the hook's earlier "config-store push version" and any digest-only usage are superseded. The other cache-tuple inputs are likewise domain-separated hashes of effective configuration: @@ -976,7 +1007,10 @@ Consumers of the resolved set in this epic: requests — grant basis (which signal class granted, per permission), the evidence's **authoritative timestamp and `valid_until`** (per evidence class), - resolved jurisdiction, and policy revision — **not** provider/version, + resolved jurisdiction **with `jurisdiction_observed_at`**, and policy + revision (the §5.5 pair) — **this list references the one normative + summary schema, the providers spec §6.3 authority-state wire record; + it is not a second schema** — and **not** provider/version, which lives only in the immutable mint tag, or a post-rotation visit would restamp a v1 identity as v2 (providers spec §6.1). Freshness is a **per-evidence-class contract**, because not diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index 844cd4752..b47bbe450 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -326,7 +326,7 @@ variant**. Therefore: local clock compared against a store-issued deadline could cross it while another instance's N+2 lease was still valid. Where the backend has no store clock, `not_before` = committer time + L + - S*fleet and every local comparison subtracts S_fleet again + S\*fleet and every local comparison subtracts S*fleet again (mint only when `now − S_fleet ≥ not_before`) — **S_fleet is the maximum pairwise fleet clock skew, a declared and monitored infrastructure bound, a separate qualification from the @@ -343,7 +343,7 @@ variant**. Therefore: (fastest-observer and slowest-committer schedules) and suspender-restart schedules are named tests: **every** N+1 instance — the one that suspended, a second that starts and reads an already-suspended state, or one recovering after - the suspender crashed — refuses to mint until `now ≥ not_before` + the suspender crashed — refuses to mint until the branch-appropriate deadline comparison passes (store-clock branch: strong-read `store_now ≥ not_before`; fallback branch: local `now − S_fleet ≥ not_before`) (globally strong read of the epoch). N+2 instances prove `active` at classification time through a **bounded lease ≤ L** (strong read at lease expiry). So suspension is fleet-effective within L, no minting @@ -376,43 +376,92 @@ variant**. Therefore: the runtime matrix, §6.2), so a withdrawn suffix cannot slip into use through the row path. **`w` consultation is keyed on the record's `valid_until`, not the rowless-classification flag** — the flag may clear after one cookie lifetime while `w` is retained through the longer max(cookie, row, S2S) horizon, and a late row must still find a live `w`; enforcement ends only when the `w` record itself expires. - **The total state table** — every (semantics, flag, strong record, - row read, `w`) combination has exactly one outcome; anything not - listed falls to the bolded default. **Negative gates are release-, - config-, and flag-invariant**: family revocations, suppression - entries, and live `w` records are read and enforced in every row of - this table, v1 semantics included — the v1 exception relaxes only the - _positive_ side (provenance, row presence, the new gating model), - never the negative one, or an N+2-written rowless withdrawal would - stop binding the moment the fleet rolled back to N+1 (the rollback - contract's N+1 obligations, migration spec §4.4): - - | Semantics | Flag | Strong records (`r`/`s`) | Row read | Live `w` entry | Outcome | - | ----------------------------------- | -------------- | ----------------------------------- | ------------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | v1 (N+1, or N+2 + old-shape config) | any | no revocation, no live suppression | any | none | **v1 positive behavior: recognized cookies are used per pre-epic rules, rows not required** — the declared v1 exception, not an outage: the pre-epic privacy posture persists until the new model activates (matrix row 14) | - | v1 (N+1, or N+2 + old-shape config) | any | revocation or live suppression | any | any | denied exactly as under the new model — negative gates do not wait for the new model | - | v1 (N+1, or N+2 + old-shape config) | any | any | any | **matching entry or saturated** | withdrawn: denied and promoted per §6.2 — a rolled-back N+1 enforces N+2-written `w` records | - | new model | any | present | found, revisions match | none | normal use — in **every** flag state, active and suspended included: the flag governs rowless _classification_ only, never row-backed operation | - | new model | any | **stub only** (no positive summary) | found (legacy or v1 row) | none | no egress, but **not** a dead end: the live-backfill path applies — a live request resolving a regime-accepted grant runs the permission-exempt `AuthorityRefresh` (admission is the observed row + live resolution, never a prior summary or matching revision), commits the positive summary, and the revision fence then opens use (permission spec §7 legacy rule) | - | new model | absent/cleared | present | **not-found (successful read)** | none | **visibility lag, not absence**: the strong record proves the row committed, so the identity is indeterminate this request — no use, no mint, no expiry (a stale replica must not fork a just-minted identity) | - | new model | absent/cleared | absent | any | none | indeterminate (no rowless classification without the flag) | - | new model | active | absent | authoritative not-found | none | rowless: expire-and-re-mint / withdrawal per §5 | - | new model | suspended | absent | any | none | indeterminate (rowless classification off; re-attestation pending) | - | new model | any | any | found | **matching entry or saturated** | denied, then promoted to family revocation (§6.2) | - | any | any | any | error / `w` read error | — | **default: indeterminate — no use, no mint, no expiry — and no writes whose admission depended on the failed read.** Admitted negative writes still proceed: a successfully read strong authority record proves family admission (§5), so a live destructive signal commits its family revocation (and a non-destructive one its suppression CAS) even when the eventual row read or classification failed — only the browser-cookie expiry waits for the revocation commit. A row read failure must never leave S2S authority live against an already-provable withdrawal | + **The classification decision is an ordered procedure, not a + Cartesian table** — the earlier five-column table was neither total + nor disjoint (overlapping negative rows with different outcomes, a + rowless row keyed on an "authoritative not-found" eventual read the + store cannot produce, and no rows for absent authority records or + nonmatching revisions). The procedure is evaluated top-down, + **first match wins**: disjointness holds by construction (each step + assumes every earlier step did not match) and totality by the final + default. **Negative gates are release-, config-, and + flag-invariant** — steps 2–4 run before the v1 exception and before + every positive path, so the v1 exception relaxes only the _positive_ + side and an N+2-written record still binds a rolled-back N+1 + (migration spec §4.4): + 1. **Any read error** (strong records, `w`, or a consulted row read) + → indeterminate: no use, no mint, no expiry, and no writes whose + admission depended on the failed read. Admitted negative writes + still proceed: a successfully read strong authority record proves + family admission (§5), so a live destructive signal commits its + family revocation (and a non-destructive one its suppression CAS) + even when the eventual row read failed — only the browser-cookie + expiry waits for the commit. A row read failure must never leave + S2S authority live against an already-provable withdrawal. + 2. **Live `w` entry** (matching suffix hash, or saturated prefix) → + withdrawn: denied, and promoted to a family revocation at first + sight of a real row (§6.2). If the family is already revoked the + promotion is an idempotent no-op — `w` and revocation agree on + denial, so their overlap has exactly one outcome. Applies under + every semantics, v1 included. + 3. **Family revocation present** → denied — all semantics, all flag + states. + 4. **Live suppression entry for a permission** → that permission is + denied (identity retained — non-destructive); evaluation + continues below for permissions without a live suppression. + 5. **v1 semantics** (release N+1, or N+2 under old-shape config) → + v1 positive behavior: recognized cookies are used per pre-epic + rules, rows not required — the declared v1 exception, not an + outage: the pre-epic privacy posture persists until the new model + activates (matrix row 14). Steps 2–4 have already run. + 6. **New model, row found:** + - authority record **absent or stub-only** (no positive summary) + → no egress, but not a dead end: the live-backfill path applies + — a live request resolving a regime-accepted grant runs the + permission-exempt `AuthorityRefresh` (admission is the observed + row + live resolution, never a prior summary or matching + revision), commits the positive summary, and the fence then + opens use (permission spec §7 legacy rule); + - positive summary present and + `row.provenance_revision == summary_revision` → **normal use**, + in every flag state — active and suspended included: the flag + governs rowless _classification_ only, never row-backed + operation; + - positive summary present, revisions **do not match** → the + fence fails: the row's identity/partner data is withheld and + the strong summary alone governs (exactly the S2S posture); a + live request re-runs `AuthorityRefresh` to re-commit and + realign the fence, then proceeds. + 7. **New model, row not found** (successful eventual read — which by + itself proves nothing): + - strong records for the derived family **absent on the + authoritative strong-class read** and flag = `active` → + **rowless**: expire-and-re-mint / withdrawal per §5 — the + strong-class absence is the proof; the eventual read is never + the evidence; + - any strong record present → **visibility lag, not absence**: + the record proves a row committed, so the identity is + indeterminate this request — no use, no mint, no expiry (a + stale replica must not fork a just-minted identity); + - flag absent, cleared, or suspended → indeterminate (no rowless + classification without an `active` flag; suspension pauses + classification pending re-attestation). + 8. **Default** — any state not matched above → indeterminate: no + use, no mint, no expiry. Graphless-era cookies never got a stub because they have no row for - the scan to find; no eventual read participates. The flag itself is specified: a named - deployment-metadata key (write-once/CAS class), set by the §4.2 - migration runbook step (migration spec §4) only on deployments that actually ran graphless - (requires the deployment-metadata capability), surviving binary - rollback, and **cleared by an explicit operator action** once the - migration window closes (quiet-period criterion in the guide) — - clearing ends rowless classification permanently. Outside the flag, - or on any read error, the state is **indeterminate**: no identity - use, no mint, no cookie expiry — "treated as absent" was the wrong - contract, since absence feeds the fresh-mint path (admitted negative - writes still proceed, per the default row above). + the scan to find; no eventual read participates. The flag itself is + specified: a named deployment-metadata key (write-once/CAS class), + set by the §4.2 migration runbook step (migration spec §4) only on + deployments that actually ran graphless (requires the + deployment-metadata capability), surviving binary rollback, and + **cleared by an explicit operator action** once the migration window + closes (quiet-period criterion in the guide) — clearing ends rowless + classification permanently. Outside the flag, or on any read error, + the state is **indeterminate**: no identity use, no mint, no cookie + expiry — "treated as absent" was the wrong contract, since absence + feeds the fresh-mint path (admitted negative writes still proceed, + per step 1). - A verified rowless cookie (`verify → VerifiedIdentity`, carrying the matched version) is **expired and replaced by a fresh mint through the @@ -595,8 +644,11 @@ The contract: - **Provenance is provider- and version-tagged, with a defined rotation schema.** Every graph row carries the minting provider id, its configuration version, and the per-permission grant evidence (grant - basis, evidence timestamp, resolved jurisdiction, policy revision) that - the S2S sync authority recomputes from (permission model spec §7). + basis, evidence timestamp, resolved jurisdiction, policy revision) + **mirrored for audit only** — the S2S sync authority recomputes from + the strong authority summary alone, never from row provenance + (permission model spec §7; the §6.3 authority-state wire record is + the one normative summary schema). Same-provider key/passphrase rotation is configuration, not a provider switch: a provider block may hold multiple `versions` entries (`[ec.providers.hmac.versions.v2] passphrase = …`) with @@ -690,17 +742,17 @@ solves. **Physical key grammar.** Core constructs every key; providers supply only the bounded suffix: -| Record class | Key | Notes | -| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ **123** bytes (128-byte total key cap minus tag and provider code — an earlier 128 here contradicted the constructor; 123/124 boundary fixtures required), portable alphabet **`[A-Za-z0-9._~-]`** (not "KV-safe per host", which would break the single cross-adapter grammar; providers with native identifiers outside it emit base64url suffixes); ≤ **123** bytes with 123/124 boundary fixtures. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | -| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | -| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | -| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | -| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | -| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8) where **each entry carries its own `valid_until`** = its withdrawal time + max(cookie lifetime, row/S2S authority horizon) — one record-level lifetime either shortchanged late entries or, rolling, let an attacker keep a saturated NAT cohort withdrawn forever; the record expires when its last entry (or the saturation flag's own pinned horizon) expires; saturation flag with its own entry-time-pinned horizon; CAS version; created-at — a `w` expiring before the row horizon would let an overflow row's identity resurface (the overflow non-promotion residual is sign-off 30); readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | -| Deployment metadata | `m` + fixed metadata name (fixed-width grammar): schema floor, graphless-migration flag, policy-activation register | Write-once/CAS class. The **floor value is encoded**: integer writer-activation schema version + minimum-reader version, ordered numerically; a binary starts only if its declared reader capability ≥ the floor's minimum-reader, which is what makes "is N+1 permitted after N+2 activates" decidable (N+1 declares N+2-reader capability, so yes). The **graphless flag's** value carries schema version, state, epoch, set-at, **`not_before`, its clock domain, and `L`** (serialized, so every observer reads the same deadline) plus the attestation; lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded. The **policy-activation register** holds the linearizable `{source_version, policy_digest, ordinal, activated_at}` value with the transition rules of permission spec §5.5 (idempotent same-activation reuse; stale `source_version` rejected) | -| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | -| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | +| Record class | Key | Notes | +| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ **123** bytes (128-byte total key cap minus tag and provider code — an earlier 128 here contradicted the constructor; 123/124 boundary fixtures required), portable alphabet **`[A-Za-z0-9._~-]`** (not "KV-safe per host", which would break the single cross-adapter grammar; providers with native identifiers outside it emit base64url suffixes); ≤ **123** bytes with 123/124 boundary fixtures. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | +| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | +| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | +| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | +| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | +| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8) where **each entry carries its own `valid_until`** = its withdrawal time + max(cookie lifetime, row/S2S authority horizon) — one record-level lifetime either shortchanged late entries or, rolling, let an attacker keep a saturated NAT cohort withdrawn forever; the record expires when its last entry (or the saturation flag's own pinned horizon) expires; saturation flag with its own entry-time-pinned horizon; CAS version; created-at — a `w` expiring before the row horizon would let an overflow row's identity resurface (the overflow non-promotion residual is sign-off 30); readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | +| Deployment metadata | `m` + fixed metadata name (fixed-width grammar): schema floor, graphless-migration flag, policy-activation register | Write-once/CAS class. The **floor value is encoded**: integer writer-activation schema version + minimum-reader version, ordered numerically; a binary starts only if its declared reader capability ≥ the floor's minimum-reader, which is what makes "is N+1 permitted after N+2 activates" decidable (N+1 declares N+2-reader capability, so yes). The **graphless flag's** value carries schema version, state, epoch, set-at, **`not_before`, its clock domain, and `L`** (serialized, so every observer reads the same deadline) plus the attestation; lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded. The **policy-activation register** holds the current `{source_version, policy_digest, ordinal, activated_at}` **plus a bounded 16-entry activation history**, with the transition rules of permission spec §5.5 (history adoption for same-activation idempotence; same `source_version` with a different digest fails closed; stale rejected) | +| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | +| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | Grammars are pairwise non-intersecting by their literal prefixes (plus the reserved hmac grammar), and every record value carries a `kind` @@ -807,12 +859,23 @@ evidence digest would give every renewal a fresh key and make draft shipped. A slot stores its `state_key`, the current evidence digest, that digest's pinned first-seen, the newest authoritative timestamp observed, **and a per-permission `observed_at` map** (bounded -by the enforced permission set): a slot update refreshes `observed_at` -only for permissions whose §4.5 semantic token actually changed against -the stored vector — unchanged permissions keep theirs, which is what -makes the permission spec's per-permission equality digests real (a -P4-only change must never refresh P1's age) without per-permission -slots; a TCF renewal (same `state_key`, newer +by the enforced permission set) — maintained by a defined transactional +algorithm, since the comparison base and copy-forward source were +otherwise ambiguous: the record stores a per-source +**`current_state_key` pointer** naming the active slot; on a live +resolution producing combined vector V, the comparison base is **the +current slot's stored vector** (the immediately preceding active +state — never the vector of whatever old slot `key(V)` happens to +match); per permission, a changed token takes the new observation's +timestamp and an unchanged token **copies `observed_at` forward from +the current slot**; the target slot — fresh, or a previously occupied +slot being returned to (A→B→A) — has its per-permission map **replaced +by the computed map** (its stale map is never merged; its pinned +first-seen digest handling is unchanged); then +`current_state_key := key(V)`, all inside the record's single CAS. +A→B→A and P4-only-change are named test vectors. This is what makes +the permission spec's per-permission equality digests real (a P4-only +change must never refresh P1's age) without per-permission slots; a TCF renewal (same `state_key`, newer `LastUpdated`) updates the slot in place, while a replay (not newer) changes nothing — replay protection derives from recency comparison, not per-value history, so no per-digest sublists are needed. 16 slots @@ -847,10 +910,24 @@ leaves saturation and novel values occupy slots normally, each with its own full TTL; if every slot still holds an unexpired state — TCF renewals legitimately extend slots past `saturated_until`, since slots are individually lived and the epoch bounds only untracked overflow — -the next novel value opens a **new epoch** whose restrictive marker -pins to its own first restrictive overflow: epochs never chain or -inherit timestamps across their boundary. Saturation is a first-class -metric — +the next novel value opens a **new epoch**. **The restrictive marker +is marker-scoped, not epoch-scoped, in lifetime**: it lives to its own +`valid_until` (first-overflow-pinned + full TTL), **outliving the +epoch that created it when the first overflow arrived late** — +discarding it at `saturated_until` would deny that first overflow its +promised lifetime, and the record holds **at most one live marker per +source**, so overlapping markers never need representing. While the +marker lives, every later restrictive overflow — same epoch or a +successor — inherits it (the anti-replay inherit rule; the declared +shortening therefore spans epoch boundaries); only after it expires +does the next restrictive overflow pin a fresh marker at its own +timestamp. "Epochs never chain timestamps" means exactly this: a +**fresh** marker derives from its own overflow, never from epoch state +or a prior marker. The grant rule at the boundary: "novel values +cannot grant" is epoch-scoped and ends at `saturated_until`; the +marker carries only the restrictive/denial state of the overflow +opt-outs, evaluated by its `valid_until` alone. Saturation is a +first-class metric — the cap and its denial behavior are **sign-off item 31**; record level: family ID, CAS version counter, schema version, stub marker (backfill, §5); unknown-field and range validation apply like every class (strong class, permission-exempt writes per the @@ -914,18 +991,18 @@ Requirements: **per-record-class consistency requirements**, because "has KV" says nothing about whether revocation is observable: - | Record class | Required semantics | - | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | - | Family revocation records | **Globally observable strong consistency** — every instance's read observes a committed revocation, not merely the writing session's own writes (writer-scoped read-your-writes is insufficient for a fleet). Cloudflare Workers KV is **not eligible** — "60 seconds or more" is an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | - | Authority-state records (suppression + positive summary) | **Globally observable strong reads AND linearizable per-key CAS** — CAS alone orders writes, but a stale successful _read_ on another instance would authorize egress after a committed suppression ("read failures fail closed" does not cover stale successes); both properties are adapter eligibility gates | - | Identity-row mutation | **Generation CAS** (conditional write on row generation) with reread/recompute on conflict — rows are heavily mutable (snapshots replaced, partner IDs merged, derived state refreshed), so unordered last-writer-wins loses newer evidence and mappings; _visibility_ may stay eventual, unordered _mutation_ may not. Fastly KV offers generation-marker conditional writes; Workers KV's documented concurrent last-write-wins is ineligible for mutation-bearing rows | - | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | - | Deployment metadata — schema floor (write-once/CAS); **graphless flag additionally requires globally observable strong reads** (N+2 lease revalidation and N+1's `not_before` barrier both need globally current reads, not just write-once) | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | - | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | - | Rowless prefix-withdrawal (`w`) records | **Globally strong reads + linearizable CAS** (same bar as authority-state), plus the **bounded listing-visibility window** the backfill scan depends on — all three are eligibility gates for the graphless migration. The strong-read obligation is **permanent for HMAC row discovery** — enforcement runs to the last entry's `valid_until`, long after the flag clears — and retention runs through the **max(cookie, row, S2S) horizon of each entry** (an earlier "cookie lifetime" cell contradicted §6.3's per-entry horizons) | - | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | - | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | + | Record class | Required semantics | + | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | + | Family revocation records | **Globally observable strong consistency** — every instance's read observes a committed revocation, not merely the writing session's own writes (writer-scoped read-your-writes is insufficient for a fleet). Cloudflare Workers KV is **not eligible** — "60 seconds or more" is an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | + | Authority-state records (suppression + positive summary) | **Globally observable strong reads AND linearizable per-key CAS** — CAS alone orders writes, but a stale successful _read_ on another instance would authorize egress after a committed suppression ("read failures fail closed" does not cover stale successes); both properties are adapter eligibility gates | + | Identity-row mutation | **Generation CAS** (conditional write on row generation) with reread/recompute on conflict — rows are heavily mutable (snapshots replaced, partner IDs merged, derived state refreshed), so unordered last-writer-wins loses newer evidence and mappings; _visibility_ may stay eventual, unordered _mutation_ may not. Fastly KV offers generation-marker conditional writes; Workers KV's documented concurrent last-write-wins is ineligible for mutation-bearing rows | + | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | + | Deployment metadata — schema floor (write-once/CAS); **graphless flag additionally requires globally observable strong reads** (N+2 lease revalidation and N+1's `not_before` barrier both need globally current reads, not just write-once); **policy-activation register additionally requires linearizable CAS + strong reads and an ordered upstream `source_version`** (backend push version, or the `ts config push` envelope sequence); **graphless deadline checks additionally require one of: store-issued current time on a strong read, or a declared bounded fleet clock skew (S_fleet)** — each its own capability cell | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | + | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | + | Rowless prefix-withdrawal (`w`) records | **Globally strong reads + linearizable CAS** (same bar as authority-state), plus the **bounded listing-visibility window** the backfill scan depends on — all three are eligibility gates for the graphless migration. The strong-read obligation is **permanent for HMAC row discovery** — enforcement runs to the last entry's `valid_until`, long after the flag clears — and retention runs through the **max(cookie, row, S2S) horizon of each entry** (an earlier "cookie lifetime" cell contradicted §6.3's per-entry horizons) | + | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | + | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | Every record class additionally declares **durability and maximum retention**: a store passing the consistency check but capping TTLs @@ -946,19 +1023,19 @@ Requirements: them is how a "yes" cell hides an unusable feature. Feature eligibility requires wired, not merely available: - | Capability | Fastly | Axum (dev) | Cloudflare | Spin | - | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | - | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | - | Prefix listing (cluster) | Used today, but **ratification requires a cited platform completeness bound, pagination behavior, and failure semantics** — the graphless scan's settle window cannot be derived from "works in practice" | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | - | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | - | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | - | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | - | Device host evidence (JA4/H2) | Yes | No | No | No | - | Authority-state: global strong reads + CAS | Conditional writes available (generation marker); **globally current read semantics to verify** — both required, wiring to verify | **Unavailable** (no store wired) | Workers KV: **ineligible**; Durable Objects: feasible, not wired | **Unavailable** | - | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | - | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | - | Deployment metadata — floor (write-once/CAS) **and graphless flag (globally strong reads + CAS)** | **Not wired** — needs a primitive distinct from the config store | **Unavailable** | DO: feasible, not wired | **Unavailable** | - | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | + | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | + | Prefix listing (cluster) | Used today, but **ratification requires a cited platform completeness bound, pagination behavior, and failure semantics** — the graphless scan's settle window cannot be derived from "works in practice" | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | + | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | + | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | + | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | + | Device host evidence (JA4/H2) | Yes | No | No | No | + | Authority-state: global strong reads + CAS | Conditional writes available (generation marker); **globally current read semantics to verify** — both required, wiring to verify | **Unavailable** (no store wired) | Workers KV: **ineligible**; Durable Objects: feasible, not wired | **Unavailable** | + | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Deployment metadata — floor (write-once/CAS), **graphless flag (globally strong reads + CAS + store-clock or S_fleet branch)**, and **policy-activation register (linearizable CAS + strong reads + ordered `source_version`)** | **Not wired** — needs a primitive distinct from the config store | **Unavailable** | DO: feasible, not wired | **Unavailable** | + | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 07992a0b3..4c2fcb95f 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -544,5 +544,5 @@ implemented. | 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance — including CSP interaction with vendor challenge pages, same-origin vendor code on the publisher origin (or an origin-isolation/sandboxing requirement), the **specific `X-DD-B` divergence** (DataDome's documented cookie-mode allow example directs `Set-Cookie` **and** `X-DD-B` to the client, while TS drops `X-DD-B` per the allowlist matrix — vendor acceptance must name that exact field), and the fail-open consequence of batch invalidation | hook §4a; `datadome-header-allowlist.md` | — | open | | 29 | Unverifiable roaming rowless cookies receive best-effort cookie-only expiry (admission rules forbid durable records for unverified values); a lost response can leave the cookie usable on the old network until re-presented | providers §5 | — | open | | 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, observability, **and the saturation collateral: under a saturated prefix, any real row (listed or overflow) is denied and revoked immediately on surfacing — a non-abuser NAT-cohort row can be revoked; `w` is retained through the max(cookie, row, S2S) horizon and consulted by `valid_until`, not the flag, so this is deterministic, not a retention accident** — all in scope | providers §5 | — | open | -| 31 | Replay-history capacity (16 per-source semantic-state slots + a saturation epoch whose restrictive marker is pinned at the **first** restrictive overflow with its own full TTL): while saturated, fresh consent cannot grant until the epoch expires, and **later restrictive overflows inherit the first marker — a shortening of their lifetime, down to nearly zero late in the epoch**; ratification chooses this knowingly — the alternatives (per-overflow state; refreshing the marker on later overflows) were rejected for unbounded storage and replay-extension respectively | permission §4.3; providers wire schema | — | open | +| 31 | Replay-history capacity (16 per-source semantic-state slots + a saturation epoch whose restrictive marker is pinned at the **first** restrictive overflow with its own full TTL): while saturated, fresh consent cannot grant until the epoch expires, and **later restrictive overflows inherit the first marker — a shortening of their lifetime, down to nearly zero near the marker's expiry (the marker outlives its epoch and can span epoch boundaries)**; ratification chooses this knowingly — the alternatives (per-overflow state; refreshing the marker on later overflows) were rejected for unbounded storage and replay-extension respectively | permission §4.3; providers wire schema | — | open | | 32 | GPP sections 24–27 (MD/IN/KY/RI) are reserved with **national-section-only** handling until an official binary layout can be vendored — a state-specific opt-out expressed only in an undecodable state section is not honored | permission §4.5; `gpp-registry-snapshot.md` | — | open | diff --git a/docs/superpowers/specs/datadome-header-allowlist.md b/docs/superpowers/specs/datadome-header-allowlist.md index 2cc9db018..698553e1f 100644 --- a/docs/superpowers/specs/datadome-header-allowlist.md +++ b/docs/superpowers/specs/datadome-header-allowlist.md @@ -24,17 +24,17 @@ column is added by the sign-off-23 opt-in, never implicitly). No wildcard rows exist — every accepted name is enumerated, and **every cell terminates in exactly one outcome**. -| Pointer | Respond (cookie mode) | Continue (cookie mode) | -| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ | -| `Set-Cookie` | typed `datadome` cookie operation (hook spec §4a parser; exactly one `datadome` field) | typed `datadome` cookie operation | -| `X-Set-Cookie` | **invalidate the batch → Continue** (mode mismatch: TS never requests header sessions in v1; a vendor response asserting one must not half-apply) | **invalidate the batch** → effects dropped | -| `Location` | forward (3xx only), replace | rejected | -| `Content-Type` | forward (owns its body), replace | rejected | -| `Cache-Control` | restricted merge; invariant pass last | rejected | -| `Pragma` | drop-individually, logged (response `Pragma` has no standardized meaning, RFC 9111 §5.4) | drop-individually, logged | -| `X-DataDome` | forward, owner-scoped typed telemetry | forward, owner-scoped typed telemetry | -| `X-DD-B` | drop-individually, logged (header-session artifact; the vendor's cookie-mode allow example emits it — the drop is the named divergence in sign-off 28) | drop-individually, logged | -| anything not listed | invalidate the batch → Continue | invalidate → effects dropped | +| Pointer | Respond (cookie mode) | Continue (cookie mode) | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ | +| `Set-Cookie` | typed `datadome` cookie operation (hook spec §4a parser; exactly one `datadome` field) | typed `datadome` cookie operation | +| `X-Set-Cookie` | **invalidate the batch → Continue** (mode mismatch: TS never requests header sessions in v1; a vendor response asserting one must not half-apply) | **invalidate the batch** → effects dropped | +| `Location` | forward on a 3xx Respond, replace; **on a non-3xx Respond → invalidate the batch → Continue** (a redirect field without a redirect status is a malformed decision) | **invalidate the batch** (effects dropped) | +| `Content-Type` | forward (owns its body), replace | **invalidate the batch** (effects dropped) | +| `Cache-Control` | restricted merge; invariant pass last | **invalidate the batch** (effects dropped) | +| `Pragma` | drop-individually, logged (response `Pragma` has no standardized meaning, RFC 9111 §5.4) | drop-individually, logged | +| `X-DataDome` | forward, owner-scoped typed telemetry | forward, owner-scoped typed telemetry | +| `X-DD-B` | drop-individually, logged (header-session artifact; the vendor's cookie-mode allow example emits it — the drop is the named divergence in sign-off 28) | drop-individually, logged | +| anything not listed | invalidate the batch → Continue | invalidate → effects dropped | Singleton-field multiplicity (`Location`, `Content-Type`, `X-DataDome`, `X-DD-B`, `X-Set-Cookie` appearing more than once) invalidates the diff --git a/docs/superpowers/specs/gpp-registry-snapshot.md b/docs/superpowers/specs/gpp-registry-snapshot.md index 2a9eff551..0eac72a20 100644 --- a/docs/superpowers/specs/gpp-registry-snapshot.md +++ b/docs/superpowers/specs/gpp-registry-snapshot.md @@ -60,5 +60,5 @@ a revision; the commit hash is the reproducible authority. **Status: placeholder until ratification.** Neither the immutable registry commit nor the conformance vectors are recorded yet; like the PSL snapshot, filling them is a pre-ratification prerequisite -(migration spec §7) — the §4.5 field mappings cannot be reproduced +(migration spec §4) — the §4.5 field mappings cannot be reproduced against a pinned registry until they land. diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index 9c6c62c66..916397385 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -382,6 +382,11 @@ state_key `tsstk1|tcf|p1=grant,p4=refuse` → serialized in the metadata schema; clock-skew and suspender-restart tests named. - The strong summary is the **sole S2S decision source** — the §7 + _(correction, R20: text-added, not verified-closed — a + provider-switching paragraph still said S2S "recomputes from" row + provenance, and the permission-side enumeration omitted + `jurisdiction_observed_at`; both fixed in R20 with the providers §6.3 + record named as the one normative schema)_ — authority paragraph and the row-schema provenance row (now "audit mirror only") no longer describe a second source. - The saturation restrictive marker pins to the **first restrictive @@ -508,14 +513,21 @@ entirely with an embedded vector (`tsevd1|gpp|p1=grant,p4=refuse` → `89b08580…`), and slots gain a per-permission `observed_at` map refreshed only for changed tokens; saturation epoch expiry is a complete transition (lazy GC, re-saturation opens a new epoch pinned to -its own first overflow, no cross-epoch inheritance); `w` capability +its own first overflow) — _correction, R20: R19's "no cross-epoch +inheritance" contradicted the marker's own full TTL when the first +overflow arrived late; the marker is now marker-scoped in lifetime, +outliving its epoch, one live marker per source_; `w` capability retention corrected to the per-entry max(cookie, row, S2S) horizon with a permanent strong-read obligation for HMAC row discovery (and the §6.2 "migration-window" label fixed); HEAD updates compare against origin-side metadata only and never touch processed-side headers; the -Vary digest is a full contract (HMAC-SHA-256, `tsvry1|` grammar, comma -join, secret-store key with versioned id, all values digested, zero-key -vector `60fdeb3a…`); the adapter ceiling matrix now exists (Axum fixed +Vary digest was called a full contract (HMAC-SHA-256, `tsvry1|` +grammar, comma join, secret-store key with versioned id, all values +digested) — _correction, R20: the R19 grammar had deterministic +collisions (absent = present-empty; comma-join member ambiguity) and no +key deployment contract; rebuilt in R20 with presence/count/ +length-prefix encoding, new vectors, and the named setting/startup +gate_; the adapter ceiling matrix now exists (Axum fixed ≥ budget; Fastly/Cloudflare/Spin qualification-pending with fail-closed startup) with the exact counted-bytes formula and over-budget-snapshot behavior; the cookie parser defines the "; " join, duplicate-datadome @@ -548,3 +560,74 @@ inline supersession note in place, not only the banner. Ratification state is unchanged and user-side: all 32 decision rows open, `decisions/` empty beyond the README, no adapter qualified, and the PSL and GPP snapshots are placeholders — all named gates. + +## Round 20 (at fe4d7bd34) + +6 P1, 7 P2, 4 P3, plus declared ratification blockers (unchanged, +user-side). All addressed. Three prior ledger claims were corrected +above as text-added-not-verified (R18 S2S sole-source; R19 Vary +contract; R19 no-cross-epoch-inheritance). + +P1: the classification contract is now an **ordered first-match-wins +procedure** replacing the five-column table that was neither total nor +disjoint — negative gates run first (w → revocation → suppression, with +the w/revocation overlap resolved by idempotent promotion), the +rowless step keys on the strong-class read's authoritative absence +(the "authoritative not-found" eventual read is called out as a state +the store cannot produce), and the missing states (absent authority +record → AuthorityRefresh backfill; nonmatching revisions → fence +fails, summary-only, refresh realigns) have steps. The +policy-activation register holds a bounded 16-entry history: exact +pairs adopt their historical ordinal (the interleaved-eviction +double-ordinal scenario is closed), same source_version with a +different digest fails closed as mixed-binary parse divergence, stale +or window-expired versions are rejected, the digest-only fallback is +deleted (source_version must be assigned exactly once upstream — push +version or the ts-config-push envelope sequence), and the register is +a named capability in both provider matrices. The last row-provenance +S2S claim is deleted (provider-switching section now says +audit-mirrored, summary-only) and the permission §7 enumeration +declares itself a reference to the one normative schema, with +jurisdiction_observed_at included. The restrictive marker is +marker-scoped in lifetime — it outlives its epoch (a late first +overflow keeps its full TTL), one live marker per source, later +overflows inherit it across epoch boundaries, fresh markers pin only +after expiry, and the epoch-scoped rule is narrowed to grant-blocking. +The Vary digest grammar encodes presence, instance count, and +length-prefixed member octets (absent ≠ present-empty per RFC 9111 +§4.1; list members cannot collide with embedded commas), with new +zero-key vectors (`c880c5e8…` present, `a2ae26cf…` absent). Origin-304 +re-derivation is defined: replay the persisted mutation IR (accepted +operation batches — deterministic core-defined semantics) over the +updated base plus the invariant pass, never re-running mutators; +IR-less entries force refetch; publication is one atomic entry commit +with insert-then-index Vary rekeying, both adapter capability cells. + +P2: the per-source slot algorithm gains a current_state_key pointer +with copy-forward from the current slot and full-map replacement on +A→B→A (named vectors); malformed/absence suppressions gain a scoped +recovery-observation clearing rule (valid grant presented after the +suppression clears it without refreshing the grant's age; opt-out +stickiness untouched); the suspension named test now states the +branch-appropriate comparison (store_now, or now − S_fleet) and the +store-clock/S_fleet branches are capability cells; the DataDome matrix +cells are all terminal (non-3xx Respond Location invalidates; Continue +Location/Content-Type/Cache-Control invalidate with effects dropped); +Respond-on-HEAD omits Content-Length (RFC 9110 §9.3.2 — HEAD bytes +cannot establish the GET length without a vendor equivalence +guarantee); the Vary HMAC key is a deployable contract +([cache] vary_digest_key_secret_name, ≥ 32 CSPRNG bytes, key-id +grammar, fail-closed startup, overlap rotation, capability + gate); +the 304 row counts four revisions, evaluates the complete RFC 9110 +§13 precondition set (412 included) after recovery, and states the +safe-update set once as the "cache-relevant fields" definition. + +P3: the eligibility table is structurally valid again (the unescaped +`tscfg1|` pipe had split the 304 row; the row is rebuilt with zero +content pipes); the GPP snapshot cites migration §4; the ledger +corrections above; the broken `\_classification*` emphasis is fixed +and the cookie citations updated to RFC 10025 (obsoleting RFC 6265). + +Ratification state: unchanged — 32 open rows, decisions/ empty beyond +the README, no adapter qualified for the identity protocol or the +header ceilings, GPP/PSL snapshots placeholders. All user-side gates. From f5a77fa0974c4641d71ceadb6f8e424717baac43 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Thu, 6 Aug 2026 13:30:31 -0700 Subject: [PATCH 22/24] docs: finalize providers and permissions specifications Includes the approved specification redesign and the earlier external round-21 edits for ordered classification, S2S cleanup, and adapter capability splits. --- ...-datadome-server-side-protection-design.md | 360 ++++-- ...2026-06-16-edgezero-based-ts-cli-design.md | 312 ++++- ...integration-response-header-hook-design.md | 392 ++++-- .../2026-07-30-permission-model-design.md | 1122 +++++++++++++---- .../2026-07-30-pluggable-providers-design.md | 666 ++++++---- ...07-30-provider-migration-rollout-design.md | 437 ++++--- .../specs/activation-journal-vectors.json | 98 ++ .../contextual-openrtb-v1-allowlist.json | 668 ++++++++++ .../specs/datadome-header-allowlist.md | 245 +++- .../specs/gpp-registry-snapshot.md | 74 +- .../policy-canonicalization-vectors.json | 77 ++ docs/superpowers/specs/pr986-review-ledger.md | 252 ++++ docs/superpowers/specs/psl-snapshot-ref.md | 24 +- .../revision-canonicalization-vectors.json | 45 + 14 files changed, 3791 insertions(+), 981 deletions(-) create mode 100644 docs/superpowers/specs/activation-journal-vectors.json create mode 100644 docs/superpowers/specs/contextual-openrtb-v1-allowlist.json create mode 100644 docs/superpowers/specs/policy-canonicalization-vectors.json create mode 100644 docs/superpowers/specs/revision-canonicalization-vectors.json diff --git a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md index 31a539527..d262beba2 100644 --- a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md +++ b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md @@ -13,7 +13,11 @@ > send `X-DataDome-X-Set-Cookie` when the header ID is used") is > **superseded for v1**: header-session mode is startup-rejected (hook > spec §4a); TS never requests it and does not forward incoming header -> ClientIDs to the vendor. +> ClientIDs to the vendor. This document's generic "TLS/client metadata" +> instruction is also narrowed by §4a: DataDome may receive only explicitly +> enumerated, request-scoped security fields under `SecurityUse`; the device +> provider remains deferred, no fingerprint-derived classification is stored, +> and unlisted host evidence is omitted. **Issue:** #317 **Date:** 2026-06-11 @@ -69,8 +73,10 @@ JavaScript SDK. default. Default-exclude Trusted Server internal routes and static assets. 2. **Endpoint default:** default to DataDome's Fastly-specific Protection API endpoint from the official Fastly Compute docs, while allowing override. -3. **Header precedence:** apply DataDome downstream headers last so DataDome - cookies/cache/challenge headers are not overwritten by generic finalization. +3. **Header precedence (updated by PR #986):** apply DataDome through the + core-owned security channel after ordinary mutators, then run the final + cache/privacy invariant pass unconditionally last. Security effects do not + override framing, cache safety, or privacy invariants. 4. **GraphQL support:** defer. 5. **Client-side tag:** auto-inject when a client-side key is configured. 6. **Methods:** protect every non-`OPTIONS` method, including `HEAD`, when the @@ -181,41 +187,76 @@ pub trait IntegrationRequestFilter: Send + Sync { pub struct RequestFilterInput<'a> { pub settings: &'a Settings, pub services: &'a RuntimeServices, - pub request: &'a Request, + /// The only request surface generic integrations receive. + pub request: &'a RedactedRequestView<'a>, } pub enum RequestFilterDecision { - Continue(RequestFilterEffects), + Continue(OrdinaryRequestFilterEffects), Respond { response: Response, - effects: RequestFilterEffects, + effects: OrdinaryRequestFilterEffects, }, } -#[derive(Default)] -pub struct RequestFilterEffects { - pub request_headers: Vec, - pub response_headers: Vec, +// Separate, core-only registration path. It is not a supertrait or optional +// field on IntegrationRequestFilter, so another integration cannot receive +// the DataDome capability through the generic runner. +#[async_trait(?Send)] +pub(crate) trait DataDomeSecurityRequestFilter: sealed::Sealed + Send + Sync { + async fn filter_datadome( + &self, + input: DataDomeSecurityFilterInput<'_>, + ) -> Result>; } -pub struct HeaderMutation { - pub name: String, - pub value: String, - pub mode: HeaderMutationMode, +pub(crate) struct DataDomeSecurityFilterInput<'a> { + pub settings: &'a Settings, + pub services: &'a RuntimeServices, + pub request: &'a RedactedRequestView<'a>, + /// Constructed by core from the normative field allowlist. No raw + /// Request/header map or AuthorizedIdentity is exposed. + pub security: &'a DataDomeSecurityRequestView<'a>, } -pub enum HeaderMutationMode { - Set, - Append, +pub(crate) enum DataDomeSecurityDecision { + Continue(DataDomeSecurityEffects), + Respond { + response: Response, + effects: DataDomeSecurityEffects, + }, +} + +#[derive(Default)] +pub(crate) struct DataDomeSecurityEffects { + pub upstream_overlay: Vec, + pub browser_effects: Vec, } ``` +`RedactedRequestView`, `DataDomeSecurityRequestView`, the security trait, and +both security operation enums are core-owned sealed surfaces. Generic +integrations cannot construct or read them or recover the underlying raw +request. Core strips `ts-*`, EID/identity material, +`X-DataDome-ClientID`, and the `datadome` cookie before building the shared +view; the security view restores only the one typed cookie value and exact +request evidence admitted by `datadome-header-allowlist.md`. Another filter +receives only the shared redacted view and cannot inherit this owner +capability. This paragraph and the hook spec §4a replace every earlier generic +`&Request`/generic security-header-mutation sketch in this document. The +ordinary effects type remains subject to the hook's ordinary attributed-batch +registry and cannot express `SecurityUse`, owner overlay, cookies, or reserved +security names. + Important behavior: -- Filters run in registration order. -- On `Continue`, request header mutations are applied immediately before the - next filter and before route matching. -- Response header mutations are accumulated and applied to the final response. +- Ordinary filters run in registration order over the redacted view. DataDome's + security owner view is evaluated in its dedicated security position and is + never passed to the next filter. +- On `Continue`, allowlist-validated upstream operations enter only DataDome's + owner-scoped publisher overlay; they never mutate the shared request. +- Typed browser effects are accumulated and applied through the hook spec's + single pointer matrix and security budget. - On `Respond`, routing short-circuits with that response while preserving any downstream response header effects that must be applied after finalization. _(Superseded: one global order applies — core finalization → ordinary @@ -250,8 +291,10 @@ pub async fn filter_request( ) -> Result> ``` -The registry outcome should contain either an immediate response plus response -header mutations, or a continue decision with accumulated response header +The registry outcome should contain either an immediate response plus typed +security operations, or a continue decision with accumulated typed security +operations and an owner-scoped publisher overlay. Generic header name/value +mutations are not part of this API. mutations. ### 3. Fastly Route Hook @@ -262,12 +305,13 @@ In `route_request()`, run filters after normal basic auth succeeds and before ```text basic auth ok → integration_registry.filter_request(...) - → Respond { response, effects }: finalize response, apply DataDome headers last, return - → Continue(effects): request is enriched; route normally; remember response effects + → Respond { response, security_effects }: validate the complete security batch + → Continue(security_effects): apply only the owner-scoped upstream overlay; route normally → route matching → EC finalize -→ generic finalize_response -→ apply request-filter response headers last +→ ordinary response mutators +→ validated security effects +→ final cache/privacy invariant pass (always last) ``` Streaming publisher responses need the same treatment before headers are @@ -275,8 +319,9 @@ committed via `stream_to_client()`. ### 4. Header Mutation Semantics -DataDome pointer headers are internal instructions and must not be forwarded. -Only headers named by the pointers should be copied. +DataDome pointer headers are internal instructions and are never forwarded. +The one normative field/pointer allowlist and decision matrix is +`datadome-header-allowlist.md`; a pointer does not authorize an unlisted name. | Pointer header | Destination | | ---------------------------- | -------------------------------------------------- | @@ -285,14 +330,14 @@ Only headers named by the pointers should be copied. Rules: -- `Set-Cookie` mutations use append mode. -- Other headers use set/replace mode. +- `datadome` cookie effects use the hook spec's typed cookie operation; raw + `Set-Cookie` is not a generic mutation. +- Every other admitted field follows its exact decision-matrix cell; there is + no generic set/replace default. - Pointer headers themselves are never forwarded. -- Header mutations must reject hop-by-hop, request-target, body framing, and - Trusted Server internal headers such as `Connection`, `Transfer-Encoding`, - `Content-Length`, `Host`, and `x-ts-*`. -- DataDome downstream headers are applied after `ec_finalize_response()` and - `finalize_response()`. +- Hop-by-hop, request-target, body-framing, credential, Trusted Server + internal, and unlisted headers invalidate the applicable batch. +- Security effects run before the final invariant pass, never after it. ## DataDome Protection Design @@ -315,8 +360,9 @@ rewrite_sdk = true enable_protection = false server_side_key_secret_store = "ts_secrets" server_side_key_secret_name = "datadome_server_side_key" -protection_api_origin = "https://api-fastly.datadome.co" timeout_ms = 1500 +complete_response_timeout_ms = 3000 +challenge_body_max_bytes = 65536 protection_excluded_methods = ["OPTIONS"] protection_excluded_asns = [] protection_excluded_ip_cidrs = [] @@ -324,6 +370,14 @@ protection_excluded_ip_cidr_sources = [] protection_ip_list_cache_ttl_seconds = 300 enable_graphql_support = false +# Security identity/lifecycle. No default exists for max age: enabling +# protection without an explicit value is a startup error. +security_cookie_max_age = 2592000 # example: 30 days; allowed 7d..=365d +security_cookie_domain = "host-only" # or one exact normalized ASCII domain +security_cookie_same_site = "Lax" # Lax | Strict | None +expose_client_id_to_origin = false +expose_host_fingerprints_to_vendor = false + # New client-side tag injection layer client_side_key = "" inject_client_side_tag = true @@ -336,6 +390,12 @@ type = "path_regex" patterns = ["(?i)\\.(avi|flv|mka|mkv|mov|mp4|mpeg|mpg|mp3|flac|ogg|ogm|opus|wav|webm|webp|bmp|gif|ico|jpeg|jpg|png|svg|svgz|swf|eot|otf|ttf|woff|woff2|css|less|js|map)$"] ``` +This block is the canonical v1 DataDome configuration inventory; the hook and +allowlist specs reference it rather than defining another schema. Unknown +legacy security/session fields are errors, not ignored compatibility toggles. +In particular, `sessionByHeader`, `session_by_header`, and any equivalent are +startup-rejected in v1. + Notes: - The literal server-side key is not stored in Rust config. Rust config stores @@ -348,8 +408,27 @@ Notes: - `client_side_key` is optional. Auto-injection emits a tag only when `inject_client_side_tag = true` and `client_side_key` is non-empty; an empty key is a valid no-op. -- `protection_api_origin` remains configurable for regional/static endpoint - selection. +- The v1 Protection API URL is the core-owned constant + `https://api-fastly.datadome.co/validate-request`. It is not operator + configurable. Supporting another DataDome region or a publisher proxy is a + new reviewed endpoint-registry entry and product/security decision, not a + free-form URL setting. Unknown legacy `protection_api_origin` fields are + startup errors so an old override cannot silently exfiltrate the server key. +- `complete_response_timeout_ms` defaults to and may not exceed 3000; + `challenge_body_max_bytes` defaults to and may not exceed 65,536. The hook + spec §4a owns the measurement/abort semantics. +- `security_cookie_max_age` is required when protection is enabled and must be + 604,800..=31,536,000 seconds. `security_cookie_domain` defaults to + `host-only`; an explicit domain must pass the hook spec's exact-domain, + domain-match, PSL, and active-scope-change checks. +- `security_cookie_same_site` accepts exactly `Lax`, `Strict`, or `None`; + `None` is valid only with the unconditionally emitted `Secure` attribute. + DataDome cookies never carry `HttpOnly`. +- Both exposure booleans default to `false`. ClientID-to-origin requires the + exact owner-overlay capability; host fingerprints require qualified JA4 + availability and sign-offs 23/28. A selected adapter that cannot preserve + admitted request-header field-line order or enforce the request/body limits + fails startup for protection rather than synthesizing different evidence. - Static-asset exclusion is represented as a default typed `path_regex` rule and should remain case-insensitive so uppercase file extensions such as `.PNG` are skipped. @@ -435,15 +514,21 @@ Responsibilities: 1. Decide whether a request should be protected. 2. Build the form-encoded Protection API payload. -3. Send `POST /validate-request` through platform services. +3. Send `POST https://api-fastly.datadome.co/validate-request` through platform + services. 4. Classify the API response. 5. Extract pointer-header mutations. 6. Return a request-filter decision. Use platform abstractions for the outbound call: -- Parse `protection_api_origin` with `url`. -- Build a `PlatformBackendSpec` with `first_byte_timeout = timeout_ms`. +- Construct the URL only from the core constant and assert at startup that its + scheme is `https`, host is exactly `api-fastly.datadome.co`, port is absent + (therefore 443), path is exactly `/validate-request`, and it has no userinfo, + query, or fragment. No request/config value participates in this URL. +- Build a `PlatformBackendSpec` with `first_byte_timeout = timeout_ms` and + automatic redirect following disabled. A 3xx response is returned to the + DataDome decision parser; it is never followed to a second authority. - Resolve/register backend with `RuntimeServices::backend().ensure(...)`. - Send an `edgezero_core::http::Request` through `RuntimeServices::http_client().send(...)`. @@ -456,7 +541,9 @@ Content-Length: X-DataDome-X-Set-Cookie: true # only when X-DataDome-ClientID is used — SUPERSEDED for v1: never sent (hook spec §4a) ``` -Payload fields should include the core fields from DataDome's official module: +The exhaustive payload field set is the Protection API request-field section +of `datadome-header-allowlist.md`. The list below is informative and may not +expand that normative allowlist: - `Key` - `IP` @@ -464,7 +551,7 @@ Payload fields should include the core fields from DataDome's official module: - `Protocol` - `Host` - `ServerHostname` -- `Request` as path plus query +- `Request` as the normalized path only; query and fragment are never disclosed - `RequestModuleName` - `ModuleVersion` - `TimeRequest` @@ -481,35 +568,36 @@ Payload fields should include the core fields from DataDome's official module: - `Connection` - `Content-Type` - `From` - - `Origin` + - `Origin` as parsed origin only - `PostParamLen` - `Pragma` - - `Referer` + - `Referer` as parsed origin only - `User-Agent` - `Via` - - `X-Forwarded-For` - - `X-Real-IP` - `X-Requested-With` - - Sec-CH and Sec-Fetch headers supported by the official module -- TLS/client metadata when available from `RuntimeServices::client_info()` - -`ClientID` source priority: - -1. `X-DataDome-ClientID` request header -2. `datadome` cookie - -When `X-DataDome-ClientID` is used, send -`X-DataDome-X-Set-Cookie: true` to the Protection API. -_(Superseded for v1: header-supplied ClientIDs are not forwarded at -all — the vendor payload's ClientID derives only from the `datadome` -cookie, so this header is never sent — hook spec §4a.)_ + - only the individually enumerated Sec-CH and Sec-Fetch fields in the + normative allowlist +- only the TLS/client metadata fields explicitly admitted by the normative + allowlist and sign-offs 23/28; JA4 egress additionally requires + `expose_host_fingerprints_to_vendor = true`, and availability alone is not + authorization. `TlsCipher` and `H2Fingerprint` are omitted in v1 for the + semantic reasons recorded in that allowlist + +In cookie-mode v1, `ClientID` comes only from a single unambiguous +`datadome` cookie. `X-DataDome-ClientID` is stripped from every shared +surface and is not forwarded to the vendor, so TS never sends +`X-DataDome-X-Set-Cookie: true`. Encoding and size rules: - URL-encode all values. -- Omit empty fields. -- Apply per-field truncation before encoding. -- Keep the global payload under DataDome's documented limit. +- Omit empty source-header fields; keep mandatory `ClientID` present with an + empty value when there is no unambiguous cookie. +- Apply the exact per-field decoded-byte limits in the normative allowlist + before encoding. +- Measure the complete form-encoded body and enforce the allowlist's 24,576-byte + ceiling before issuing the call; overflow takes metered fail-open and never + triggers ad hoc field dropping. ### Client Metadata @@ -525,6 +613,7 @@ that adapters can populate when available: ```rust pub struct ClientInfo { pub client_ip: Option, + pub client_port: Option, pub tls_protocol: Option, pub tls_cipher: Option, pub tls_ja4: Option, @@ -535,8 +624,13 @@ pub struct ClientInfo { ``` Fastly can populate `tls_ja4` and `h2_fingerprint` from the request APIs already -used by the JA4/debug device-signal code. Other adapters may leave these fields -empty. +used by the JA4/debug device-signal code. Other adapters may leave those +optional fingerprint fields empty. `client_ip` and `client_port` are required +for a release-qualified Protection API call and come from the adapter's trusted +connection metadata, never a request header. If either is unavailable, the +adapter skips the vendor call through the metered fail-open path and remains +unqualified until vendor sign-off explicitly accepts a different profile; it +never invents port `0` or substitutes a forwarded header. ### Protection API Response @@ -562,23 +656,23 @@ fail open and continue without effects. For challenge statuses: 1. Build a response using DataDome's API response status and body. -2. Copy only headers listed in `X-DataDome-headers`. -3. Append `Set-Cookie` values. +2. Validate the complete decision-scoped pointer batch against + `datadome-header-allowlist.md` and the typed-cookie contract. +3. Apply the accepted security batch atomically. 4. Do not contact the publisher origin. -5. Still run Trusted Server response finalization, then apply DataDome headers - last. +5. Run the final cache/privacy invariant pass after the security batch. ### Allowed Requests For allow status `200`: -1. Copy headers listed in `X-DataDome-request-headers` into the request before - Trusted Server route matching. -2. Accumulate headers listed in `X-DataDome-headers` for the final browser - response. +1. Apply only the owner-scoped publisher-upstream fields admitted by + `datadome-header-allowlist.md` before route matching; the default is no + ClientID exposure. +2. Validate and retain the decision-scoped browser security batch. 3. Continue normal route matching. -4. Apply accumulated DataDome downstream headers after EC and generic response - finalization. +4. Apply ordinary response mutators, then the security batch, then the final + invariant pass. ## Client-Side Auto-Injection @@ -621,17 +715,24 @@ Add: - `IntegrationRequestFilter` - `RequestFilterInput` - `RequestFilterDecision` -- `RequestFilterEffects` -- `HeaderMutation` -- `HeaderMutationMode` -- request-filter storage in `IntegrationRegistryInner` -- builder method `with_request_filter` -- registry method to run filters +- `OrdinaryRequestFilterEffects` +- sealed `DataDomeSecurityRequestFilter`, `DataDomeSecurityFilterInput`, and + `DataDomeSecurityRequestView` +- typed `DataDomeSecurityDecision`, `DataDomeSecurityEffects`, + `DataDomeUpstreamOperation`, and `DataDomeBrowserOperation` +- separate ordinary-filter storage and one core-owned DataDome security slot in + `IntegrationRegistryInner` +- public builder method `with_request_filter` for ordinary filters; a + crate-private `with_datadome_security_filter` callable only by the built-in + DataDome registration path +- separate registry runners; the ordinary runner's input type cannot carry the + security view - unit-test helpers for filters ### `crates/trusted-server-core/src/integrations/mod.rs` -Re-export the new request-filter types. +Re-export only the ordinary request-filter types. The sealed DataDome security +trait, input/view, and operations remain crate-private. ### `crates/trusted-server-core/src/integrations/datadome.rs` @@ -675,7 +776,8 @@ Populate new `ClientInfo` fields from Fastly request/environment when available. - Apply request header mutations before route matching. - Carry response header mutations through all non-streaming and streaming response paths. -- Apply DataDome/filter response headers last. +- Apply DataDome/filter response effects through the hook spec's security + channel, followed by the invariant pass. ### `trusted-server.toml` @@ -698,11 +800,13 @@ Update after implementation to describe: ### Registry Tests -- filter runs in registration order -- `Continue` applies request headers before next filter -- response header effects accumulate -- `Respond` short-circuits later filters -- append/set header modes behave correctly +- ordinary filters run in registration order over `RedactedRequestView` +- DataDome alone receives the sealed typed security view +- `Continue` applies only validated owner-overlay operations before publisher + origin; another filter never observes them +- `Respond` short-circuits later filters and discards ordinary batches under the + hook's security ordering +- generic operations cannot express reserved security names or cookies ### DataDome Config Tests @@ -710,6 +814,8 @@ Update after implementation to describe: - protection disabled does not require server-side key secret store/name fields - protection enabled requires non-empty server-side key secret store/name fields - protection fails open when the configured server-side key secret cannot be read +- legacy/free-form `protection_api_origin`, session-header, and unknown security + fields fail startup - invalid regex fails startup - injection disabled allows empty `client_side_key` - injection enabled with empty `client_side_key` emits no head insert and does @@ -729,15 +835,30 @@ Update after implementation to describe: ### Payload Tests - form encoding is correct -- empty fields are omitted -- `ClientID` comes from `X-DataDome-ClientID` before cookie - _(superseded for v1: cookie-only — the header is stripped and never - used for the vendor payload, hook spec §4a)_ -- `X-DataDome-X-Set-Cookie` is sent when header-based ClientID is used - _(superseded for v1: never sent, hook spec §4a)_ +- empty source-header fields are omitted while mandatory `ClientID` remains + present as an empty value +- the outbound authority/path is exactly the core-owned HTTPS endpoint and 3xx + responses are never followed +- `Request` contains normalized path only, with query/fragment absent, and + `Referer` contains origin only +- `IP`/`Port` come only from trusted connection metadata; their absence skips + the call, and raw `true-client-ip`, `x-forwarded-for`, and `x-real-ip` values + never enter the payload or `HeadersList` +- `ClientID` comes only from a single unambiguous `datadome` cookie +- incoming `X-DataDome-ClientID` is stripped and + `X-DataDome-X-Set-Cookie` is never sent in cookie-mode v1 - `datadome` cookie is parsed safely -- long fields are truncated according to configured limits -- request headers list is generated deterministically enough for tests +- repeated list-valued source headers normalize in received field-line order + with literal `, ` separators, while repeated singleton, + `authorization`, or `content-length` headers skip the call without choosing + first/last; empty and comma-containing values match the normative allowlist +- multiple cookie field lines use the normative `; ` join for `CookiesLen` and + parsing; duplicate or malformed `datadome` pairs leave mandatory `ClientID` + empty and expose no other cookie value +- long fields are truncated according to the one normative allowlist +- the cross-adapter repeated-field corpus produces byte-identical normalized + form fields, lengths, `HeadersList`, and reject/omit outcomes; an adapter + without that capability cannot enable protection ### Response Classification Tests @@ -748,8 +869,8 @@ Update after implementation to describe: - `5xx` fails open - pointer headers are not forwarded - request enriched headers are applied to allowed requests -- downstream headers are applied to final responses -- `Set-Cookie` appends instead of replacing +- admitted security fields are applied atomically before final invariants +- the typed `datadome` cookie never overwrites another cookie name ### Route Tests @@ -757,8 +878,9 @@ Update after implementation to describe: - auth challenge short-circuits before DataDome - DataDome challenge bypasses publisher origin - allowed DataDome response enriches request before publisher origin -- DataDome downstream headers apply to buffered responses -- DataDome downstream headers apply before streaming response headers commit +- DataDome security batches apply to buffered responses before final invariants +- DataDome security batches and final invariants both complete before streaming + response headers commit ## Acceptance Criteria @@ -775,12 +897,12 @@ passes. - [x] DataDome challenge responses return without contacting the origin. Covered by an adapter route test that returns the DataDome challenge response even with no publisher-origin fallback. -- [x] Allowed requests receive DataDome request-enrichment headers. Covered by a - registry test that applies DataDome-style request mutations before routing. -- [x] Final responses receive DataDome downstream headers/cookies. Covered by - adapter route tests for allowed and challenged responses. -- [x] `Set-Cookie` is appended, not coalesced or overwritten. Covered by pointer - header route tests for DataDome downstream cookies. +- [ ] Allowed-request enrichment conforms to the owner-scoped allowlist and + default-disabled ClientID exposure in the hook spec. +- [ ] Final responses use the hook spec's atomic security batch and final + invariant ordering on every adapter/path. +- [ ] The typed `datadome` cookie contract and complete response-pointer matrix + replace generic `Set-Cookie`/header mutation. - [x] Static assets and internal Trusted Server routes are excluded by default. Covered by adapter route tests for discovery and default static-extension exclusions. @@ -791,7 +913,14 @@ passes. - [x] GraphQL body parsing is not implemented in v1 and is clearly documented. - [x] Existing DataDome first-party proxy behavior remains unchanged. Existing DataDome proxy/rewrite tests pass as part of full workspace verification. -- [x] `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets --all-features -- -D warnings`, and `cargo test --workspace` pass after implementation. Verified on 2026-06-15. +- [ ] `cargo fmt --all -- --check` and the repository's target-matched test and + lint aliases pass after implementation: `cargo test-fastly`, + `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin`, + `cargo clippy-fastly`, `cargo clippy-axum`, + `cargo clippy-cloudflare`, `cargo clippy-cloudflare-wasm`, + `cargo clippy-spin-native`, and `cargo clippy-spin-wasm`. Do not use bare + `cargo test --workspace` or workspace-wide all-feature clippy for this + multi-WASM-target repository. ## Resolved Questions @@ -813,13 +942,14 @@ passes. deadline in v1. _(The hook spec §4a now adds the 3000 ms complete-response deadline on the monotonic clock with defined measurement points; both bounds apply.)_ -2. **Client metadata scope:** JA4 and H2 fingerprint values are sent only in the - form-encoded Protection API payload to DataDome. They are not forwarded to - the publisher origin or returned to the browser unless DataDome independently - returns mapped enriched headers. Include them in v1 when the platform exposes - them because DataDome recommends TLS fingerprints and these signals are - useful for distinguishing browser and automation network stacks. Omit the - fields when unavailable. +2. **Client metadata scope:** only JA4 may be optionally admitted to the + form-encoded Protection API payload, never publisher origin, browser, graph, + another integration, or raw logs. Availability is not authorization: omit + it unless `expose_host_fingerprints_to_vendor = true`. `TlsCipher` is omitted + because the platform exposes a negotiated cipher while the vendor field + means ordered offered ciphers; `H2Fingerprint` is not a documented + Protection API field. Admit no host evidence outside + `datadome-header-allowlist.md`. 3. **Challenge status source of truth:** follow the Protection API docs in v1: `301`, `302`, `401`, `403`, and `429` are challenge statuses when `X-DataDomeResponse` matches the HTTP status. diff --git a/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md b/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md index 82ff2a755..3edccf807 100644 --- a/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md +++ b/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md @@ -20,6 +20,7 @@ The command surface is: ts config init ts config validate ts config push +ts config gc ts auth login --adapter ts auth status --adapter @@ -131,8 +132,10 @@ ids = ["secrets"] default = "secrets" ``` -The initial `ts config push` only writes config-store entries. The `secrets` -store is declared for runtime/future use but is not written by this CLI spec. +The initial `ts config push` writes the immutable config object and stages its +settings candidate through the deployment-metadata capabilities specified in +§5. It does not write a secret-store entry. The `secrets` store is declared for +runtime/future use but is not written by this CLI spec. Platform store names are not stored in `trusted-server.toml`. They are resolved by EdgeZero via its environment overlay, for example: @@ -144,31 +147,214 @@ EDGEZERO__STORES__SECRETS__SECRETS__NAME=publisher-a-ts-secrets ## 5. Runtime payload contract -`ts config push` writes a single logical Trusted Server app-config blob by -default. It does **not** publish flattened per-setting entries. +`ts config push` publishes one logical Trusted Server app-config snapshot by +default. It does **not** publish flattened per-setting entries; each snapshot +is a new immutable versioned object as defined below. -| Key | Value | +| Logical root | Value | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `app_config` by default, or `--key ` when supplied | Serialized `edgezero_core::blob_envelope::BlobEnvelope` whose `data` is the validated Trusted Server settings JSON | +Publication is versioned, not an overwrite of one live blob. The adapter maps +the logical identity `(root, push_sequence)` to an immutable physical object; +the object is written once, read back, and hash-verified. The strong +policy/config/model activation register names the sole active object. Runtimes never +treat the mutable logical root's latest value as active configuration. This +indirection is required even for adapters whose native config store exposes +only `put`: a globally unique, never-reused sequence gives every publication a +new object, while the deployment-metadata CAS supplies the active pointer. + The envelope contains: - a version field owned by EdgeZero; +- `push_sequence: u64` constrained to `0..=2^53-1`, scoped once per deployment/application across all + logical config-blob keys, allocated + exactly once from Trusted Server's linearizable deployment-metadata + config-sequence register before publication; - the validated app-config JSON data; - a SHA-256 hash over EdgeZero's canonical JSON form of `data`; +- a sequence-binding hash + `SHA-256("tscfgseq1|" || push_sequence.to_be_bytes() || data_hash_bytes)`, + where the domain tag is UTF-8, the integer is unsigned 64-bit big-endian, + and `data_hash_bytes` is the 32 decoded bytes of the preceding hash; the + known-answer vector is in + `docs/superpowers/specs/revision-canonicalization-vectors.json`; - generation timestamp metadata. -Runtime loading must verify the envelope hash before constructing `Settings`. +Runtime loading must verify both the data hash and sequence-binding hash before +constructing `Settings`. +The sequence is metadata, not part of `data`, and is cryptographically bound +to that exact data by `sequence_binding_hash`; a future envelope signature +signs that binding hash rather than the data hash alone. Allocation may leave a gap if publication fails but +never reuses a value. Concurrent pushes CAS the sequence register; a loser +re-reads and retries. A rollback republishes old `data` with a new sequence. +Allocation at the portable maximum is a hard deployment error, never wrap or a +larger JSON integer. +An adapter without the deployment-metadata allocator rejects multi-instance +config/policy activation; ordinary config-store `put` is not treated as CAS. If an adapter must split a large envelope to satisfy platform limits, the entry -at the logical key may be an adapter-owned pointer that identifies chunks. The -adapter/runtime loader must reconstruct and verify the envelope before exposing -settings to application code. +for that immutable logical identity may be an adapter-owned manifest that +identifies immutable chunks. The adapter/runtime loader must reconstruct and +verify the envelope before acknowledging readiness or exposing settings to +application code. A failed candidate or aborted activation leaves only an +unreferenced immutable object; a garbage collector may remove it after the +activation register's operational history and the permission spec §5.5 +time-based retention rules both permit removal, never while `active`, +`candidate`, operational history, or a retained activation-journal record +references it. Register eviction alone is never evidence that an object is old +enough to collect. The minimum journal/blob retention is 30 days and grows to +the longest processed-artifact, cookie-scope migration, rollback, or audit +horizon. Its not-before time uses the journal store's timestamp plus the +permission spec's 60-second promotion allowance; CLI/process wall time never +shortens it. `ts config gc` obtains the qualified journal inventory, computes the +reachable set, and refuses deletion if journal listing is incomplete or its +retention clock is uncertain. + +Before installing a candidate, the deployment controller obtains the +authoritative `{membership_epoch, members[]}` snapshot defined by the +permission spec §5.5. The CLI cannot synthesize, shrink, or override that set. +A membership change aborts and restages the candidate; `--force` never bypasses +unanimous readiness, the bounded serve-admission lease drain, the all-request +quiescence barrier, or serve admission. The candidate snapshots the positive +deployment-qualified `serve_admission_lease_bound_ms`. The draining CAS +records, from its qualified store clock at the CAS linearization point, a +`promotion_not_before_unix_ms`; that clock cannot satisfy the gate before the +full real interval has elapsed. The register rejects an early promotion even +if all member acknowledgments are present. Promotion is allowed only after the +controller has written and read-verified the immutable activation-journal +entry and the active-register CAS binds its ID as the new journal head. + +`ts config push` can stage and promote only a **settings candidate** and copies +the active model epoch, minimum binary generation, and row schema floor +unchanged. It cannot construct or promote a model candidate, and no `--force` +or config value can cross that boundary. The one-time +`pre_epic_v1` → `permissions_v2` transition is an authenticated deployment- +controller operation executed by the migration runbook: it stages the exact +model candidate, collects the fleet proof, and commits the single-register CAS +specified in permission §5.5. That controller operation is deliberately not a +general-purpose initial CLI command; exposing it later requires its own typed +command and cannot be emulated by raw config-store or metadata writes. +After that CAS, the same authenticated controller owns mirror completion: it +strong-reads active and `m00`; a missing or lower mirror is CAS-set to exactly +`active.row_schema_floor`, equality is an idempotent no-op, and an unreadable +mirror or failed CAS/read-verification remains closed for retry. The controller +then strong-reads and verifies exact equality before declaring the operation +complete. Retrying after a crash is idempotent. The operation never lowers +`m00` and never changes or authorizes active; a mirror higher than active is +rejected before any write as an inconsistency that fails closed for +investigation. + +### 5.1 Activation journal object and GC protocol + +The journal uses the same qualified immutable config-object service under the +reserved logical root `ts_activation_journal`, never the mutable app-config +root or the identity graph. Its logical object ID is lowercase +`SHA-256("tsactj1|" || RFC8785-JCS-UTF8(journal))`; adapters map +`("ts_activation_journal", object_id)` to a write-once physical object. The +object materializes exactly these fields and rejects unknown/missing fields: + +Every JSON number in the journal, including every number nested in an active +tuple, is an integer in `0..=9,007,199,254,740,991` (2^53 − 1). Booleans, +floats, negative values, and larger otherwise-valid `u64` values are rejected +before JCS; implementations may use wider internal integers but cannot emit +them here. Store-supplied lifecycle timestamps use the same portable range, +and addition that would exceed it fails closed. This profile makes the JCS +object ID identical in JavaScript, Rust, and every adapter rather than relying +on a language's larger integer type. + +- `schema_version = 1`; `attempt_id` as 32 lowercase hex characters from 16 + CSPRNG bytes, allowing a timed-out attempt to publish a new object; +- `candidate_incarnation` as the exact candidate's never-reused 32 lowercase + hex CSPRNG identity for `config`/`model`, or null for `checkpoint`; +- `previous_journal_id` and `pruned_through_journal_id`, each 64 lowercase hex + or null under the link/pruning rules below; +- `expected_activation_generation: u64` and `transition_kind` exactly + `config`, `model`, or `checkpoint`; +- `drain_attempt: u64`, which is the exact nonzero candidate drain attempt for + `config`/`model` and zero for `checkpoint`; +- `serve_admission_lease_bound_ms: u64`, the exact positive + deployment-qualified bound snapshotted by the candidate, and + `promotion_not_before_unix_ms: u64`, the exact store-clock gate written by + that drain attempt; both are zero only for a `checkpoint`; +- complete `displaced_active` and `activated_active` tuples from permission + §5.5, including settings bindings, policy identity, model epoch, minimum + binary generation, row schema floor, and logical activation generation; +- `membership_epoch: u64`, sorted unique `ready_members` and + `quiesced_members` using the stable member grammar, authenticated + `controller_id`, and `retain_for_ms: u64` constrained + to at least 2,592,000,000 and the longest applicable artifact, cookie-scope, + rollback, and audit horizon. + +The cross-language known-answer vector is +`docs/superpowers/specs/activation-journal-vectors.json`; every controller, +runtime verifier, and GC must reproduce both JCS bytes and object ID and reject +every numeric boundary vector. For the +first promotion, `previous_journal_id` is null only when the register head is +null and expected generation is zero. Every later config/model promotion must +name the exact current head and has null `pruned_through_journal_id`; the active +register CAS rejects any link/generation mismatch. For config/model entries, +`expected_activation_generation` must equal current active's logical +activation generation, and activated active must set it to that value + 1; +both member lists must equal the candidate snapshot's complete sorted member +list, `membership_epoch` must equal that snapshot's epoch, `drain_attempt` must +equal the candidate's current attempt and every quiescence acknowledgment, and +`candidate_incarnation` must equal every readiness/quiescence binding, +`serve_admission_lease_bound_ms` and `promotion_not_before_unix_ms` must equal +the candidate's exact drain fields, the admission-lease bound must be positive, +the immutable-store `created_at` for the journal must be at or after the +promotion-not-before time and no more than 60 seconds before the promotion CAS, +and the promotion CAS must independently enforce that its register store clock +has reached that time. These comparisons are defined only because the +activation register and immutable object service expose the same qualified, +authenticated Unix-millisecond time domain; adapters with incomparable clocks +fail activation qualification rather than comparing local timestamps. +Independently, `displaced_active` must equal current active and +`activated_active` must equal the candidate's computed post-CAS tuple. Overflow +is a hard error. A checkpoint +uses the current membership epoch, empty `ready_members` and +`quiesced_members` lists, and identical displaced and activated tuples +(including unchanged activation generation), with null +`candidate_incarnation`, `drain_attempt = 0`, +`serve_admission_lease_bound_ms = 0`, and +`promotion_not_before_unix_ms = 0`; it cannot stand in for fleet readiness or +quiescence. + +The immutable store returns authenticated `created_at_unix_ms` object metadata +from the shared qualified activation time domain and maintains a separate extend-only +`delete_not_before_unix_ms` lifecycle value. On every config or journal object +write, the adapter atomically initializes deletion protection to at least store +creation time + 30 days. For a promotion journal it extends protection for the +journal and both named blobs to at least `created_at + 60 seconds + +retain_for_ms` before the active CAS may bind the journal. These lifecycle +values can only increase. Therefore failed publication, aborted candidates, +losing journal attempts, and other unreferenced objects still have a store-clock +not-before value even though no successful promotion names them. + +The object service's qualification supplies snapshot-consistent complete +listing for both logical roots: a listing returns one snapshot generation and +opaque pagination token; every page is from that generation, and mutation or +expiry of the token forces GC to restart without deleting. GC first completes +the listing, traverses and verifies the journal from the active head, and builds +the active/candidate/history/journal reachable set. Missing objects, broken +hashes/links, unknown schema, cycles, incomplete pages, or uncertain lifecycle +metadata abort the run. Deletion then uses object-ID CAS and is allowed only +when the object is unreachable and its store-enforced not-before has passed. + +Journal pruning is an authenticated controller operation, never implicit GC. +Only when every record reachable from the current head is older than its full +retention horizon may the controller publish a `checkpoint` whose displaced +and activated tuples both equal current active, whose previous ID is null, and +whose `pruned_through_journal_id` is the old head. One register CAS verifies the +unchanged active tuple/generation and replaces only the journal head. The +checkpoint names and protects the current active blob; old journal objects +remain until their individual not-before values pass. Frequent activation can +therefore retain a longer chain but can never cut a still-required segment. Reserved future keys, not written in this initial spec: | Key | Future purpose | | --------------------- | --------------------------------------------------------------------- | -| `ts-config-signature` | Optional signature/DSSE envelope over the blob hash | +| `ts-config-signature` | Optional signature/DSSE envelope over the sequence-binding hash | | `ts-config-metadata` | Optional JSON metadata: version, published_at, valid_until, policy_id | Request-signing public/private state is intentionally out of scope for this @@ -322,24 +508,74 @@ Behavior: 1. Runs the same Trusted Server typed app-config validation as `ts config validate`. -2. Builds a `BlobEnvelope` from the validated app-config JSON. -3. Delegates read/diff/consent/dry-run/write behavior to EdgeZero's typed config - push primitive using: - - adapter from `--adapter`; - - manifest from `--manifest`; - - logical config store from `--store`; - - config entry key from `--key` or default; - - local mode from `--local`; - - dry-run mode from `--dry-run`; - - adapter runtime config from `--runtime-config`, when supplied. +2. Allocates `push_sequence` through the selected adapter's Trusted Server + deployment-metadata capability (dry-run reads and reports the next value + but does not reserve it). +3. Builds a `BlobEnvelope` from the validated app-config JSON and allocated + sequence. +4. Writes the envelope under the new immutable `(logical root, push_sequence)` + identity, reads it back, and verifies both hashes. It never + overwrites an object for an already allocated sequence. +5. CAS-installs the exact immutable object as the sole activation candidate; + the candidate has a new never-reused CSPRNG incarnation, binds current + active's complete tuple and logical activation generation, includes logical root, source version, data hash, + effective-config revision, policy digest, proposed policy ordinal, and the + unchanged active model fields. A competing or existing candidate makes the + CAS fail. The newly written unreferenced object is safe to collect later; it + never becomes live by being the most recently written blob. +6. Fleet readiness and controller promotion follow the permission spec §5.5. + Only promotion changes the active configuration. A config-only push still + takes this path but retains the policy ordinal. + +The underlying immutable-object read/diff/consent/dry-run/write behavior +delegates to EdgeZero's typed config push primitive using: + +- adapter from `--adapter`; +- manifest from `--manifest`; +- logical config store from `--store`; +- config entry key from `--key` or default; +- local mode from `--local`; +- dry-run mode from `--dry-run`; +- adapter runtime config from `--runtime-config`, when supplied. `--store` selects the logical config store for the Trusted Server config blob. `--key` selects the entry key within that config store. -`--dry-run` must not mutate platform or local adapter state. It should still -validate config, compute the local envelope, resolve the EdgeZero push target, -and report what would be written. Full config values should not be printed by -default. +`--dry-run` must not allocate a sequence, write an immutable object, or mutate +the activation register. It validates config, computes a provisional envelope +using the reported next sequence, resolves the EdgeZero push target, and +reports the immutable identity and candidate tuple that would be written. +Because another push may win, that sequence is explicitly advisory. Full +config values should not be printed by default. + +### 7.5 `ts config gc` + +```bash +ts config gc \ + --adapter \ + [--manifest ] \ + [--store ] \ + [--key ] \ + [--dry-run] \ + [--yes] \ + [--runtime-config ] +``` + +The command applies only §5.1's qualified immutable-object inventory and +deletion protocol; it never guesses physical keys or prunes the journal head. +It resolves the app-config root from `--key` (default `app_config`) and the +fixed `ts_activation_journal` root in the same selected config store, completes +one snapshot-consistent paginated inventory, verifies all hashes, lifecycle +metadata, active/candidate/history references, and the journal chain, then +computes unreachable objects whose store-enforced not-before has passed. +`--dry-run` prints only object IDs, roots, reasons, and lifecycle timestamps and +does not delete. Without `--dry-run`, deletion requires `--yes` or interactive +confirmation and uses the object-ID CAS from §5.1. Any uncertainty aborts the +entire run before the first delete; a partial platform deletion error stops the +run, reports exact completed IDs, and is safe to retry because reachability and +object-ID CAS are re-evaluated. Publishing a checkpoint is a separate +authenticated controller operation and is never an implicit side effect of +this command. ## 8. EdgeZero integration boundary @@ -350,8 +586,9 @@ There are two integration modes: 1. Pure lifecycle delegation for `ts auth`, `ts provision`, `ts serve`, `ts build`, and `ts deploy`. -2. Trusted Server config initialization/validation plus EdgeZero typed blob push - for `ts config validate` and `ts config push`. +2. Trusted Server config initialization/validation plus EdgeZero typed blob + push for `ts config validate` and `ts config push`, and the qualified + immutable-object inventory/CAS-delete path for `ts config gc`. Pure lifecycle delegate commands should call EdgeZero command/library APIs with the parsed CLI arguments and selected adapter. They should not perform Trusted @@ -359,7 +596,9 @@ Server config transformation, direct platform API calls, or adapter-specific command construction. `ts config push` is intentionally different: it validates Trusted Server app -config first, then delegates blob config-store writes to EdgeZero. +config first, then delegates blob config-store writes to EdgeZero. `ts config +gc` delegates listing, lifecycle metadata, and object-ID CAS deletion but owns +the Trusted Server reachability/journal validation in §5.1. Allowed implementation approach: @@ -501,9 +740,24 @@ contact real platforms in unit tests. `--dry-run`, `--no-env`, `--no-diff`, `--yes`, and `--runtime-config` to EdgeZero; - `--dry-run` performs no mutation; +- stages only a settings candidate bound to the complete active tuple and + activation generation; config push cannot alter model fields; - does not write secret-store entries; - does not print full config values by default. +### 13.6 `config gc` + +- complete snapshot pagination is required before the first delete; +- active, candidate, history, journal-chain, and protected-blob reachability + each prevent deletion; +- broken hash/link, cycle, expired pagination token, uncertain store clock, or + missing lifecycle metadata aborts with zero deletion; +- not-before boundary, dry-run, object-ID CAS conflict, partial-error retry, and + interactive/`--yes` confirmation follow §7.5; +- GC cannot publish a checkpoint or alter active/model state; +- the activation-journal known-answer vector verifies identically in runtime, + controller, and GC tests. + ## 14. Implementation sequencing 1. Update this spec and docs to the blob app-config contract. @@ -511,7 +765,9 @@ contact real platforms in unit tests. validation. 3. Collapse `crates/trusted-server-cli` to the thin downstream-CLI shape: direct EdgeZero args/run functions plus TS-owned `config init`. -4. Route `config validate` and `config push` through EdgeZero typed blob APIs. +4. Route `config validate` and `config push` through EdgeZero typed blob APIs; + add the qualified listing/lifecycle/object-CAS surface required by `config +gc` without platform-specific logic in Trusted Server. 5. Keep `edgezero_enabled` in `trusted_server_config` and restore any accidental coupling to `app_config`. 6. Keep runtime blob loading verified and avoid Trusted Server-owned platform diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index 6a0a5e94e..b2c508262 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -29,6 +29,11 @@ mutators to the outbound response for HTML document responses it processed. - `IntegrationRegistration::builder(ID).with_response_mutator(...)` registers a mutator; `IntegrationRegistry::apply_response_headers(...)` applies all registered mutators in registration order. +- Every registration carries a nonzero `behavior_revision: u32`, bumped for + any change to its response decision, operation semantics, declared read set, + or security field list. Integration IDs match + `[a-z0-9][a-z0-9-]{0,63}` and are unique. The registry revision hashes the + ordered registration list — order is behavior, so the array is never sorted. - **The mutator API is structured operations, not header-map access.** A mutator returns (or is handed a recorder for) typed operations — `append(name, value)` and `replace(name, value)` (v1 is headers-only; @@ -110,7 +115,39 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may computation → cache-key construction → body/metadata commit** — with three identity rules. Cache matching uses the **exact final publisher request** (post-overlay view; keying from the redacted view would - collapse personalized variants). **Every** `Vary`-nominated request value is stored **only as a keyed + collapse personalized variants). + + The final `Vary` name list has one cross-adapter grammar. Core collects every + `Vary` response field line from the snapshot plus mutation, parses each as an + HTTP comma-list, trims optional whitespace around every member, and requires + each non-empty member to be an RFC field-name token. It lowercases ASCII, + removes duplicates, and sorts unique names by unsigned ASCII byte order. An + empty member or invalid token introduced by a mutation invalidates that + batch. If the reverted snapshot itself is malformed, the invariant replaces + the final value with `Vary: *`, forces `no-store`, and writes no cache + artifact. `*` in either source likewise dominates every other member: the + normalized result is the single `*` and is uncacheable. For an ordinary + list, the wire response emits one lowercase `, `-joined value, while the + variant descriptor stores `vary_names` as the exact sorted JSON array of + lowercase strings. Thus field-line grouping, case, and input ordering cannot + produce different cache identities. + + A normalized name nominates one request header. Digest construction obtains + **all** values for that header from the exact final publisher request after + overlay, preserving received field-line order and value octets; it does not + comma-fold, trim, or split those request values. The `` bytes in the + HMAC input below are always the normalized lowercase ASCII name, and a name + is digested once even if it appeared repeatedly in `Vary`. Known-answer + normalization fixture: snapshot/mutation lines `Vary: X-Tenant , + Accept-Encoding` and `Vary: accept-encoding` produce wire value + `accept-encoding, x-tenant` and descriptor value + `["accept-encoding", "x-tenant"]`, then digest each nominated request + field's original instances in their received order. Fixtures also cover + differently grouped lines, case variants, duplicate names, empty members, + invalid tokens, `*`, absent versus present-empty request fields, and one value + containing a comma. + + **Every** `Vary`-nominated request value is stored **only as a keyed digest** — every value, not a sensitivity classification an unknown credential field could slip past: HMAC-SHA-256 with domain tag `tsvry1|`, over an input that encodes **presence, instance count, and @@ -120,29 +157,74 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may absence, and two members `a`,`b` collided with one member `a,b`), which could select a representation built under a different credential or tenant. Absent field → `tsvry1||a`; present → - `tsvry1||p|` then, per member in received order, + `tsvry1||p|` with `count` as ASCII decimal, then, per member in received order, `|:` with `len` the ASCII-decimal byte count. Output lowercase hex (64 chars). **The key is a deployable contract, not an implementation detail**: the setting `[cache] vary_digest_key_secret_name` names a platform-secret-store - entry (≥ 32 CSPRNG bytes); key ids match `[a-z0-9-]{1,16}` and - version every cache entry; startup **fails** when response caching is - enabled and the key does not resolve (digests are never computed - unkeyed); rotation introduces a new id while the previous stays - resolvable — entries under any resolvable id keep matching, others - miss and refill; the whole requirement is an adapter capability and - startup gate, since every caching adapter needs it. Known-answer - vectors under the all-zero 32-byte test key: + entry containing one versioned keyring JSON object: + `{ "schema_version": 1, "current_key_id": "", "keys": + [{ "id": "", "key_base64url": "" }] }`. `keys` is + sorted by `id`, contains 1..4 unique entries, rejects unknown fields, + and every value decodes to exactly 32 CSPRNG bytes. An id is derived, + not operator-invented: the first 16 lowercase hex characters of + SHA-256 over the raw key bytes; a supplied mismatch or one id bound to + different bytes is fatal. The current id must exist in the array. + Every cache entry stores its id; raw keys never enter config, cache + artifacts, logs, or metrics. Startup **fails** when response caching is + enabled and the keyring/current key does not resolve (digests are never + computed unkeyed). + + Lookup uses a stable per-representation **variant index** so the key ID is + discoverable before the variant artifact is addressed. The base index key is + the cache tuple excluding `Vary` values. Each bounded index descriptor stores + only the normalized final `Vary` name list, key ID, corresponding keyed + digests, artifact key, artifact expiry, and artifact revision tuple — never + raw request values. A reader loads the index, groups descriptors by key ID, + resolves each referenced key (one atomic keyring refresh if an ID is + unknown), recomputes the digests from the exact final publisher request, and + fetches only a descriptor whose complete name/digest tuple matches. A + still-unknown ID, malformed descriptor, missing artifact, expiry, or revision + mismatch is a miss for that descriptor, never a comparison under another + key. Multiple matching descriptors are corruption and make the entire base + lookup a miss with a metric; index order never chooses a winner. + + Publication writes and verifies the immutable artifact first, then CAS-adds + or replaces its complete descriptor in the index. Rekeying or a changed + `Vary` set inserts the new artifact/descriptor before removing the old + descriptor; a crash may leave a safely unreachable artifact or two + nonmatching descriptors but cannot point to a partial artifact. Index + capacity eviction removes expired descriptors first and otherwise the + least-recently-used complete descriptor; it never rewrites a digest under a + new key ID. This is the one meaning of “insert-new-then-index-update rekey” in + the capability matrix and 304 rules. + + Rotation atomically replaces the secret entry with a new valid keyring + containing the new current key **and all still-live previous keys**. Fleet + propagation may be mixed only in the safe direction: a process with the old + keyring can write/read the old id; the variant index exposes that id before + lookup, so a process seeing an unknown descriptor id refreshes the keyring + once, then treats a still-unknown descriptor as a cache miss. It never + guesses, probes with the current key, or computes unkeyed. A previous key may + be retired only after no unexpired index descriptor references it **and** the + maximum processed-artifact lifetime plus the adapter's qualified keyring + refresh bound has elapsed since it stopped being current. Key IDs are never + reused. Secret replacement atomicity, maximum propagation/refresh time, and + unknown-id refresh are explicit adapter capability cells and startup gates; + an unqualified adapter disables response caching rather than weakening the + grammar. Cross-adapter fixtures cover old→mixed→new propagation, unknown-id + refresh/miss, premature retirement rejection, and id/material mismatch. + Known-answer vectors under the all-zero 32-byte test key: `tsvry1|authorization|p|1|10:Bearer abc` → `c880c5e8c36febc0b1581c92f1d598fded34391626e67372ed63b2857d8a7b6b`; absent-field form `tsvry1|x-tenant|a` → `a2ae26cf529a5843a25f1448acc4e90016d4c1dce0ffda5662e3ac459433e1ab`. And a response derived from a request carrying an **identity-bearing TS - overlay** (the DataDome ClientID overlay) is forced `private, - no-store` unless an explicit per-overlay contract says otherwise — + overlay** (the DataDome ClientID overlay) is forced `private, no-store` + unless an explicit per-overlay contract says otherwise — the `Authorization` rule protects origin credentials, and this rule - protects the identity TS itself injected. Parsing itself is a **shared core parser with - fail-closed normalization**, not four adapter interpretations: + protects the identity TS itself injected. Parsing itself is a **shared core + parser with fail-closed normalization**, not four adapter interpretations: a `Cache-Control` value that fails the shared grammar has an **enumerated result, not a "most restrictive reading"** (restrictions are independent axes, so no single most-restrictive point exists): @@ -206,19 +288,25 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may Integration IDs are **startup-unique, enforced**: registry construction rejects a duplicate ID (current code silently coalesces, which corrupts attribution and budgets), with a duplicate-ID test in - the done-when. The registration also carries the **version the cache - tuple consumes, with a bump contract**: the version MUST change with - every output-semantic change of the mutator (review-checklist item; - where the mutator's behavior is fully declared configuration, the - version is a content hash of that declaration, making the bump - automatic), and the build-time invariant revision MUST bump with any - parser or merge-rule change — otherwise a deploy silently reuses old - post-hook finals and a new privacy restriction waits for cache - expiry. Until then the + the done-when. The registration's `behavior_revision` follows §2's bump + contract; configuration-dependent behavior is captured separately by the + effective-config digest. Model-only activation is separate from both, so the + **one cache revision tuple** contains exactly + `integration_registry_revision`, `effective_config_revision`, + `active_policy_digest`, `active_policy_ordinal`, `model_epoch`, + `activation_generation`, and `hook_invariant_revision` from the strong active + tuple at publication. Every processed artifact, mutation IR/read-set bundle, + variant descriptor, and variant-index update stores that complete tuple; a + lookup, local conditional, HEAD update, or 304 replay requires byte-for-byte + equality with current active. In particular, the `permissions_v2` model CAS + misses every `pre_epic_v1` artifact even though config/policy bytes did not + change. Core also carries a nonzero + `HOOK_INVARIANT_REVISION: u32`, bumped for every parser, merge, budget, + cache-artifact, or invariant semantic change. The cache tuple stores all + fields above, so a deploy cannot silently reuse old finals while a new + restriction waits for cache expiry. Until then the operation set is headers-only, and `Set-Cookie` is fully reserved. - `Set-Cookie` is fully reserved in v1 (§3 deferral). Violations are - rejected - at the operation layer (§2) and + Violations are rejected at the operation layer (§2) and logged at `warn` with the integration id. The reserved lists are single constants next to the definitions they protect, not duplicated in the hook. @@ -259,11 +347,27 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may separators), validated against core's budget at startup so a batch that passes core can never fail only on one adapter; a snapshot already **over** the core budget before any mutation rejects every - mutation batch — the budget bounds additions and never bricks an - over-budget origin response, which passes through and is counted) - bounds the sum across integrations — enforced in registration - order, so which operations are rejected when a budget trips is - deterministic. + ordinary batch — the budget bounds additions and never bricks an + over-budget origin response, which passes through and is counted; + security follows the replacement/reserve rule below) + bounds the sum across integrations. The budget has normative priority + partitions: ordinary mutators may consume at most **112 headers / 24 KiB**, + reserving 16 headers / 8 KiB for the core-owned security channel. Ordinary + batches remain registration-ordered within their partition. A security + `Continue` batch uses the reserve and, if necessary, evicts whole accepted + ordinary batches in reverse registration order until it fits; it never + removes half a batch and never drops origin fields. Ordinary output can + therefore never crowd out security effects. If the immutable origin head + plus the security batch alone exceeds the full budget, the security batch + is rejected atomically and the request follows the documented security + fail-open path with a dedicated metric — origin fields are not silently + sacrificed. A security `Respond` owns a replacement + response: all ordinary mutation batches are discarded and the challenge + is validated against the full 128-header / 32-KiB budget. A base publisher + response already over the full budget still passes unchanged, but no + ordinary mutation applies; security `Continue` applies only if the final + response fits after all ordinary batches are removed. These are separate + outcomes and metrics. | Adapter | Header-count / total-bytes ceiling (capability cell) | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | @@ -274,7 +378,29 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may A recorded cell below core's 128-header / 32 KiB budget is a startup error (shrink the core budget or raise the ceiling — never a silent - per-adapter divergence). Each mutator receives an **immutable, redacted snapshot of the + per-adapter divergence). + + Hook/cache eligibility has the following concrete adapter cells; any + `qualification-pending` cell fails startup when the depending feature is + selected: + + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | --------------------------------------------------- | ------------------------------ | -------------------------------- | + | Runtime secret lookup for Vary-HMAC/DataDome | wired secret store; qualify key-rotation behavior | dev secret binding required | qualify Workers secret binding | qualify component secret binding | + | Persisted processed artifact + mutation IR/read sets | qualification-pending | in-process dev implementation required; non-durable | qualification-pending | qualification-pending | + | Atomic artifact/metadata entry commit | qualification-pending | implementation required | qualification-pending | qualification-pending | + | `Vary` variant index + insert-new-then-index-update rekey | qualification-pending | implementation required | qualification-pending | qualification-pending | + | DataDome field-line order, trusted IP/port, fixed HTTPS backend/no-redirect, and exact form limits | qualification-pending | qualification-pending | qualification-pending | qualification-pending | + | SecurityUse JA4 request evidence | platform value available; exact-field/payload qualification and sign-offs 23/28 pending | unavailable | unavailable | unavailable | + + The qualification commit records storage lifetime, maximum object size, + concurrency semantics, torn-write behavior, and fault-injection evidence; + “platform has KV” is not a qualifying cell. + `expose_host_fingerprints_to_vendor = true` also requires a qualified + SecurityUse JA4 cell; unsupported or pending is a startup error, while the + default `false` remains portable. + + Each mutator receives an **immutable, redacted snapshot of the response head** (status and headers as of its turn, prior integrations' accepted operations applied) as its read context; it never holds a mutable reference (§2). Redaction is a security boundary, not @@ -285,13 +411,19 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may **excludes every `Set-Cookie` value and every reserved identity, consent, and privacy header value** (names may be listed as present; values are withheld). + Each registration also declares the complete set of response fields its + decision may read, including status as a distinguished input. Core records + the union with the accepted operation batch. Undeclared reads are a hard + conformance failure in tests; an integration unable to declare a complete + read set marks itself `revalidation = "refetch"`, which forbids IR replay + after any origin metadata change. Operations arrive as **attributed batches bound to a registration ID** — one batch per integration per response, ordered by registration, with the security channel's batch (§4a) ordered **after** ordinary response mutators — one global order, core finalization → ordinary mutators → security effects → invariant pass — so the - security layer's precedence over publisher-facing mutations holds - without a second ordering claim; the current flat effects vector satisfies neither + security layer's precedence over publisher-facing mutations holds through + both position and its reserved/response-owning budget rule; the current flat effects vector satisfies neither attribution nor budgets and is restructured accordingly. Validation and budgeting are **atomic per batch**: a batch that exceeds its budget is rejected whole (logged, attributed), never partially @@ -311,39 +443,107 @@ no-store` in that case regardless of the mutation; `max-age`/`s-maxage` may Which responses the hook runs on, enumerated so two implementations cannot diverge silently: -| Response | Hook runs? | -| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Processed HTML document (rewritten by TS) | Yes | -| Streamed processed document | Yes — operations apply to the header block before first byte | -| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | -| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | -| Redirect (3xx) | No | -| Error responses TS itself generates (4xx/5xx) | No | -| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: when the processed 200 was cached, its **final post-hook header set and the accepted mutation-operation batches that produced it (the persisted mutation IR)** were stored alongside the representation, **versioned by a fleet-stable tuple of four revisions** — the integration-registry revision (content hash over the ordered (integration ID, version) list), the config revision (the `tscfg1`-tagged effective-config digest), the policy revision (the (digest, activation-ordinal) pair from the permission spec's §5.5 register — the earlier "config store's globally assigned push version" definition is superseded), and the build-time invariant revision; local counters would collide across instances and binaries. There are two distinct 304 cases. A **locally generated conditional hit** (TS answers the client's conditional from its own fresh stored artifact) re-emits the persisted finals when **all four revisions** match, else cache-miss. An **origin-revalidation 304** is handled staged, then atomic, never in place: TS stages the 304's metadata off-record and diffs it against the separately stored **origin-side** metadata (origin validators and origin `Content-Length` describe origin bytes, not the rewritten artifact). (a) Any **byte-coupled representation field changed** (`Content-Encoding`, `Content-Type`, validators, digests — **changed, not merely present**: an ordinary 304 repeats the matching validator) → nothing publishes; the entry is invalidated and a full 200 is fetched and processed before any serve (RFC 9111 §3.2). (b) Only fields of the **enumerated safe-update set** changed — exactly: `Cache-Control`, the four enumerated CDN cache fields (under their reserved rules), `Expires`, `Date`, `Age`, `Vary`, and the registry-admitted mutable fields; **this set is the one definition of "cache-relevant fields," stated once** — → the new finals are **derived deterministically without re-running mutators** (mutators may be nondeterministic and run only at fill time): replay the persisted mutation IR — whose append/replace/merge semantics are core-defined deterministic functions of the operations plus the base — over the updated origin metadata, then re-run the invariant pass; an entry lacking its IR (an older cache schema) is unsafe → full refetch. The updated origin-side metadata, re-derived finals, and IR publish in **one atomic entry commit**; a changed `Vary` rekeys by **insert-new-entry-then-update-index ordering**, so a torn state yields a miss, never a wrong hit — single-entry atomic commit and this rekey discipline are adapter capability cells. (c) Nothing changed → re-emit as in the local-hit case. Artifact absence or a revision mismatch makes the recovery fetch **unconditional — every conditional field is stripped, the client's and TS's own alike** (forwarding the client's condition could return another 304 TS holds no usable bytes for); TS processes the full 200 under current revisions, then evaluates the client's **complete precondition set per RFC 9110 §13 against the new processed validators** — a failing `If-Match` or `If-Unmodified-Since` yields `412`, a matching `If-None-Match` or `If-Modified-Since` yields `304`, anything else the full response. `Set-Cookie` and validators follow ordinary 304 rules and are never replayed. Absent metadata → cache miss | -| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals stored as a **distinct head-only artifact type that never satisfies a later GET** — when a stored GET artifact exists a HEAD may **update** it only when the comparison — made against the separately stored **origin-side** metadata, never the processed artifact's (origin and rewritten lengths differ by construction, so the processed length is never the comparand and HEAD metadata never touches processed-side headers) — finds validators and origin `Content-Length` matching and no byte-coupled representation field changed (RFC 9111 §4.3.5); qualifying updates land origin-side under the same atomic-commit discipline as the 304 rules, and any mismatch **invalidates** the stored GET artifact rather than updating it | -| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | +In this table, **persisted post-hook finals** means the cache-safe ordinary +artifact only: origin metadata plus accepted ordinary mutation IR and the +cache/privacy invariant result, with `Set-Cookie`, core request-specific +identity fields, security-channel effects, and origin validators excluded. +The security request filter evaluates every request before cache selection; a +fresh `Respond` bypasses the artifact, while a fresh `Continue` batch is +applied to the persisted ordinary artifact and the invariant pass reruns before +emission. This applies equally to ordinary hits, local conditionals, +origin-revalidation 304s, and HEAD. Therefore "`Set-Cookie` is never replayed" +means never replayed from storage; a freshly validated per-request typed cookie +operation may still emit on that response. Security `Respond` outputs are +always `private, no-store` and never become artifacts. + +An **unconditional recovery fetch** first saves the client's preconditions for +later local evaluation, then removes every upstream conditional/range field +(`If-Match`, `If-None-Match`, `If-Modified-Since`, `If-Unmodified-Since`, +`If-Range`, and `Range`) so origin must return full bytes. TS processes that +200 under current revisions, constructs the processed validator, and only then +evaluates the saved client preconditions as the authoritative server for the +transformed representation. Another bodyless origin 304 can never satisfy +recovery. + +The 304 **safe-update set** is exactly `Cache-Control`, `Expires`, `Date`, +`Age`, `Vary`, `Surrogate-Control`, `CDN-Cache-Control`, +`Cloudflare-CDN-Cache-Control`, and `Edge-Control`. The phrase +"registry-admitted mutable fields" in the matrix denotes an empty set in v1; +adding any name requires a reviewed spec/registry revision and conformance +fixture, never a runtime wildcard. + +| Response | Hook runs? | +| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Processed HTML document (rewritten by TS) | Yes | +| Streamed processed document | Yes — operations apply to the header block before first byte | +| Pass-through proxy response (not processed) | No — TS is a transparent proxy for it | +| Served-from-cache processed document | Serves the **persisted post-hook finals** captured at cache fill (revision-versioned; mismatch = miss) — mutators run at fill/processing time only, so a normal hit and a conditional hit return identical policy metadata **by construction**, with no purity or determinism assumption about mutators | +| Redirect (3xx) | No | +| Error responses TS itself generates (4xx/5xx) | No | +| `304 Not Modified` for a processed representation | **Persisted-metadata pass**: a cached processed 200 stores its final post-hook headers, accepted mutation-operation batches, and the union of every mutator's declared response-field read set (the persisted mutation IR), versioned by §3's complete cache revision tuple, including model epoch and logical activation generation. A local conditional hit re-emits persisted finals only when the complete tuple matches current active. An origin-revalidation 304 is staged and diffed against separately stored origin-side metadata. (a) Any byte-coupled field changed (`Content-Encoding`, `Content-Type`, validators, digests) → invalidate and fetch/process a full 200. (b) A changed metadata field that intersects any persisted mutator read set, or an artifact/mutator lacking a complete read-set declaration, is also unsafe → full 200 refetch and ordinary hook execution; deterministic replay of old operations cannot stand in for re-evaluating a decision made from changed inputs. (c) If every changed field is outside every declared read set and belongs to the enumerated safe-update set (`Cache-Control`, reserved CDN cache fields, `Expires`, `Date`, `Age`, `Vary`, registry-admitted mutable fields), replay the persisted deterministic operations over updated origin metadata and rerun invariants. Updated origin metadata, finals, IR, read sets, and complete revision tuple publish in one atomic entry commit; changed `Vary` uses insert-new-entry-then-index-update ordering. (d) No change → re-emit persisted finals. Artifact absence or any tuple mismatch triggers an unconditional recovery fetch so TS obtains bytes. For processed-document GET/HEAD routes, TS is explicitly the authoritative server for the transformed representation: after processing the full 200 it evaluates RFC 9110 §13 preconditions against **processed** validators; this is not evaluation of origin validators by an intermediary cache. Other methods are never eligible for this recovery path. `Set-Cookie` and origin validators are never replayed. Absent metadata → cache miss | +| `HEAD` of a processed document | **Serves the persisted GET artifact when one exists** — parity with GET/304 by construction, since RFC 9111 §4.3.5 lets HEAD metadata update a stored GET response and mutators are not required to be deterministic; with no persisted artifact, HEAD is processed like a GET (headers only) and its finals stored as a **distinct head-only artifact type that never satisfies a later GET** — when a stored GET artifact exists a HEAD may **update** it only when the comparison — made against the separately stored **origin-side** metadata, never the processed artifact's (origin and rewritten lengths differ by construction, so the processed length is never the comparand and HEAD metadata never touches processed-side headers) — finds validators and origin `Content-Length` matching and no byte-coupled representation field changed (RFC 9111 §4.3.5); qualifying updates land origin-side under the same atomic-commit discipline as the 304 rules, and any mismatch **invalidates** the stored GET artifact rather than updating it | +| Informational `1xx`, `204`, `205`, `206` | No — enumerated so adapters do not infer independently | This deliberately narrows #782's general "outbound response" phrasing to processed documents (§6). ## 4a. The security channel — normative closed boundary -The security channel (today: DataDome) is not a general exception; every -degree of freedom is closed: +The security channel (today: DataDome) runs under a distinct, typed +`SecurityUse` authority rather than the advertising permissions P1/P4. +`SecurityUse` permits bot/fraud evaluation and challenge continuity only; +it never authorizes TS-controlled advertising identity, graph linkage, partner egress, +other integrations, or general raw-value observability. Request-scoped raw +security evidence may be disclosed only to the fixed DataDome Protection API +endpoint and only from that integration's explicit field allowlist; it is not +persisted in the identity graph, exposed to publisher origin or other +integrations, or emitted in logs. It carries its own configured retention and +deletion path. An advertising opt-out does not erase a +strictly security-scoped identifier, while an authenticated deletion request +or expiry under the security retention policy does. This is not a general +exception. The path-only `Request` remains publisher-originated data and is +explicitly covered by vendor retention/DSR sign-off; query strings and full +referrers are never in the security view. Every degree of freedom is closed: + +- **Host evidence is not a back door to the deferred device provider.** + `[device] provider = "fastly"` remains startup-rejected and no JA4-derived + classification is stored. If DataDome's Protection API is allowed to receive + request-scoped `TlsProtocol`/`JA4` evidence, its registration enumerates each field, + proves vendor necessity and payload bounds, and keeps it ephemeral under + `SecurityUse`; those exact consumers and fields are part of product/vendor + sign-offs 23/28. `TlsCipher` is omitted because the platform value has + different semantics from the vendor field, `H2Fingerprint` is not in the + vendor contract, and all other host evidence is omitted. +- **Deletion and retention name the system boundary honestly.** TS stores no + server-side DataDome identifier mapping. On an authenticated TS deletion + request it excludes the route from vendor validation, emits a typed + `datadome` cookie deletion, and sends no ClientID to DataDome for that + request; re-presentation retries deletion. A lost browser response can leave + the cookie until its configured `security_cookie_max_age`, which is the + bounded residual sign-off 23 accepts. This operation does **not** claim to + erase data already held by DataDome: vendor-side retention and data-subject + deletion require a named contractual/API procedure in the decision record. + If no such vendor procedure exists, operator documentation says so and may + not describe TS cookie deletion as vendor-data deletion. - **Typed security-cookie operation with a concrete lifecycle, not header strings.** The channel emits cookies only through a typed operation, and the registration is not a placeholder — for DataDome it pins, **aligned to documented vendor behavior where hardening was - not intended**: cookie name exactly `datadome`; `Domain` per - DataDome's own guidance (the module sets it; TS validates it does not - exceed the registrable domain, computed against the **vendored + not intended**: cookie name exactly `datadome`; one configured ownership + tuple for every set and deletion: path exactly `/` and + `security_cookie_domain = "host-only"` (default) or one explicit normalized + ASCII domain. Host-only mode requires the vendor `Domain` attribute to be + absent; explicit-domain mode requires it to equal the configured domain + exactly — TS never accepts a different domain and never rewrites one scope + into another. The explicit value cannot exceed the registrable domain, + computed against the **vendored Mozilla PSL snapshot** `docs/superpowers/specs/psl-snapshot-ref.md` — ICANN + private sections, IDNA-mapped; IP-literal or single-label hosts fall back to host-only), path `/`; `Secure` mandatory; `SameSite` configurable `Lax` (default) / `Strict` / `None` (`None` requires `Secure`), matching the vendor's endpoint options; - the returned `Domain` must additionally **domain-match the current + the configured/returned `Domain` must additionally **domain-match the current request host** per RFC 10025, the current cookie specification obsoleting RFC 6265 (domain-match and the PSL boundary check are separate requirements) and a vendor cookie using `Expires` is normalized to its Max-Age equivalent (both present → `Max-Age` wins, @@ -366,8 +566,10 @@ degree of freedom is closed: wall clock at parse time (the shared skew-bounded basis; a result of 0 is a deletion), and the 512-byte limit measures the **normalized** serialized `name=value` plus attributes in bytes; - `Max-Age` at most **31,536,000 seconds** (the vendor's one-year cap — - the earlier 396-day figure exceeded it); size ≤ **512 bytes** + `Max-Age` is capped by required operator configuration + `security_cookie_max_age` in the vendor-supported range 7 days through + **31,536,000 seconds** (one year); the returned cookie may be shorter but + never longer. There is no silent one-year default. Size ≤ **512 bytes** (DataDome's current Fastly-module limit; 4 KiB was ours, not theirs). Where the contract **is** deliberately narrower than the vendor — the spec-pinned pointer allowlist starting at ClientID-only against @@ -404,12 +606,35 @@ degree of freedom is closed: other integrations' request views, publisher-origin proxy forwarding, proxy/click/Testlight upstreams, auction/page-bids request serialization, and logs (redaction list) — each surface a tested row - of the inventory; only the security channel itself observes it; vendor egress goes only to DataDome - endpoints; deletion is always possible; and whether TS's own - destructive withdrawal also expires it is exactly the open half of - **sign-off item 23** — the carve-out is _pending ratification_, not - ratified, and the permission inventory's cookie deferral stands until - it closes. No other request filter inherits the cookie capability. + of the inventory; only the security channel itself observes it; vendor + egress goes only to the fixed DataDome Protection API authority and path; + redirects are not followed; deletion is always possible + through the `SecurityUse` lifecycle; and advertising withdrawal never + grants access to or reuses the identifier. No other request filter + inherits the cookie capability. +- **Cookie ownership makes deletion total for the scope TS creates.** While + DataDome is enabled, `datadome` is a security-owned name across the final + response: before the security batch applies, core removes and meters every + origin, core, cached, or ordinary-mutator `Set-Cookie` for that name; + unrelated cookie names remain separate field lines. Only the typed security + operation may emit it. Authenticated deletion emits the same configured + `(name, domain mode/domain, path)` tuple with `Max-Age=0`; it does not guess a + scope from the request cookie, whose wire form carries no Domain or Path. + Candidate validation rejects a change of domain mode/domain while the + previous active DataDome configuration can still have a live cookie. The + supported migration is disable + wait at least the previous + `security_cookie_max_age` + activate the new scope; a faster scope change + requires a separate bounded deletion-fan-out design. The permission spec + §5.5 whole-settings serve fence applies before cookie processing. A bounded + old-generation admission validation may survive only during the pre-promotion + drain; the register's promotion-not-before plus member quiescence proves it + and every admitted effect ended before the activation CAS. After that CAS, + no instance may emit, refresh, or delete a `datadome` cookie until it has + loaded and leased the exact new active tuple. A stale instance stops at serve + admission rather than extending the old scope. Fixtures cover origin + collision, host-only vs explicit-domain set/delete, attempted domain change, + and duplicate request cookies. The deletion claim therefore covers every + cookie this contract can create, not arbitrary pre-contract scopes. - **The incoming `X-DataDome-ClientID` request header is owner-only, like the cookie.** DataDome prioritizes the header over the cookie, so leaving it in the shared request would hand other integrations and @@ -446,20 +671,25 @@ degree of freedom is closed: priority rule exists in v1: the header session form (`X-Set-Cookie`) is matrix-governed as batch-invalid, so "header form wins" is unreachable and deleted. -- **Request-header pointers are a positive, enumerated allowlist.** +- **Request-header pointers are a positive, enumerated allowlist with no + default publisher-origin identifier exposure.** "Documented enrichment headers" is not enforceable; the registration enumerates the exact names from the **checked-in allowlist file `docs/superpowers/specs/datadome-header-allowlist.md`** — spec-pinned - today to exactly **`X-DataDome-ClientID`**; every other `X-DataDome-*` + today to exactly **`X-DataDome-ClientID`**, admitted only when the + operator explicitly sets + `[integrations.datadome] expose_client_id_to_origin = true` (default + `false`); every other `X-DataDome-*` field is rejected until a reviewed commit adds it to that file ("documented enrichment set, listed one by one" without an actual list was a wildcard whose contents could change outside the spec) — - resolving what was a contradiction: - ClientID propagation is required by the existing DataDome contract - and test, and its identity-class nature is precisely why it applies - only to an **owner-scoped publisher-upstream overlay**, never the - shared request that later integrations read, with its vendor egress - ratified under sign-off 23. Everything else — authentication, + resolving what was a contradiction. When the opt-in is false, the + vendor-returned ClientID is discarded and the publisher origin is not + an identifier observer. When true, it applies only to an owner-scoped + publisher-upstream overlay, never the shared request; startup logs the + additional consumer, operator documentation must disclose its purpose + and retention, and a fixture proves no other surface can read it. + Everything else — authentication, `Cookie`, `Forwarded`/`X-Forwarded-*`, other identity, consent, and routing-authority fields — is rejected by name and by class: a compromised endpoint must not replace origin credentials, inject @@ -516,8 +746,9 @@ degree of freedom is closed: `X-DD-*`, and `Pragma`, letting one conforming implementation accept the vendor's documented `Set-Cookie X-DD-B` allow-example while another invalidated the whole batch). No `X-DD-*` wildcard exists: - every name is enumerated, `X-DD-B` included (drop-individually in - cookie mode — dropping it does not break cookie sessions). The + every name is enumerated, `X-DD-B` included and forwarded exactly once + as the vendor's documented cookie-mode browser-response signal; it is + never copied into publisher-upstream or another integration. The documented vendor responses (both the challenge example and the `Set-Cookie X-DD-B` allow example) are **verbatim fixtures asserting the decision survives** and exactly the mapped fields emit. @@ -550,11 +781,15 @@ degree of freedom is closed: ## 4. Done-when (from #782, sharpened) 1. Trait + builder + registry application, each public item documented. -2. **The pre-existing `RequestFilterEffects.response_headers` channel - remains a distinct, core-owned security channel — §4a defines its - closed boundary.** Folding it into this hook would break its one real - consumer: DataDome sets headers **and cookies** on 200, 301/302, 401, - 403, and 429 responses — response classes (§3a) this hook never runs +2. **The old generic `RequestFilterEffects.response_headers` channel is + removed.** DataDome uses the separate sealed, core-registered + `DataDomeSecurityRequestFilter` and typed `DataDomeSecurityEffects` defined + by its design; §4a defines that closed boundary. Generic request filters + receive only `RedactedRequestView` and ordinary attributed effects and + cannot express the security view, owner overlay, cookie operation, or + reserved security header. The dedicated channel is necessary because + DataDome sets headers **and cookies** on 200, 301/302, 401, 403, and 429 + responses — response classes (§3a) the ordinary response hook never runs on, with cookie emission v1 reserves. 3. **At least one real consumer ships in the same PR** — an existing integration registering a mutator for a real need (or, failing a real @@ -576,7 +811,14 @@ degree of freedom is closed: origin's cache restrictions + public replacement → restriction preserved (pass-through responses never run the hook, §3a); a cache-hit serve re-applying mutations without weakening the stored classification; a `Vary` mutation neither - dropping core-required values nor bypassing the snapshot; each of the four enumerated CDN fields (`Surrogate-Control`, + dropping core-required values nor bypassing the snapshot; the §3 + normalization fixture proves field-line grouping/case/order collapse to one + sorted lowercase descriptor, repeated names digest once, request value + instances retain octet/order boundaries, malformed/empty mutation members + reject, a malformed snapshot becomes `Vary: *` plus `no-store`, and `*` + writes no artifact; a `pre_epic_v1` artifact/index descriptor is a miss + immediately after the model-only `permissions_v2` activation CAS even when + config and policy digests are unchanged; each of the four enumerated CDN fields (`Surrogate-Control`, `CDN-Cache-Control`, `Cloudflare-CDN-Cache-Control`, `Edge-Control`) individually stripped; and a rejected `Content-Encoding` mutation. diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 345288a03..660e966e1 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -43,9 +43,13 @@ source-interface is **explicitly deferred**, not silently dropped — §10 records the divergence, and adding a source later means adding a grant- or opt-out-class input to §4's taxonomy, not a new resolution algorithm. -Scope: the model governs decisions Trusted Server makes. Downstream RTB -partners receive the full, unmodified regulatory context and make their own -compliance decisions. +Scope: the model governs decisions Trusted Server makes. A downstream +protocol receives the full, unmodified regulatory context only when that +protocol normatively requires it and the destination is an authorized +privacy-signal consumer. Raw consent strings are request-scoped transport +data, not general identity metadata: ordinary identity rows retain the +normalized per-permission provenance and a digest, never the raw string +(§7; providers spec §6.3). ## 2. Vocabulary: enforced permissions only @@ -122,9 +126,10 @@ overrides. Each permission resolves to an **acquisition rule**: ```toml # Illustrative schema example — NOT the shipped policy. The shipped -# example (trusted-server.example.toml) pairs these groups with the most -# protective rules.default; the permissive default below demonstrates the -# reserved key. +# example (`trusted-server.example.toml`) defines a `protective-default` +# group (`regime = "gdpr"`, `default = "requires_signal"`) and points +# `rules.default` to it; the permissive default below demonstrates the +# reserved key for the migration-preserving posture. [permissions.groups.gdpr-eu] regime = "gdpr" default = "requires_signal" @@ -141,14 +146,20 @@ default = "requires_signal" regime = "none" default = "granted" +# Optional explicit entries live under one typed child map. Quoted permission +# IDs are required because they contain hyphens. An explicit entry replaces +# this group's default for only that permission. +[permissions.groups.us-opt-out.permissions] +"store-on-device" = "requires_signal" +"select-personalised-ads" = "requires_signal" + [permissions.rules] FR = "gdpr-eu" -# US privacy gating applies per configured privacy state, matching today's -# state-list behavior; country-level US traffic (a Wyoming request, or one -# whose geo provider yields no region) stays non-regulated. One US/ -# rule per configured privacy state: +# US privacy gating has a protective country-wide floor. State rules may +# tighten or specialize it, but country-only geo and regionless US traffic +# never fall through to a non-regulated grant. "US/CA" = "us-opt-out" -US = "non-regulated" +US = "us-opt-out" # Overrides name explicit acquisition rules — no +/- sigil syntax; TOML # expresses the target state directly. "US/CO" = { group = "us-opt-out", overrides = { select-personalised-ads = "requires_signal" } } @@ -158,8 +169,12 @@ US = "non-regulated" default = "non-regulated" ``` -A group's `default` covers unlisted permissions; a group may also name -permissions explicitly. Overrides map identifier → acquisition rule, so any +A group's `default` covers permissions absent from its optional +`permissions` child map. That map has the exact type +`BTreeMap`; direct permission-shaped keys on the +group object are unknown fields and rejected. An explicit map entry replaces +the group default for exactly that permission. A group without `default` must +list every vocabulary permission exactly once. Overrides map identifier → acquisition rule, so any target state (including `requires_signal`) is expressible — PR #838's `+`/`-` sigil scheme could not express "requires a signal", the most common real-world override. @@ -191,7 +206,14 @@ Validation rejects: house rule-key format corresponding to ISO 3166-2 `US-CA`; - references to permissions outside the enforced vocabulary (§2); - references to undefined groups, and groups missing the `regime` class; +- group identifiers outside `[a-z0-9][a-z0-9-]{0,63}`; the lowercase ASCII + grammar makes reference equality, JCS bytes, logs, and metrics identical on + every runtime; - groups that neither list every permission nor provide `default`; +- duplicate explicit permission entries, unknown permission IDs, a direct + permission-shaped group key outside the `permissions` child map, or a value + outside `granted | denied | requires_signal`; explicit entries take + precedence over `default` and there is no merge-by-file-order behavior; - a present `[permissions]` section without a `rules.default` entry (§5.4 depends on it existing — its absence must be a validation error, not a runtime surprise); @@ -227,13 +249,12 @@ explicit, commented exceptions in the test — never silent. Both legacy lists are in scope, not only the GDPR one — and the US check is region-shaped: **every configured `consent.us_states.privacy_states` entry must have a matching `US/` rule**, and the country-level `US` rule must -resolve non-regulated (today applies privacy gating only to the configured -states), or the divergence is an explicit commented exception. An adapter -whose geo provider cannot resolve regions degrades **intentionally and -declaredly**: regionless US traffic hits the country rule — non-regulated, -today's behavior for non-privacy-state traffic; an operator preferring -protective country-wide gating writes `US = "us-opt-out"` as their own -declared choice. +resolve to at least the `us-opt-out` protective floor; a more permissive +country fallback is invalid. An adapter whose geo provider cannot resolve +regions therefore degrades intentionally to country-wide US privacy gating, +never to `non-regulated`. A provider that can resolve a region still uses +the matching `US/` rule first, and an operator may configure a +stricter country rule. ### 3.5 Shipped-table coverage @@ -243,7 +264,11 @@ nothing in PR #838's validation covered: a mistyped country key (`DL:` for `DK:`) parses cleanly, starts cleanly, and silently drops a member state to the fallback rule. Countries intentionally unlisted are governed by the `rules.default` entry (§3.2); the example policy documents that fallback -inline, and ships it as the most protective baseline. +inline and ships the exact `protective-default` group described in §3.2 +(`regime = "gdpr"`, every permission `requires_signal`). The separate +migration-preserving fixtures deliberately point `rules.default` to +`non-regulated` and declare that divergence (migration spec §5); the example +and migration fixture are not aliases. ## 4. Signal precedence — normative @@ -252,21 +277,21 @@ opt-out) cannot reproduce today's US behavior, where no-signal traffic is blocked but an **explicit non-opt-out** value grants: - **Opt-out signals**, in two subclasses assigned by the §4.5 mapping: - **destructive** opt-outs (GPC; sale opt-outs; USP opt-out) revoke and - trigger withdrawal; **non-destructive** opt-outs (sharing, - targeted-advertising) revoke the permissions they map to but never - destroy the stored identity — a targeted-ads choice must not tombstone. - Both subclasses are honored **globally**, not only in the jurisdictions whose law defines - them — a deliberate, more-protective simplification: scoping a browser's - explicit opt-out to a geolocation guess would honor it for some visitors - and ignore it for others based on IP evidence. (For jurisdictions outside - US states this is a declared behavior change; migration spec §2 records - it.) + **use opt-outs** (GPC, sale, sharing, targeted-advertising, and USP sale + opt-out) suppress the permissions they map to and persist negative authority + but do **not** destroy the first-party identity merely because sale or + sharing stopped; **destructive withdrawal signals** are limited to an + explicit storage-consent withdrawal, an authenticated deletion request, + or the TCF Purpose-1 refusal conditions of §4.2. Both subclasses are + honored **globally**, not only in the jurisdiction whose law defines them — + scoping an explicit choice to a geolocation guess would honor it for some + visitors and ignore it for others. “Global” follows the identity's use; + it does not broaden a sale/sharing choice into deletion. - **Grant signals** (affirmative permission): a decodable TCF record consenting to the purpose; an **explicit GPP non-opt-out value** (e.g. - `sale_opt_out = false`); a **US Privacy string present and not opting - out** — including the "not applicable" flag, which today's tests pin as - allowing. Grant signals are what let a `requires_signal` US rule + `sale_opt_out = false`); a **US Privacy string present with an explicit + `N` value**. _Not Applicable_, missing, reserved, unknown, and unsupported + values never grant. Grant signals are what let a `requires_signal` US rule preserve today's "no signal → block, explicit non-opt-out → allow" behavior, which neither `granted` nor a TCF-only grant class could express. **Which grant evidence a rule accepts is regime- and @@ -285,7 +310,7 @@ blocked but an **explicit non-opt-out** value grants: the migration matrix. Auction dispatch blocking separately would not help; identity use would already be authorized. Opt-out signals and refusals remain regime-agnostic (global), as before: scoping applies - only to what can _grant_, never to what can _revoke_. + only to what can _grant_, never to what can _suppress_. - **Refusals**: a decodable TCF record refusing the purpose. A refusal is neither a grant nor an opt-out — it blocks acquisition (precedence 3) @@ -294,10 +319,11 @@ blocked but an **explicit non-opt-out** value grants: **Precedence, highest first:** 1. Policy `denied` — never set, regardless of any signal. -2. **Opt-out signal — always revokes**, regardless of any consent record - present. A GPC header revokes `store-on-device` and +2. **Opt-out signal — always suppresses its mapped use**, regardless of any + consent record present. A GPC header suppresses `select-personalised-ads` even when an accompanying TCF string consents to - them. _(This is the rule PR #838 inverted: its resolution returned from + it; it does not itself revoke `store-on-device` or request deletion. + _(This is the rule PR #838 inverted: its resolution returned from inside the TCF branch before ever reaching the opt-out check, so a consenting CMP string made the browser's GPC signal a no-op — a CCPA-facing regression. The pre-existing tests pinning this rule — @@ -350,12 +376,11 @@ group label, since a group can mix rules across permissions. The triggers, exhaustively — nothing else withdraws: -1. **A destructive opt-out signal (per §4.5's destructive column: GPC, - sale opt-outs, USP opt-out) withdraws in every jurisdiction, whatever - the baseline.** Non-destructive opt-outs (sharing, - targeted-advertising) never trigger this — they revoke acquisition - only. (For US states this preserves today's behavior; elsewhere it is - the declared change of §4's global-opt-out rule.) +1. **An explicit storage withdrawal or authenticated deletion request + withdraws in every jurisdiction, whatever the baseline.** GPC and + sale/sharing/targeted-advertising opt-outs are use restrictions, not + deletion requests: they persist suppression for their mapped permissions + but never trigger family revocation by themselves. 2. **A TCF record refusing `store-on-device` withdraws iff the baseline is `requires_signal` or `denied` — and only when the refusal is carried by the live request.** A persisted-KV consent record @@ -382,8 +407,9 @@ The triggers, exhaustively — nothing else withdraws: 4. **Absence of signal never destroys identity.** A visitor who has not yet made a choice is never stripped of an existing identity. -Withdrawal checking follows §4 precedence: an opt-out signal triggers -withdrawal even when a consenting TCF record is present. +Withdrawal checking follows §4 precedence: a destructive signal from the +exhaustive list above triggers withdrawal even when other evidence grants; +a use opt-out suppresses only its mapped use and never enters this path. `ec_storage_withdrawn` (or its successor) gets direct unit coverage for every trigger above; in PR #838 the headline "withdrawal expires identity" behavior had no unit test at all. @@ -402,8 +428,8 @@ and the fail-closed marker: or breaks the protocol — **rows that lack the field derive it deterministically** as a function of (record kind, provider namespace, canonical graph key), per the providers spec §6.3 derivation (`tsfam1|` + record-kind byte + provider code + graph key). Determinism is the - point: a withdrawal arriving on the **first post-upgrade request** — a - GPC-carrying visitor whose v1 row has no family field and has never been + point: an explicit storage withdrawal arriving on the **first + post-upgrade request** — from a visitor whose v1 row has no family field and has never been backfilled — computes the same family ID that every future reader of that row computes, so the revocation record is discoverable even if the writer crashes before ever touching the member row — and the write is @@ -423,7 +449,7 @@ and the fail-closed marker: replacement (which today discards the original row's identity and metadata, making sibling discovery impossible). - **Negative authority has its own permission-exempt record, with a - complete transition contract.** A live refusal or opt-out must clear + complete transition contract.** A non-destructive refusal or use opt-out must clear prior positive provenance, but the row write that would do it requires `store-on-device` — which the refusal just unset — and identity rows may be eventually consistent. The **suppression record** @@ -432,10 +458,19 @@ and the fail-closed marker: **Creation is cause-aware and mostly read-free.** A live resolution whose outcome for a permission is unset writes suppression when the - cause is a **signal state** — refusal, non-destructive opt-out, - malformed-present — **unconditionally** — meaning independent of _prior positive authority_, never independent of **family admission** (every durable write still passes the providers spec §5 admission arms; for an observed v1 row the non-destructive sequence applies) — with no row read needed for the decision itself: conditioning + cause is a **signal state** — a refusal that is not destructive under + §4.2, a use opt-out, or malformed-present — **unconditionally** — + meaning independent of _prior positive authority_, never independent of **family admission** (every durable write still passes the providers spec §5 admission arms; for an observed v1 row the non-destructive sequence applies) — with no row read needed for the decision itself: conditioning on observing positive provenance through an eventually consistent row loses the race where a stale replica hides a just-committed grant. The + old rowless identity is the explicit boundary: its live refusal, + malformed-present state, or use opt-out still denies this request but creates + no `s`, `q`, `fam`, or `w` record. If P1 permits ordinary same-request + re-minting, the new graph-backed family's authority-state commit includes + that live suppression (or its negative intent) before the new cookie or + identity is usable; otherwise no durable state exists and the next + presentation is reevaluated. The `w` class remains destructive-withdrawal + only. The one cause that inherently needs prior state — applicable **absence** clearing a previously positive permission — uses a narrow **permission-exempt suppression-decision read** exposing only the @@ -462,38 +497,47 @@ and the fail-closed marker: when its evidence timestamp is **newer than or equal to** the stored entry's; ties resolve to the more restrictive state. So a delayed grant with `LastUpdated = 100` never clears a suppression whose - refusal carried `200`, while a genuine re-consent at `300` does. **Every suppression entry carries its own `valid_until`, derived from - its evidence class's TTL, and an expired entry is inert** — treated as - cleared without a write, lazily garbage-collected. Without this, an - expired TCF refusal under a `granted` baseline would deny forever: - normalization says an expired record is absent and "must not revoke - indefinitely", yet the surviving suppression would block the baseline - grant that same table promises — the two contracts now agree, in the - normalization table's favor. (Destructive opt-outs tombstone and need - no suppression longevity; non-destructive opt-out entries expire on - the consent-TTL horizon of the evidence that created them.) The + refusal carried `200`, while a genuine re-consent at `300` does. + **Suppression expiry is cause-specific.** Refusal, malformed, and absence + entries carry `valid_until` derived from the evidence or retired-authority + horizon and become inert at expiry. A valid use opt-out (GPC, + sale/sharing/targeted-advertising, or USP sale opt-out) is different: + it remains effective until a strictly newer, explicit, regime-accepted + opt-in/authorization clears it, or until deletion of the identity makes + the record unnecessary. Passage of a consent TTL alone never restores + sale/sharing or personalized-ad use. **"Strictly newer" requires an + authoritative order, not later receipt:** either a regime-accepted TCF + grant whose valid `LastUpdated` is after the opt-out evidence, or an + authenticated same-subject authorization action that commits a monotonic + authorization revision in the strong authority record. A bare GPP/USP + not-opted-out value has no authoritative timestamp and therefore cannot by + itself clear a persisted use opt-out; treating its new receipt time as + recency would let replay of an older string restore processing. This epic + does not invent an authenticated authorization endpoint: until a separate + approved flow supplies that revision, only qualifying authoritative TCF + evidence or identity deletion can clear such a suppression. The transition table (causes without an intrinsic timestamp — malformed records decode no `LastUpdated`, absence has no source — use their **observation timestamp**, server receipt on the shared clock basis within the skew window; cross-source comparison uses the authoritative timestamp where one exists, else the observation timestamp, ties restrictive), by stored cause: **opt-out from a timestamp-less - source** — within its lifetime, cleared only by a grant with an - authoritative timestamp newer than its observation; its lifetime is - the ordinary consent-TTL `valid_until`, at which it goes inert - automatically (**TTL-sticky** — the one rule chosen among three that - circulated: not user-sticky-forever, and not the migration spec's - former "irreversible artifact requiring administrative clear", which - is superseded; administrative clear remains an optional early exit — - sign-off 16 — **with one declared exception**: an opt-out arriving as - a restrictive _overflow_ while its source's replay history is - saturated inherits the live restrictive marker (first-overflow-pinned; - it outlives its epoch and can span epoch boundaries, providers spec - wire schema) and may receive less than a full lifetime, down to - nearly zero near the marker's expiry (the exception is carried by - sign-offs 16 and 31, and the alternatives — per-overflow state, - marker refresh — were rejected for unbounded storage and - replay-extension respectively); **TCF refusal** — cleared by any regime-accepted grant with newer + source** — cleared only by the ordered explicit authorization defined + above; an + exact semantic replay keeps its original first-seen and cannot clear or + refresh authority age. **A currently presented restrictive value still + starts a new restrictive episode after an ordered clear**: a timestamp-less + source cannot prove that its presentation predates the authenticated clear, + so the privacy-protective result is a new suppression transition whose + clearing floor is the current `authorization_revision`, while the evidence's + original first-seen remains unchanged. A later clear therefore needs another + authenticated monotonic increment (or qualifying newer TCF evidence). + Restrictive evidence never clears positive or negative state. Replay-history + saturation never shortens a newly + observed restrictive choice: grant-class history is evicted before + restrictive history, and a restrictive overflow updates the bounded + per-permission restrictive marker to preserve at least the full + opt-out horizon. **TCF refusal** — cleared by any regime-accepted grant with newer authoritative evidence; **malformed-present / absence** — cleared by any regime-accepted valid grant with newer evidence, including a timestamp-less grant whose first-seen is newer, **and — recovery @@ -511,7 +555,9 @@ and the fail-closed marker: evidence (these causes are not user opt-outs, so stickiness does not apply — without recovery, one truncated request would deny a GPP-only user for the suppression's full TTL). Policy - changes never clear user-signal suppressions. + changes never clear user-signal suppressions, and administrative repair + may delete a suppression only with an auditable record of the consumer's + newer authorization or deletion request. **Anti-replay for timestamps.** A future-dated record is rejected as malformed beyond the skew window; within it, the record's digest is @@ -520,7 +566,7 @@ and the fail-closed marker: an opt-out would keep re-normalizing to "now" and clear it. Equality is **source-specific**: for GPP/USP the digest is the **canonical per-permission semantic result** of §4.5 aggregation alone — two - encodings (or `N` vs explicit N/A) with the same meaning are the same + encodings of the same explicit applicable value are the same evidence and keep the original first-seen, so alternating equivalent values cannot renew authority; for TCF the digest is the semantic result **plus the authoritative `LastUpdated`** — a genuine CMP @@ -565,7 +611,10 @@ and the fail-closed marker: identity row** — deciding "no prior authority" from an eventual not-found loses the race where a just-committed grant is invisible on a stale replica. A suppression/authority read failure **fails closed** - like a revocation read failure; retention must outlive the positive + like a revocation read failure. Every positive identity decision fresh-reads + family revocation, authority/suppression, pending outbox, applicable `w`, + and the global breaker; successful absence/health/authority has no lease. + Only a typed restrictive result may be cached, and only to deny. Retention must outlive the positive authority it masks (providers spec durability/retention capability). **The strong record is the commit point — the two-record protocol is @@ -597,25 +646,84 @@ and the fail-closed marker: see the providers spec §5 order, where eligibility begins at the **authority-state commit**, not the row commit. - **Write failure fails closed for the live request**, and the S2S residual is unbounded - for a never-returning visitor (sign-off 11), with fault tests for - suppress-vs-clear races, repeated-value sequences, and the + **Write failure fails closed beyond the live request.** A deployment that + enables persisted identity use must also provide a durable + negative-intent outbox, independent of both the identity row's eventual + store **and the strong target record's failure domain**. + If a family-revocation or suppression CAS fails, the same request durably + enqueues the idempotent negative intent before it can complete; workers + retry it until the strong record commits. Every live, cached, and S2S + identity decision checks the per-family outbox before positive use; a + pending revocation denies the family and a pending suppression denies its + mapped permission. If neither the target record nor the outbox can commit, + a globally visible identity safety breaker disables all positive identity + mint, use, graph access, and egress until repair; negative repair, + withdrawal, and authenticated deletion paths remain enabled. An adapter + that provides none of these primitives is ineligible for stateful identity. + “Independent” is a qualification result, not a second key prefix in one + store: target and outbox use distinct durability/failure domains. The + breaker may share the outbox domain only when that domain proves this + failure contract: if it cannot accept either the family enqueue or the + breaker CAS, every subsequent strong outbox/breaker read fails rather than + returning a stale successful absence. Every positive decision performs + those reads fresh; no success lease is allowed. Implementations may cache + only a typed restrictive result — revoked, suppressed, pending, or + breaker-tripped — and may use that cache only to deny. A cached restrictive + result cannot construct an `AuthorizedIdentity`, clear or acknowledge state, + or drive a CAS; stale denial may reduce availability after recovery but can + never authorize use. Absence, health, and positive authority are never + cached for a positive decision. Under the qualified fault + model, target failure leaves outbox/breaker available, while outbox-domain + failure leaves either the target committed or all positive readers closed. + An adapter that cannot prove those outcomes is ineligible even if all three + APIs individually advertise CAS. + + The outbox has one total state machine. Each family record carries + `schema_version`, a monotonic `queue_revision`, and a bounded map of pending + negative transitions keyed by the deterministic §6.3 provider-wire + `intent_id`. That wire schema is the sole definition of the materialized JCS + transition payload: cause, source class, evidence time and digest, permission, + state, validity, and the clearing-floor authorization revision all + participate in identity; enqueue time and queue metadata do not. A producer + that cannot construct the complete payload cannot enqueue a transition and + must commit the global breaker. Only family revocation and + creation/strengthening of suppression + enter the map; a failed clear is never queued as negative intent because the + existing denial is already safe. Enqueue CAS-unions entries: family + revocation is absorbing; per-permission conflicts use the authority-state + transition comparator (newer authoritative evidence, restrictive on a tie), + and an older arrival cannot replace newer negative evidence. The cap is 32 + pending intents per family. Revocation consumes one slot and suppression + entries consume one per permission/source; exact duplicates consume none. + Capacity overflow must commit the global breaker before returning and cannot + evict a negative intent. + + A worker applies the exact transition idempotently to the target, then + CAS-removes that `intent_id` only if the queue revision and stored bytes still + match. Target success followed by acknowledgment failure leaves a harmless + pending denial and retry; acknowledgment can never precede target commit. + Empty records are deleted by CAS. Queue retention is the maximum horizon of + all contained transitions plus the recovery/audit window and can never be + shorter than the target negative state. Unknown schema, malformed intent, + read error, revision regression, or retention uncertainty denies the family + rather than skipping the queue. + Fault tests cover + suppress-vs-clear races, enqueue/enqueue merge, target-success/ack-failure, + stale-worker acknowledgment, capacity overflow, outbox replay, + target-domain outage, outbox-domain outage, breaker propagation, and the stale-provenance-read case. - **The cookie expires only after the family record commits.** -- **If the family-record write itself fails, nothing durable exists** — - the cookie stays and the durable client-side signal (GPC, CMP-stored - TCF) retries the whole withdrawal on the next request. Mitigations: - while graph **writes are degraded** (health signal), S2S partner egress - and sync updates fail closed on that instance (providers spec §6.2); - the failure is logged at `error` with a metric feeding the operational - repair path. The residual that remains — a single failed write on an - otherwise healthy graph, for a visitor who **never returns** — is - **unbounded**, not "bounded by return latency": return latency has no - bound for a non-returning visitor, and the per-instance breaker does - not reach other instances. Accepting this residual instead of building - a durable external retry queue is **product sign-off item 11** - (migration spec §8), not a footnote. +- **If the family-record write itself fails, negative intent still becomes + durable before browser state changes.** The cookie stays; the writer + enqueues the family-scoped intent in the required outbox and retries the + family record asynchronously. A failed outbox enqueue trips the global + identity safety breaker; a per-instance health flag is insufficient because a + different instance could otherwise continue partner use for a visitor who + never returns. The error and breaker state are logged and metered, and the + breaker clears only after the outbox and strong record are healthy, the + queue is drained through its recovery watermark, and an audit event records + the controller action. - **Consistency and retention are backend contracts with a single normative home**: the providers spec consistency matrix (§7). It — not this spec — states the requirement, and it requires **globally observable @@ -634,7 +742,7 @@ and the fail-closed marker: untouched, S2S behavior per degraded mode, retry completes; member tombstone N fails after the family record → identity already revoked for every reader, cleanup retries; the same-signal retry path end to end; - **first post-upgrade request is a withdrawal** (v1 row, no family field, + **first post-upgrade request is an explicit storage withdrawal** (v1 row, no family field, derived ID; crash between family record and row write; reader of the untouched v1 row still sees the revocation). @@ -652,51 +760,50 @@ valid sources.** Current runtime resolves conflicts first and can select an expired record before clearing both sources; expiry-first is a **declared change** (migration matrix) that removes that path: -| Input state | Effective record / outcome | Status | -| ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -| Standalone TCF and GPP-embedded TCF disagree, mode `restrictive` | **Whole-record selection comparing the P1 ∧ P4 conjunction only** — today's algorithm, preserved (an earlier draft's lexicographic (P1, P4) tuple would have changed split-purpose outcomes): if exactly one record's conjunction is false, `restrictive` selects it; **equal conjunctions — including split-purpose disagreements — keep the standalone record**, as current code does | Preserved — pinned against current tests | -| Same, mode `permissive` | Same conjunction comparison, selecting the record whose conjunction is true; equal conjunctions keep the standalone record | Preserved — same pinning | -| Same, mode `newest` | Whole-record selection by **`LastUpdated`** subject to the existing freshness threshold; a tie, an incomparable pair, or timestamps inside the threshold fall back to the `restrictive` rule above (itself deterministic) | Preserved — same pinning | -| Expired consent record | Treated as **absent entirely** — grants nothing, refuses nothing, withdraws nothing; the baseline applies. Under a `granted` baseline that means the grant stands: an expired refusal is not current evidence and must not revoke indefinitely | Preserved | -| One valid record + a second malformed record of the same family | The **valid record governs**; the malformed one is ignored with a `warn` log. Fail-closed-on-malformed (below) applies only when no valid record of that family exists | Decided here | -| One valid record + one **expired** record of the same family | The valid record governs — the expired one dropped at pipeline step 2, before conflict resolution ever saw it | **Changed (declared)** — current runtime resolves the conflict first and can select the expired record | -| **Expired** live record + still-valid persisted-KV record | The expired live record is absent entirely (step 2), so it does **not** suppress the fallback: the persisted record substitutes, subject to its own TTL and the full pipeline — "live wins" applies to live records that still exist after expiry filtering | Decided here | -| Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | -| Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**, then flows through the full normalization pipeline (syntax, expiry, conflict) like any live record; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | **Changed (declared)**: current code returns immediately after the KV load, bypassing expiry and conflict normalization | -| Proxy/mirror mode | **Minimal opt-out extraction still runs; full semantic decoding is skipped.** Because opt-outs are globally authoritative (§4), proxy mode must not suppress them: the §4.5-mapped opt-out fields (GPP US sections) and the US Privacy string are decoded — nothing else — alongside syntax validation, so a valid SaleOptOut or USP opt-out revokes and withdraws exactly as outside proxy mode. No grants are ever derived from records in proxy mode; a present record otherwise blocks grants (fail-closed); absent → baseline. GPC needs no decoding | **Changed (declared)**: today proxy mode skips decoding entirely — fail-open under permissive baselines and, worse, opt-out-blind | -| GPP / US Privacy fields | Per the normative field mapping of §4.5 — fields are not interchangeable signals; **explicit N/A is grant-class (not-opted-out), absent grants nothing** — one meaning, everywhere | Decided here (§4.5) | -| Malformed-but-present record, no valid record of that family | **Blocks grants** (fail-closed acquisition — it does not degrade to "absent", which under a `granted` baseline would turn garbage into a grant, the fail-open path in both #838 and the first draft of this spec). Never triggers withdrawal — destruction requires an affirmative, decodable signal (§4.2) | Changed (declared) | +| Input state | Effective record / outcome | Status | +| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Standalone TCF and GPP-embedded TCF disagree, mode `restrictive` | **Whole-record selection comparing the P1 ∧ P4 conjunction only** — today's algorithm, preserved (an earlier draft's lexicographic (P1, P4) tuple would have changed split-purpose outcomes): if exactly one record's conjunction is false, `restrictive` selects it; **equal conjunctions — including split-purpose disagreements — keep the standalone record**, as current code does | Preserved — pinned against current tests | +| Same, mode `permissive` | Same conjunction comparison, selecting the record whose conjunction is true; equal conjunctions keep the standalone record | Preserved — same pinning | +| Same, mode `newest` | Whole-record selection by **`LastUpdated`** subject to the existing freshness threshold; a tie, an incomparable pair, or timestamps inside the threshold fall back to the `restrictive` rule above (itself deterministic) | Preserved — same pinning | +| Expired consent record | Treated as **absent entirely** — grants nothing, refuses nothing, withdraws nothing; the baseline applies. Under a `granted` baseline that means the grant stands: an expired refusal is not current evidence and must not revoke indefinitely | Preserved | +| One valid record + a second malformed record of the same family | For standalone-vs-embedded **TCF**, the valid record governs and the malformed one is ignored with a `warn` log. This row does not govern GPP's independently parsed multi-section aggregation; §4.5's mapped-section blocker does | Decided here | +| One valid record + one **expired** record of the same family | The valid record governs — the expired one dropped at pipeline step 2, before conflict resolution ever saw it | **Changed (declared)** — current runtime resolves the conflict first and can select the expired record | +| **Expired** live record + still-valid persisted-KV record | The expired live record is absent entirely (step 2), so it does **not** suppress the fallback: the persisted record substitutes, subject to its own TTL and the full pipeline — "live wins" applies to live records that still exist after expiry filtering | Decided here | +| Persisted-KV consent record, live record present | **Live wins**, always; the stored record is never consulted | Preserved | +| Persisted-KV consent record, no live record | Substitutes as the effective record **iff within the same TTL as a live record**, then flows through the full normalization pipeline (syntax, expiry, conflict) like any live record; staler → absent. This narrow read is exempt from the graph-read permission gate (§7) — determining `store-on-device` cannot itself require `store-on-device` | **Changed (declared)**: current code returns immediately after the KV load, bypassing expiry and conflict normalization | +| Proxy/mirror mode | **Minimal opt-out extraction still runs; full semantic decoding is skipped.** Because opt-outs are globally authoritative (§4), proxy mode must not suppress them: the §4.5-mapped opt-out fields (GPP US sections) and the US Privacy string are decoded — nothing else — alongside syntax validation, so a valid SaleOptOut or USP opt-out suppresses P4 exactly as outside proxy mode. No grants are ever derived from records in proxy mode; a present record otherwise blocks grants (fail-closed); absent → baseline. GPC needs no decoding | **Changed (declared)**: today proxy mode skips decoding entirely — fail-open under permissive baselines and, worse, opt-out-blind | +| GPP / US Privacy fields | Per the normative field mapping of §4.5 — fields are not interchangeable signals; **explicit N/A and absence both grant nothing**; only an explicit applicable not-opted-out value can grant | Decided here (§4.5) | +| Malformed-but-present record, no valid record of that family | **Blocks grants** (fail-closed acquisition — it does not degrade to "absent", which under a `granted` baseline would turn garbage into a grant, the fail-open path in both #838 and the first draft of this spec). Never triggers withdrawal — destruction requires an affirmative, decodable signal (§4.2) | Changed (declared) | ### 4.5 US signal field mapping — normative GPP and US Privacy fields map to specific permissions with specific -effects; they are never interchangeable, a field's absence or N/A value -behaves per its table row — **explicit _Not Applicable_ is grant-class -(not-opted-out), preserving current USP tests and GPP `NotApplicable` -handling; only a genuinely absent field contributes nothing** (this is -the single normative statement; an earlier "N/A contributes nothing" -rule is dead, and the P4-authorizing consequence is sign-off item 17) — -and only the fields marked destructive trigger -withdrawal. Section IDs and versions are those of the IAB GPP +effects; they are never interchangeable. A field's absence, explicit +_Not Applicable_ value, reserved value, unknown value, or unsupported +version contributes nothing and can never authorize processing. Only an +explicit applicable “did not opt out” value is grant-class, and no +sale/sharing/targeted-advertising field is destructive. Section IDs and +versions are those of the IAB GPP specification pinned by the vendored snapshot; adding a section or field is a change to this table. -| Source · field | Value | `store-on-device` (P1) | `select-personalised-ads` (P4) | Destructive withdrawal? | -| -------------------------------------------- | --------------------------- | -------------------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | -| GPP US section · `SaleOptOut` | opted out | opt-out | opt-out | **Yes** (preserves today) | -| GPP US section · `SaleOptOut` | not opted out | grant | grant | — | -| GPP US section · `SharingOptOut` | opted out | — | opt-out | No | -| GPP US section · `SharingOptOut` | not opted out | — | grant | — | -| GPP US section · `TargetedAdvertisingOptOut` | opted out | — | opt-out | **No** — a targeted-advertising choice must never destroy the stored identity | -| GPP US section · `TargetedAdvertisingOptOut` | not opted out | — | grant | — | -| US Privacy · `opt_out_sale` | `Y` | opt-out | opt-out | **Yes** (preserves today) | -| US Privacy · present, `N` or N/A | — | grant | grant | — (today's tests pin N/A as allowing; USP carries no distinct targeted-advertising field, so it never maps to one) | -| Any field | explicitly _Not Applicable_ | as the field's not-opted-out row above | as the field's not-opted-out row above | — | -| Any field | absent | — | — | — | - -**N/A vs absent (restating the single rule):** explicit _Not -Applicable_ = grant-class; absent = nothing; a non-applicable section's -fields grant nothing (their opt-outs still count, per step 2). +| Source · field | Value | `store-on-device` (P1) | `select-personalised-ads` (P4) | Destructive withdrawal? | +| -------------------------------------------- | --------------------------- | ---------------------- | ------------------------------ | ----------------------- | +| GPP US section · `SaleOptOut` | opted out | — | opt-out | No | +| GPP US section · `SaleOptOut` | not opted out | — | grant | — | +| GPP US section · `SharingOptOut` | opted out | — | opt-out | No | +| GPP US section · `SharingOptOut` | not opted out | — | grant | — | +| GPP US section · `TargetedAdvertisingOptOut` | opted out | — | opt-out | No | +| GPP US section · `TargetedAdvertisingOptOut` | not opted out | — | grant | — | +| US Privacy · `opt_out_sale` | `Y` | — | opt-out | No | +| US Privacy · `opt_out_sale` | `N` | — | grant | — | +| Any field | explicitly _Not Applicable_ | — | — | — | +| Any field | absent / unknown / reserved | — | — | — | + +**N/A vs explicit non-opt-out:** _Not Applicable_ is not affirmative +permission and contributes nothing. An explicit applicable not-opted-out +value can grant only the permission mapped by its row and only under the +regime/applicability rules below. **Unknown section IDs contribute nothing — and bound what embedded-GPC scanning can promise.** A section ID outside the pinned map @@ -713,8 +820,8 @@ per-section by these rules. `GpcSegmentIncluded` and `Gpc` fields; a request with embedded `Gpc = true` and no `Sec-GPC` header was previously unspecified despite the global-GPC rule. Normatively: embedded `Gpc = true` in **any** -section is the same **destructive global opt-out** as the header -(aggregated with it by OR — opt-outs are never jurisdiction-filtered); +section is the same **global P4 use opt-out** as the header (aggregated +with it by OR — opt-outs are never jurisdiction-filtered); `GpcSegmentIncluded = false`, an absent segment, or `Gpc = false` contributes nothing; a malformed optional GPC segment renders that section malformed-present (blocks grants, never withdraws). @@ -729,29 +836,17 @@ section malformed-present (blocks grants, never withdraws). the state sections — `US/CA` ↔ 8, `US/VA` ↔ 9, `US/CO` ↔ 10, `US/UT` ↔ 11, `US/CT` ↔ 12, `US/FL` ↔ 13, `US/MT` ↔ 14, `US/OR` ↔ 15, `US/TX` ↔ 16, `US/DE` ↔ 17, `US/IA` ↔ 18, `US/NE` ↔ 19, `US/NH` ↔ 20, - `US/NJ` ↔ 21, `US/TN` ↔ 22, `US/MN` ↔ 23; **IDs 24–27 (MD/IN/KY/RI) are mapped as - _reserved-pending-official-schema_** — the public official registries - currently expose sections only through 23, so 24–27 have IDs but no - reproducible published binary layout; until the vendored snapshot can - carry an official layout, those four states behave as - no-section states (national section only) and the map does **not** - claim official-registry coverage for them (an earlier revision - claimed both "no section" and later "official through 27" — each - wrong in its own direction). A truncated map silently loses - opt-outs — a Texas (16) sale opt-out must not vanish. **The current decoder is an explicit prerequisite gap**: it - (and `iab_gpp` 0.1.2) supports sections 7–23 only and models `usnat` - v2 while the snapshot pins v1 — implementation must reject versions - the library happens to decode but the snapshot disallows. Sections - 24–27 are **not** a decoder work item and carry **no accepted - version** (the snapshot lists them in a separate _reserved_ table, - not the accepted-version table — an accepted-version entry plus - "inert" prose let two implementations diverge): with no reproducible - official layout they are reserved and inert (national-only for those - states — sign-off 32). Their **presence differs from an unknown - section only in logging**: both contribute nothing, but a reserved - ID is expected-inert while an unknown ID is flagged for snapshot - review. The earlier "Maryland opt-out must not vanish / extend the - decoder" reading is withdrawn. + `US/NJ` ↔ 21, `US/TN` ↔ 22, `US/MN` ↔ 23, `US/MD` ↔ 24, + `US/IN` ↔ 25, `US/KY` ↔ 26, and `US/RI` ↔ 27. All accepted versions + and layouts are pinned by `gpp-registry-snapshot.md`; the current + snapshot accepts version 1 for sections 24–27 from the official IAB + registry commit named there. A truncated map silently loses opt-outs, + so every accepted section is an implementation prerequisite rather + than an inert placeholder. **The current decoder is an explicit + prerequisite gap**: it (and `iab_gpp` 0.1.2) does not implement the + complete pinned set and models `usnat` v2 while the snapshot pins v1. + Implementation must add the missing official layouts and reject + versions the library happens to decode but the snapshot disallows. The implementation PR cross-checks this list against both the current decoder's section set and the official registry, and the accepted version per section is **pinned to the vendored registry snapshot @@ -776,7 +871,21 @@ section malformed-present (blocks grants, never withdraws). on non-`us-privacy` requests grant nothing. Regionless US traffic: national section only. A configured privacy state with no state-specific section uses the national section alone. -3. **State-over-national, per field — for grants only:** where an +3. **Malformed mapped sections participate before value aggregation.** Each + mapped section is syntax/version-validated independently. A + malformed/truncated mapped section or mapped section at an unsupported + version blocks every **grant** for every permission any field in that + section maps to; in v1 that is P4. The blocker is global rather than + jurisdiction-filtered because a decodable opt-out in the same section would + be global under step 2. It never manufactures an opt-out, suppression, or + destructive withdrawal. Valid opt-outs in other sections are still honored. + Consequently valid national grant + malformed state, valid state grant + + malformed national, and two valid grants + one malformed mapped foreign + state all deny the affected grant. Unknown **unmapped** section IDs remain + the explicitly bounded exception described above: they contribute nothing + rather than making all future registry additions fail closed. This + conservative mapped-section policy is product sign-off item 33. +4. **State-over-national, per field — for grants only:** where an applicable state section carries a field, its value governs that field's **grant** derivation; the national section fills only fields the state section lacks. This precedence **never suppresses an @@ -784,10 +893,40 @@ section malformed-present (blocks grants, never withdraws). section's same field says not-opted-out — step 2's global rule wins, or a state string could erase a globally authoritative national opt-out. -4. **Aggregate across what remains applicable:** an opt-out (of either +5. **Aggregate across what remains applicable:** an opt-out (of either subclass) in any applicable field beats a grant from another — restrictive aggregation. +**OpenRTB `gpp_sid` construction is derived, never copied.** After the ordered +algorithm above, core constructs one sorted, duplicate-free integer array from +the pinned section IDs actually present in the decoded GPP header that either +(a) contributed a valid global opt-out or mapped-malformed grant blocker, or +(b) were applicable to the resolved transaction for grant evaluation. A pinned +and decoded GPP TCF section 2 is included when it supplied the effective TCF +record. Unknown unmapped IDs, foreign sections that contributed nothing, IDs +not present in the GPP header are omitted. A known mapped section at an +unsupported version remains identifiable from the decoded GPP header and is +included when its malformed-present blocker contributed; if the GPP header +itself cannot be decoded well enough to enumerate section IDs, no transport +pair is constructable. The serializer emits +`regs.ext.gpp` and `regs.ext.gpp_sid` atomically or emits neither; it never +sends raw GPP with a guessed, empty-by-error, or client-copied SID array. + +The request companion `__gpp_sid`, when present, uses the exact ASCII grammar +`section-id *( "," section-id )`, where `section-id` is a positive base-10 +integer without sign, whitespace, or leading zero. Input order is immaterial +and is canonicalized to a sorted set; a duplicate is malformed. The companion +is used only for consistency checking and is not the source of the OpenRTB +field. Exact set equality with the derived applicable set is accepted. +Absence is allowed because core can derive the set. A mismatch, duplicate, +non-decimal value, or reference to a mapped ID absent from the GPP header is a +malformed auxiliary signal: it blocks grants for the union of recognized +mapped permissions implicated by either set, preserves every decodable opt-out, +and never manufactures withdrawal. The derived set remains the only egress +value. Named fixtures cover absent companion, reordered/duplicate input, +foreign-state omission, global opt-out inclusion, section-2 inclusion, +mapped-version failure, and unknown-unmapped omission. + `SharingOptOut` and `TargetedAdvertisingOptOut` are new enforcement inputs — current code consults only the sale field — and are declared as such in the migration matrix. @@ -804,19 +943,28 @@ case-insensitively. ### 5.2 Lookup failure -Provider selected, lookup resolves nothing for a request → the configured -`[geo] default_country` rules apply (per #779). An adapter whose geo +Provider selected, lookup resolves nothing for a request → the compiled-in +protective failure profile applies: both permissions `requires_signal` and +`regime = "gdpr"`. `default_country` is not a provider-outage fallback; it +is used only for the explicitly acknowledged static-jurisdiction mode of +§5.3. An adapter whose geo implementation can never resolve anything must not accept the selection at all — that is the capability check of providers spec §6, and it prevents a "selected but always empty" provider from silently converting every request to §5.3 semantics without §5.3's guard. -Declared residual: when the default country's baseline is permissive, a -per-request lookup failure is a per-request grant to traffic of unknown -origin — this path is not fail-closed, and the spec does not pretend it is. -The lookup-failure rate is exported as a metric and logged, so an elevated -rate (a degraded geo backend silently converting traffic to the default) is -observable rather than invisible. +The lookup-failure rate is exported as a metric and logged. A deployment may +use a bounded operational circuit breaker to stop auction dispatch during a +prolonged failure, but it may never substitute a permissive country rule for +unknown origin. Recovery to ordinary jurisdiction rules occurs only after a +successful live lookup. + +Named divergence fixtures pin both sides of migration matrix row 5: lookup +failure with absent, malformed, expired, or regime-inapplicable evidence denies +both permissions and contextualizes dispatch; lookup failure with an explicit +valid regime-accepted grant may grant only its mapped permission under +`requires_signal`. The latter is a declared behavior change, never described +as preservation of today's deny-all path. ### 5.3 No geo provider selected @@ -844,8 +992,9 @@ log always prints the effective baseline and whether geo is live. ### 5.4 Defaults, two distinct fallbacks -`[geo] default_country` is required; startup fails without it (per #779). -It covers requests that resolve **no country at all**. Countries that +`[geo] default_country` is required only when no provider is configured and +`assume_single_jurisdiction = true`; startup fails without it in that mode. +It does not cover a selected provider's lookup failure (§5.2). Countries that resolve but match no rule fall to the policy's `rules.default` entry (§3.2). The two fallbacks are deliberately separate: "we could not place this request" and "we placed it somewhere we have no rule for" are @@ -857,98 +1006,395 @@ migration story unresolvable (migration spec §2, rows 5 and 7). A **policy revision** has one identity used everywhere: the pair **(content digest, activation ordinal)**. The digest is SHA-256 with -domain tag `tspol1|` over the canonical JSON of the parsed policy (keys -sorted lexicographically by UTF-8 code unit, numbers shortest -round-trip, defaults materialized, no insignificant whitespace; -cross-language vectors required) — identity, so an A→B→A rollback -yields A's digest again. The ordinal comes from the **policy-activation register** — +domain tag `tspol1|` over the canonical JSON of the parsed policy — +**canonicalization is RFC 8785 (JCS), referenced normatively**, not a +home-grown rule list (key ordering, number formatting, string +escaping, and Unicode handling are exactly JCS's), applied after +defaults are materialized, with a policy-schema profile making the +remaining cases unreachable: validation rejects non-finite numbers, +numbers outside the exactly representable integer range +(absolute value above 2^53 − 1) unless the field is string-typed, and +`null` values (materialized defaults mean `null` never appears); +negative zero serializes as JCS mandates. The machine-readable, +cross-language conformance fixtures are pinned in +`docs/superpowers/specs/policy-canonicalization-vectors.json`; every +runtime and the push tool must reproduce both the canonical UTF-8 bytes +and digest, and must reject every rejection vector before activation. A +digest difference is a startup failure, so canonicalization cannot be +approximately specified. The digest is pure content identity, so an +A→B→A rollback yields A's digest again. + +The ordinal comes from the **policy/config/model activation register** — deployment-metadata name `02` (providers spec §6.3), a linearizable -register holding the current `{source_version, policy_digest, ordinal}` -**plus a bounded history of the last 16 activations**. Current-value-only -was shown to break activation identity: an interleaved registration -evicted the pair, and a same-push latecomer then minted a second -ordinal for one activation — two `(digest, ordinal)` identities for one -push. Transition rules, evaluated on a strong read + CAS: - -- `(source_version, policy_digest)` **found in the history** → adopt - that entry's ordinal, no increment — idempotent for every instance - of the same activation however late it arrives within the window, so - one activation has exactly one `(digest, ordinal)` fleet-wide. -- `source_version` found in the history with a **different digest** → - **fail closed at startup**: one pushed configuration parsing to two - canonical-policy digests is mixed-binary parse divergence — a hard - incompatibility, never a novel pair, never a new ordinal. -- `source_version` **newer** than every history entry → a new - activation: CAS-append `(source_version, digest, max_ordinal + 1)`, - evicting the oldest history entry. -- `source_version` older than the newest and absent from the history → - **stale, rejected** (an instance restarting on old config can - neither mint an ordinal nor regress the register; a laggard older - than the 16-entry window is also rejected — it must fetch current - config, not activate). - -`source_version` must be an **ordered identifier assigned exactly once -upstream**: the config store's push version where the backend has one, -else the **monotonic push sequence the `ts config push` envelope stamps -into the blob**. The earlier digest-only fallback is **deleted** — a -digest cannot distinguish a deliberate rollback from a stale-instance -restart, and it re-minted ordinals for a single activation. A -deployment with neither ordered identifier is not eligible for -multi-instance policy activation (an adapter capability cell, providers -spec §7). A→B→A remains a **third activation** — new `source_version`, -A's digest, a new ordinal — digest for identity, ordinal for order, -exactly as revision identity requires. Order is adapter-independent -(per-instance counters ordered nothing across a fleet). Authority wire records, the S2S recompute, and the hook cache -tuple all use this same pair; the hook's earlier "config-store push -version" and any digest-only usage are superseded. The other cache-tuple -inputs are likewise domain-separated hashes of effective configuration: -integration-registry revision = `tsreg1|` over the canonical-JSON -`(id, version)` list; config revision = `tscfg1|` over the effective -config blob — so adapters derive identical revisions from identical -configuration. - -A policy edit propagates through the config store, so a fleet briefly -mixes revisions. The contract: instances stamp every resolution and -every provenance write with the (digest, ordinal) they used (already -required by §7); the mixing window is bounded by config propagation and -observable via the activation-ordinal metric; and mixed-revision -irreversibility is bounded and **accepted, not denied** (sign-off 19): -destructive withdrawal triggers are user signals, never policy (§4.2 -trigger 3) — the one revision-sensitive destructive case (trigger 2 -under a now-`denied` baseline) requires an affirmative user refusal at -the evaluating instance, which is safe under either revision. S2S -recomputation always evaluates against the instance's current revision -and records it. One divergence is explicitly accepted rather than -fenced: during convergence, a live refusal under a `granted`-revision -instance suppresses while the same refusal under a tightened-revision -instance destroys (trigger 2) — the destructive outcome is the target -revision's intended behavior arriving early on part of the fleet, -coordinated activation fencing is not worth its machinery, and the -acceptance is sign-off item 19. Rolling a policy revision back restores -acquisition rules but **cannot resurrect tombstoned identities**; the -migration guide says so where operators will read it. +register holding `active`, an optional settings `candidate`, an optional +`model_candidate`, an `activation_journal_head`, and a bounded history of the +last 16 activations. `candidate` and `model_candidate` are mutually exclusive; +installing either while the other exists is rejected. +`active` is `{logical_root, immutable_blob_id, +source_version, data_hash, config_revision, policy_digest, ordinal, +model_epoch, minimum_binary_generation, row_schema_floor, +activation_generation}`. `activation_generation` is a logical active-tuple +`u64`, not the backing store's CAS/version token: installing a candidate or +readiness entry does not change it; each successful settings or model promotion +increments it by exactly one, and overflow is a hard deployment error. This is +the stable generation used by serve admission while candidate readiness is +changing. Each deployment additionally qualifies one +`serve_admission_lease_bound_ms`: the maximum interval from the +invocation of a successful linearizable admission read until every admission +derived from that read is locally invalid, including timer-rate error, delayed +response, suspend/resume, and every other adapter timing uncertainty. It is a +portable positive integer, is immutable while any member is traffic eligible, +and is not configurable through the settings blob; changing it requires +traffic to be stopped and the deployment capability to be requalified. Every +candidate snapshots the exact bound and every member readiness entry attests +it. Candidate installation rejects zero, a value different from the currently +qualified deployment bound, or an attempted bound change while any member is +traffic eligible. The activation register and immutable journal-object service +expose one authenticated, nondecreasing Unix-millisecond time domain; an +adapter with distinct or merely offset-local clocks is unqualified. The time +domain and its backing service are immutable while any member is traffic +eligible or a candidate exists; migration requires stopped traffic, no +candidate, and fresh qualification. The register's trusted clock qualification guarantees that its +promotion-not-before condition cannot become true until at least that much +real elapsed time after the draining CAS; forward steps, rate error, and +uncertainty fail the check rather than shorten the interval. `model_epoch` is +exactly `pre_epic_v1` or `permissions_v2`. Every binary exposes one immutable +monotonic `binary_generation` build constant. The initial N+1/N+2 generations +are 1/2, and values are never reused. A settings +`candidate` carries a `candidate_incarnation` of 32 lowercase hex characters +from 16 CSPRNG bytes, never reused in the deployment scope, plus the bound +active activation generation and complete active tuple, the complete proposed +content-binding tuple, copies the active model +fields byte-for-byte, and adds `proposed_ordinal`, the snapshotted +`serve_admission_lease_bound_ms`, an immutable authoritative +`fleet_snapshot`, phase (`preparing` or `draining`), readiness entries, and +quiescence entries. It also carries mutable `drain_attempt: u64`, initialized +to 0, and mutable `promotion_not_before_unix_ms`, null until each drain begins. +A `model_candidate` has its own never-reused candidate incarnation and +carries the exact bound active activation generation and complete active tuple, +all three proposed model fields, the immutable authoritative `fleet_snapshot`, +the same snapshotted admission-lease bound, phase, drain attempt, +promotion-not-before value, and readiness/quiescence entries. A snapshot +is `{membership_epoch, members[]}` where members are sorted unique stable +deployment-instance IDs. It is produced only by the authenticated deployment +membership controller from the set of instances eligible to receive traffic; +an application process, config publisher, or readiness writer cannot nominate +or remove members. Each readiness entry is authenticated as its member ID and +contains the membership epoch plus the complete immutable candidate identity +(every candidate field except mutable phase, drain attempt, +promotion-not-before, and the +readiness/quiescence maps). That identity includes the candidate incarnation; +a delayed entry from an aborted or restaged candidate can never validate even +when every content and fleet field is otherwise identical. The same member's +quiescence entry additionally binds the `draining` phase, exact nonzero +`drain_attempt`, and exact `promotion_not_before_unix_ms`, and is valid only +after that member has atomically closed new request admission and completed or +cancelled every request admitted under the bound activation generation. +Readiness cannot stand in for quiescence. The +immutable blob ID is the adapter mapping of `(logical_root, source_version)`; +`data_hash` is the verified envelope data hash, and `config_revision` is the +effective-config digest defined below. A readiness entry is therefore not bound +merely to source version and policy digest: a byte change, logical-root change, +config-only change, membership change, or ordinal change makes an old +acknowledgment inapplicable, while the `preparing` → `draining` phase CAS does +not. The register's 16-entry history is an operational +rollback window, not the audit or garbage-collection clock. Every promotion +also appends an immutable, hash-linked activation-journal record containing the +previous journal head, expected logical active `activation_generation`, complete +displaced and new tuples, membership epoch, readiness and quiescence sets, +the admission-lease bound and promotion-not-before time, retention horizon, +and controller identity. The journal object's qualified immutable-store +metadata supplies store-issued `created_at`; its canonical schema, object ID, +known-answer vector, lifecycle, listing, genesis, and pruning rules are +normative in CLI §5.1. The controller writes and read-verifies that object +before promotion; the one register CAS both promotes the candidate and changes +`activation_journal_head` to its object ID. A losing CAS leaves an unreferenced +journal object, never an active tuple without a journal entry. Journal records +and every blob they name are retained for +at least 30 days and for at least the maximum processed-artifact, cookie-scope +migration, rollback, and audit horizon, whichever is longer. Rapid pushes can +evict a tuple from the 16-entry register but never shorten that time-based +retention. “Atomic” here means the register CAS binds the already verified +immutable journal object; it does not assume a cross-store transaction. The CAS +verifies the journal object's authenticated `created_at` in that common time +domain, rejects creation before the candidate's exact promotion-not-before +value, and rejects an object more than 60 seconds old. The earliest deletion +time adds that 60-second promotion allowance to the +required retention horizon, so even the latest permitted promotion receives +the full horizon. Local process time never starts or shortens retention. An +adapter without store-issued journal time, that binding, a qualified journal +listing/read path, and lifecycle enforcement is ineligible for multi-instance +activation. History and journal records never make an old configuration +eligible. `proposed_ordinal` equals the active ordinal for an unchanged policy +digest and active ordinal + 1 for a changed digest, with the overflow rule +below. Before the first activation, the compiled-in +protective configuration is the synthetic active tuple with logical root and +blob ID `builtin`, source version and ordinal 0, and reproducible +data/config/policy digests derived by the same grammars from materialized +compiled defaults; its model fields are `pre_epic_v1`, minimum binary +generation 1, row schema floor 1, and activation generation 0, and it permits +no destructive interpretation of historical evidence. Candidate abort is an +authenticated deployment-controller operation, is audit-recorded with the +complete candidate tuple and reason, and never rewrites `active` or reuses the +candidate's source version, blob identity, or candidate incarnation. + +Every nonnegative integer carried into the activation journal — source version, +policy ordinal, model/binary/schema/activation generation, membership epoch, +drain attempt, admission-lease bound, promotion-not-before, readiness fields, +and retention — is additionally constrained +to the portable JCS range `0..=2^53-1`. The register may use a wider integer +primitive internally, but candidate installation rejects a value outside that +range; no activation can create an unjournalable active tuple. + +`source_version` is an ordered `u64` within that portable range, scoped once per +deployment/application across all config-blob names. A single scope matches +the fixed deployment-metadata register key and avoids two independently +ordered streams aliasing one register. +Where the platform does not expose a trustworthy ordered push version, the +push tool allocates envelope `push_sequence` from a separate linearizable +**config-sequence register** in deployment metadata: strong-read + CAS +`next := current + 1`, then publish the envelope carrying `next`. Reaching the +portable maximum is a hard deployment error. Allocation +gaps after a failed publish are allowed; reuse is forbidden. A restore or +rollback republishes old content under a new sequence. The config store +itself is not assumed to provide conditional publication — its current +get/put/delete interface does not — and an adapter without the deployment- +metadata allocator is ineligible for multi-instance config/policy activation. +The CLI envelope design must add this field and allocator interaction. + +Activation is **prepare then commit**, never “first instance wins”: + +1. The push tool writes and read-verifies a new immutable envelope object, + then CAS-installs its complete content-binding tuple as the sole candidate. + Merely overwriting a mutable `app_config` key cannot stage or activate + anything. A second candidate CAS is rejected until the current candidate + activates or is explicitly aborted; an unreferenced object written by the + loser is inert and later garbage-collected. +2. Every member of the candidate's immutable authoritative deployment- + membership snapshot loads **that exact immutable object**, verifies the + envelope data and sequence-binding hashes, materializes defaults, validates + config and policy semantics, derives both config and policy digests, and + CAS-records readiness for the complete candidate tuple. A member that + cannot load the object or derives any different field fails closed and + never acknowledges. + Membership is frozen for that candidate. A new traffic-eligible member, a + replacement instance with a new stable ID, or removal of a dead member + changes `membership_epoch`, automatically aborts the candidate, and requires + prepare to restart against a new snapshot. A controller cannot shrink a + candidate in place to manufacture unanimity. Autoscaled instances may start + during prepare, but receive no traffic until they load the current active + object and enter the next authoritative membership epoch. +3. Only after every member in that snapshot is ready does the controller CAS + the same candidate from `preparing` to `draining`, increments + `drain_attempt` by one, atomically clears every quiescence entry, and, from + the register's trusted store clock at that CAS's linearization point, sets + `promotion_not_before_unix_ms` to the checked sum + `store_now + serve_admission_lease_bound_ms`; + overflow aborts and restages the candidate. At lease expiry, each member + atomically stops admitting **all** new requests unless a renewal strong-read + returns a non-draining result; a read that fails, is delayed past expiry, or + observes `draining` cannot renew. The member's authenticated activation + watcher also strong-reads the drain state, invalidates the local lease, + closes admission, drains or cancels every request admitted under the bound + activation generation, then writes its authenticated quiescence entry. A + member may acknowledge only when no such + request or its background work can still reach origin, bidder, partner, + vendor, cache publication, identity mutation, or another configurable + effect. A member that has observed `draining`, or whose prior admission + lease has expired, gives new requests the deployment-unavailable response + with no configurable egress. An unexpired lease may continue admitting the + previous generation only until its hard bound; no admission read that + linearizes after the drain CAS can create or renew such a lease. A member + may acknowledge quiescence before the promotion-not-before time only when + its own admission gate is already closed, but time alone never substitutes + for that member's acknowledgment. A delayed watcher or acknowledgment + extends the unavailable interval and cannot weaken the fence. A controller + failure leaves admission closed until an authenticated + `cancel-drain` CAS restores `preparing` and atomically clears every + quiescence entry and the promotion-not-before value, a + candidate abort removes the candidate, or a valid promotion completes; a + local timeout cannot reopen traffic. Cancellation does not itself authorize + traffic: each member must strong-read the restored phase before acquiring a + new lease. Resumed traffic makes every earlier quiescence acknowledgment + inapplicable. +4. Only after every member in that snapshot is both ready and quiescent, and + the register's trusted store clock has reached the exact + promotion-not-before value, does the deployment controller construct and + read-verify the immutable journal object and + CAS-promote the candidate to `active` while binding that object as the new + journal head. The register itself rejects promotion while its trusted store + clock is earlier than the candidate's exact + `promotion_not_before_unix_ms`; a controller sleep or wall-clock comparison + cannot satisfy this condition. Expiry of the bound only closes further old + admissions — authenticated quiescence is still required to prove that the + last admitted request and every asynchronous effect ended. If its policy digest + differs from current `active`, promotion verifies + `proposed_ordinal == max_ordinal + 1` and uses that new policy ordinal; + mismatch or portable-range overflow is a hard deployment error, never wraparound. If + the digest is unchanged, promotion verifies `proposed_ordinal` equals the + current ordinal. For a + config-only push, promotion advances `source_version` but retains the + existing ordinal — unrelated configuration must not manufacture a new + policy identity. A settings promotion copies the current model epoch, + minimum binary generation, and row schema floor byte-for-byte; config cannot + advance or roll them back, and sets activation generation to the bound + generation + 1. **Every** promotion appends the complete previous active + tuple to operational history, including config-only pushes. Audit and + immutable-blob retention come from the independently time-bounded journal, + so evicting operational history never loses the displaced snapshot. +5. Instances continue serving the entire previous active configuration while + preparation is incomplete; the newest physical blob has no “latest wins” + semantics, then stop during the explicit drain above. After promotion, + members load and verify the new active tuple before reopening admission. + **Every request**, including requests that do not use identity, must present + a live admission validation when it atomically registers at the local gate. + The validation covers both candidate phase and the complete `active` tuple, + including `activation_generation` and model fields. An admitted request may + outlive that validation's admission window; its gate/refcount registration, + rather than a mid-request lease renewal, keeps it inside the drain and + quiescence proof. With no live validation, admission strong-reads the + register and may lease only a + successful non-draining result for at most the deployment's qualified + `serve_admission_lease_bound_ms`. Lease age starts no later than invocation + of that linearizable read, never response receipt, so latency shortens the + usable interval and a response arriving at or after expiry cannot admit. + The lease binds the deployment, stable member ID, exact active tuple, and + bound; it is process-local, is invalid after restart or suspend/resume, and + cannot survive read, timer, or renewal uncertainty. + `draining` rejects admission as above; otherwise the request uses settings + loaded from that exact active blob whose data/config/policy hashes match. + Local admission closing and request/background-effect registration use one + atomic gate/refcount that compares the validation's exact active tuple with + the gate's current tuple before incrementing, so quiescence cannot race a + last admission and a delayed old-generation read cannot enter after reopen. + The fence covers routing, + auction serialization, integration selection, DataDome, response mutation, + cache lookup/replay, identity, and destructive paths. The v1 fence's strong + read may be amortized only by this activation-scoped lease; it grants no + lease for authority, revocation, outbox, `w`, breaker, or other privacy + state. A mismatch stops processing before origin, + bidder, partner, or vendor egress, refreshes the complete settings object, + and admits later requests only after every binding verifies. Failure to read + or load active returns the deployment-unavailable response and performs no + configurable egress. An + instance starting or restarting likewise loads, verifies, and obtains a + fresh admission validation on active before the traffic controller marks it + serving. After promotion a member reopens only after loading the promoted + tuple and strong-reading a lease for its new generation. No identity-only exception, + mutable-root fallback, stale-on-error path, or partial per-subsystem refresh + is permitted. + +Model/writer activation is a second transition on the **same register**, so it +cannot race or drift from settings activation: + +1. With new-shape settings active and no settings candidate, the controller + CAS-installs a `preparing` `model_candidate`. Its immutable identity is the + never-reused candidate incarnation, bound active activation generation, + bound complete active tuple, proposed + model epoch `permissions_v2`, proposed minimum binary generation 2, + proposed row schema floor 2, snapshotted + `serve_admission_lease_bound_ms`, and fleet snapshot; its mutable state is + phase, drain attempt, promotion-not-before value, readiness, and + quiescence. Re-entry at the same exact identity is idempotent; any different + concurrent candidate is rejected. +2. Every traffic-eligible member in the authoritative snapshot must load the + bound active settings, prove `binary_generation >= 2`, validate the v2 + provider/permission writer, and authenticate readiness for the complete + model candidate. N+2 runs `pre_epic_v1` behavior until this promotion; it may + not write v2 rows, positive authority, or durable use suppression merely + because its binary understands them. A membership or active-settings + activation-generation change aborts and restages the model candidate. +3. After unanimous readiness, the controller CASes the model candidate to + `draining`, increments the model candidate's `drain_attempt`, and atomically + clears old quiescence entries while setting the same store-clock + `promotion_not_before_unix_ms`; every member applies the settings procedure's all-request + admission stop, completes or cancels every `pre_epic_v1` request, and + records bound quiescence. Local timeout cannot resume admission. +4. After unanimous readiness **and quiescence**, the controller creates the same immutable + journal object required for a settings promotion and one register CAS + verifies the promotion-not-before time has passed, the bound active + activation generation, and complete tuple, changes + `model_epoch`, + `minimum_binary_generation`, and `row_schema_floor` together, clears the + model candidate, increments `activation_generation`, appends operational + history, and binds the journal head. Policy digest/ordinal and settings + source version do not change. +5. Every serve-admission read checks the model fields as well as the settings + tuple. After promotion, a binary with generation below the minimum stops + before all request processing and cannot start serving; a qualifying N+2 + binary enables the new live gate/writer only after observing that exact + active activation generation. There is therefore no interval in which N+1 + serves while N+2 writes v2. The old deployment-metadata `m00` schema-floor key is only a + monotonic startup compatibility mirror written after this CAS; it has no + authority to enable writes or serving. After a successful model CAS, the + authenticated deployment controller owns the idempotent mirror completion + step: strong-read `active` and `m00`; if `m00` is missing or lower, CAS it + to exactly `active.row_schema_floor`, while equality is an idempotent + no-op; then strong-read and verify exact equality before declaring the + transition complete. A crash retries the same operation; it never lowers + `m00` and never changes or authorizes the active register. An unreadable + mirror or failed CAS/read-verification keeps startup closed and is retried. + A mirror higher than active is rejected before any write and cannot be + auto-lowered: startup fails for register/journal inconsistency + investigation. The authoritative schema floor advances only in the single + fenced model transition. + +Head rules are therefore unambiguous: equal active `source_version` plus equal +data/config/policy digests adopts the active ordinal; equal version plus any +different digest or blob identity is a hard parse-divergence failure; lower +version is stale and rejected; +higher version is staged and is not active until fleet commit. A duplicate +publication never creates another ordinal. A higher-version config-only push +with the active policy digest retains the ordinal after readiness but changes +the active blob/data/config tuple. A→B→A remains a +third activation with a new `source_version`, A's digest, and a new ordinal +because A differs from the then-active B. + +Authority wire records and S2S recomputation use the active policy `(digest, +ordinal)` pair. The hook's one complete cache revision tuple additionally binds +`model_epoch`, logical `activation_generation`, and its hook-invariant revision +exactly as the hook spec §3 defines, so model-only activation cannot replay a +pre-epic artifact. Its registry and config inputs are domain-separated SHA-256 +hashes of JCS UTF-8 bytes: integration-registry revision = `tsreg1|` plus the +registration-order array of `{id, behavior_revision}`; config revision = +`tscfg1|` plus the complete typed effective config after defaults. Registry +array order is preserved because mutator order is behavior. The config form +contains secret **references**, never resolved secret bytes, and excludes +runtime observations. Both emit lowercase 64-hex digests and must reproduce +`docs/superpowers/specs/revision-canonicalization-vectors.json`. Tests cover a +pre/post-model-CAS cache miss as well as concurrent push allocation, publish +gaps, equal-version idempotence, equal-version +digest mismatch, same-digest higher-version ordinal retention, stale restart, +candidate abort, partial readiness, promotion, old-instance behavior after +promotion, and A→B→A. + +Mixed-revision irreversible behavior is **prohibited**, not accepted. Config +distribution may be mixed during preparation, but only the complete `active` +tuple authorizes identity decisions, and destructive effects are fenced to +that tuple (wire provenance records its policy pair). Rolling back acquisition +policy still cannot resurrect an identity +withdrawn by a valid user signal; rollback itself follows the same staged +activation protocol. ## 6. Failure-mode matrix — normative -| Condition | Resolution behavior | -| ---------------------------------------------------- | --------------------------------------------------------------------------------------------- | -| Geo lookup fails at request time (provider selected) | `default_country` baseline | -| No geo provider configured | `default_country` baseline, guarded by §5.3 | -| Country resolved, no matching rule | Policy `rules.default` | -| Region resolved, no region rule | Country rule | -| No `[permissions]` section | Compiled-in fallback: everything `requires_signal`, `regime = "gdpr"` | -| S2S sync request (no user signals) | Authorized by stored provenance re-validated against current policy (§7) | -| Malformed policy | Rejected at config push / startup (§3.3) — never per request | -| No `default_country` | Startup failure | -| Undecodable TCF/GPP record (present but malformed) | Blocks grants (fail-closed acquisition, §4.4); never withdraws; opt-out signals still honored | -| Signals contradict (opt-out + consent) | Opt-out wins (§4) | - -The intended posture is fail-closed, with its two exceptions stated rather -than glossed: geo lookup failure resolves to the configured default (§5.2's -declared, metered residual — permissive defaults make this path fail-open), -and the §5.3 static-jurisdiction configuration exists only behind an -explicit operator acknowledgment. Every other ambiguous state resolves to -the configured baseline or more restrictive. +| Condition | Resolution behavior | +| ----------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| Geo lookup fails at request time (provider selected) | Protective failure profile: both permissions `requires_signal`, `regime = "gdpr"` | +| No geo provider configured | `default_country` baseline, guarded by §5.3 | +| Country resolved, no matching rule | Policy `rules.default` | +| Region resolved, no region rule | Country rule | +| No `[permissions]` section | Compiled-in fallback: everything `requires_signal`, `regime = "gdpr"` | +| S2S sync request (no user signals) | Authorized by stored provenance re-validated against current policy (§7) | +| Malformed policy | Rejected at config push / startup (§3.3) — never per request | +| No `default_country` in acknowledged no-provider static mode with a jurisdiction consumer | Startup failure; otherwise the field is not required (§5.3–§5.4) | +| Undecodable TCF/GPP record (present but malformed) | Blocks grants (fail-closed acquisition, §4.4); never withdraws; opt-out signals still honored | +| Signals contradict (opt-out + consent) | Opt-out wins (§4) | + +The intended posture is fail-closed. Geo lookup failure uses the protective +profile, and the §5.3 static-jurisdiction configuration exists only behind +an explicit operator acknowledgment. Every other ambiguous state resolves +to the configured baseline or more restrictive. ## 7. Enforcement points @@ -992,6 +1438,100 @@ Consumers of the resolved set in this epic: | Authority-state / suppression-decision read (§4.3) | **exempt**, narrowly scoped | Returns only family ID, per-permission authority summary, and suppression entries — no identity values, no partner data; a test proves nothing else escapes | | `AuthorityRefresh` provenance write (§4.3) | **exempt**, strictly scoped | Commits current-live-resolution provenance only; enables suppression recovery without reopening `GraphOps` | + **Raw regulatory transport is a separate positive allowlist.** The + current allowlist contains only OpenRTB-compatible auction dispatch, and + only the protocol field actually defined for that source: TCF in + `user.ext.consent`, GPP in the atomic `regs.ext.gpp` + derived + `regs.ext.gpp_sid` pair defined in §4.5, and + US Privacy in `regs.ext.us_privacy`. A destination registration declares + the fields it supports; TS sends the minimum matching source set, never a + generic bundle of every raw signal. APS/direct auction APIs are not assumed + equivalent to OpenRTB and receive no raw string until their checked-in + protocol registration names the required field. Publisher origin, + proxy/click/Testlight, identify and sync endpoints, ordinary integrations, + identity rows, and observability sinks are explicitly not consumers. + Unknown destinations default deny. Tests enumerate every allowed + destination × field and assert every other egress view is structurally + unable to access the raw strings. + + **Contextual OpenRTB is a positive projection, not “the ordinary request + minus EC.”** Whenever auction dispatch is allowed while + `select-personalised-ads` is unset, core serializes a + `ContextualAuctionView` constructed independently from the ordinary auction + object. The **sole normative v1 output schema** is the checked-in + `docs/superpowers/specs/contextual-openrtb-v1-allowlist.json`; descriptive + prose cannot add a field. Its path language is dot-separated exact JSON + member names, with `[]` denoting every element of the immediately preceding + array. Object and array containers are implicit and exist only when at least + one admitted descendant requires them; admitting a container never admits + another child. `site` and `app` are mutually exclusive, and `imp` contains + at least one element. V1 supports only the allowlisted banner and video + impression shapes. A native, audio, or DOOH impression, or any impression + that cannot be represented entirely by one allowlisted shape, makes the + contextual serializer fail with no dispatch. + + Each manifest rule gives one exact leaf path, JSON scalar type, + cardinality, derivation class, and, where present, the complete value enum. + Cardinalities have these exact meanings: `required_single` is present once + in its object; `optional_single` is absent or present once; + `required_array` is a non-empty array whose every scalar element matches the + rule; `optional_array` is absent or such a non-empty array; + `required_array_member` is present once in every instance of the nearest + enclosing object-array element; and `optional_array_member` is absent or + present once in each such element. Empty optional arrays are omitted, never + encoded. JSON numbers must be finite; integers use the OpenRTB field's + declared range. Strings are valid UTF-8, are normalized by the field's + OpenRTB grammar, and are rejected rather than truncated when they exceed its + bound. + + The manifest's `cross_field_rules` are equally normative. Paths sharing an + `[]` segment are evaluated within the same array-element binding, never + across different impressions/nodes. `all_or_none` requires every named leaf + or none; `required_nonempty_object_arrays` requires the named object array + to exist with at least one element whenever its parent exists; and + `at_least_one_complete_group` requires at least one listed group to be fully + present whenever its parent exists. Thus GPP and its nonempty derived SID + array are atomic, an emitted supply chain has at least one complete node, + banner `w`/`h` are paired, and every emitted `banner.format` element has + both dimensions. Unknown rule kinds or container paths fail startup. + + The derivation vocabulary is closed by that manifest. `fresh_transaction` + is a new CSPRNG value for this dispatch and is never copied from or derived + from EC, IP, consent, DataDome, graph, or stored provenance; `inventory` is + a value from validated publisher inventory configuration, never a request + free-form or extension value; `request_coarse` is only the typed coarse + request fact named by the exact path; `privacy` is produced by the + permission resolver or the raw-regulatory allowlist immediately above; and + `constant` is the literal named by the rule. In v1 `device.lmt` is therefore + exactly integer `1`. `device.os` has no version, `device.language` is one + normalized primary language subtag, and `device.geo.country` is an uppercase + ISO 3166-1 alpha-2 code; a finer or malformed source is omitted rather than + rounded ad hoc. + + The serializer is generated from, or startup-validated byte-for-byte + against, this manifest. A conformance walker expands every final encoded + JSON leaf to the same normalized path and requires it to match exactly one + rule with its type, cardinality, derivation tag, and enum, then evaluates + every container and cross-field rule over the same final tree. An unknown, + duplicate, ill-typed, untraceable, or unlisted leaf is a serialization error + and produces **no bidder request**. This includes every unlisted `ext` + member: there is no arbitrary JSON pass-through. V1 has no destination + extension registrations; adding one requires a separate checked-in, + machine-readable manifest using this same closed grammar and named for that + destination. A destination that cannot consume this exact projection also + receives no request — TS never falls back to the ordinary serializer. + + Consequently the v1 output has no EC or other user identifier, IP/IPv6, + user agent, IFA, client forwarding header, hardware/network/screen + fingerprint, precise geo, region/city/ZIP/metro, page/referrer/store URL, + query/fragment, demographics, keywords, segments, custom data, or + non-allowlisted extension. `user` can exist only as the implicit parent of + the exact allowlisted `user.ext.consent` leaf when the destination's raw + regulatory registration requires TCF. Conformance tests inspect the final + encoded HTTP headers and body, poison every forbidden source (including + nested extension objects), and prove that each poison is absent or the + dispatch is suppressed. + With **no EC provider configured**, identity use fails closed: a cookie value present on the request never egresses anywhere — never vacuously allowed (#838's `ec_allowed` was `is_none_or`, vacuously true with no @@ -1007,7 +1547,7 @@ Consumers of the resolved set in this epic: requests — grant basis (which signal class granted, per permission), the evidence's **authoritative timestamp and `valid_until`** (per evidence class), - resolved jurisdiction **with `jurisdiction_observed_at`**, and policy + tagged jurisdiction provenance (defined below), and policy revision (the §5.5 pair) — **this list references the one normative summary schema, the providers spec §6.3 authority-state wire record; it is not a second schema** — and **not** provider/version, @@ -1054,13 +1594,25 @@ Consumers of the resolved set in this epic: the stored grant's source class (§4's regime-scoped table). Any of these → no update, row flagged for the operational cleanup of §4.2 trigger 3. Sync never mints authority of its own. **Stored - jurisdiction ages too**: batch sync has no live geo, so the - jurisdiction it recomputes against is the one from the last browser - visit — and a visitor who moved from a permissive into a GDPR + jurisdiction ages too**: batch sync has no live geo. The strong summary's + jurisdiction is a tagged value, never an unlabelled country: + - `Live { jurisdiction, provider_id, observed_at }` comes from a successful + live geo lookup. + - `StaticDefault { jurisdiction, config_revision, observed_at }` comes only + from acknowledged §5.3 mode; S2S accepts it only while the active config + revision still selects the same static jurisdiction. + - `ProtectiveLookupFailure { provider_id, profile_revision, observed_at }` + records §5.2 evaluation but **never authorizes context-free S2S egress**. + A later successful live lookup must replace it first. The live request may + still perform only what its protective-profile resolution authorizes. + + For `Live` or matching `StaticDefault`, batch sync recomputes against the + stored jurisdiction from the last browser visit — and a visitor who moved + from a permissive into a GDPR jurisdiction would otherwise keep old-rule egress for up to the row lifetime. A stored jurisdiction older than the **consent-TTL - horizon** — age measured as now − `jurisdiction_observed_at`, the - summary's dedicated field written **only by live geo resolution** + horizon** — age measured as now − the tag's `observed_at`, written by the + browser-request resolution that produced `Live` or `StaticDefault` (providers spec §6.3; evidence timestamps are not a proxy: TCF `LastUpdated` can predate the live lookup, and a policy-baseline grant has no wall-clock evidence timestamp at all) — fails closed @@ -1084,21 +1636,24 @@ Consumers of the resolved set in this epic: 4. **Server-side auction dispatch** — gated on the policy `regime` class, normatively: - | Regime | Dispatch rule | Preserves | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | - | `gdpr` | Dispatch only with a decodable, unexpired TCF record consenting to Purpose 1. Malformed, expired, or absent record → **no bid request leaves** (no-bid response). | Today's GDPR/unknown arm | - | `us-privacy` | Dispatch proceeds in every signal state, including opt-out — the opt-out strips identity (rows above) but the contextual auction runs. | Today's US-state arm | - | `none` | Dispatch proceeds. | Today's non-regulated arm | - | **Any regime, TCF-sourced effective record** — a raw TC string on the request, a GPP section-2 hint (both detected **before decoding**), or a persisted-KV fallback record of TCF origin (§4.4) | The `gdpr` row applies: dispatch requires the _effective_ record to be decodable, unexpired, and consenting to Purpose 1. A **malformed or expired** raw signal therefore blocks dispatch — today a malformed raw TCF blocks, and gating this arm on decodability would have silently relaxed that. A US or non-regulated request carrying a Purpose 1 refusal is likewise blocked. | Today's raw-signal arm — **must not regress** | + | Regime | Dispatch rule | Preserves | + | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | + | `gdpr` | Dispatch only with a decodable, unexpired TCF record consenting to Purpose 1. Malformed, expired, or absent record → **no bid request leaves** (no-bid response). | Today's GDPR/unknown arm | + | `us-privacy` | Dispatch proceeds in every signal state. When P4 is set, the ordinary egress inventory applies. When P4 is unset, including any mapped opt-out, dispatch is allowed only through the exact `ContextualAuctionView` above; serializer or destination-registration failure produces no bid request. | Today's dispatch posture, with contextuality made enforceable | + | `none` | Dispatch proceeds. | Today's non-regulated arm | + | **Any regime, TCF-sourced effective record** — a raw TC string on the request, a GPP section-2 hint (both detected **before decoding**), or a persisted-KV fallback record of TCF origin (§4.4) | The `gdpr` row applies: dispatch requires the _effective_ record to be decodable, unexpired, and consenting to Purpose 1. A **malformed or expired** raw signal therefore blocks dispatch — today a malformed raw TCF blocks, and gating this arm on decodability would have silently relaxed that. A US or non-regulated request carrying a Purpose 1 refusal is likewise blocked. | Today's raw-signal arm — **must not regress** | The **compiled-in fallback policy has `regime = "gdpr"`** (§3.1) — the no-policy posture must be the most protective for dispatch too, and a regime-less fallback would leave dispatch undefined. When dispatch is blocked, nothing leaves for that request: no PBS/APS call, no UA/IP/geo - forwarding to bidders. When dispatch proceeds, what the request may - carry is governed row-by-row by the egress inventory; the full - regulatory context (consent strings) is always forwarded so downstream - partners make their own decisions (§1). + forwarding to bidders. When dispatch proceeds, what the request may carry + is governed row-by-row by the egress inventory and, whenever P4 is unset, + by the stricter positive contextual projection. Full regulatory + strings are forwarded only to a destination whose protocol normatively + requires them and whose registration declares it an authorized + privacy-signal consumer; every other destination receives normalized + outcomes or no regulatory field (§1). The client-cycle resolve endpoint (own spec, currently on hold) would be a further consumer if and when it proceeds. @@ -1119,6 +1674,15 @@ further consumer if and when it proceeds. (consent, opt-out, malformed, expired, absent), including the no-policy fallback regime, asserting both the dispatch decision and that a blocked dispatch emits no outbound request. +- The contextual serializer is tested as a positive schema against final + encoded OpenRTB bytes. Fixtures place forbidden values in every ordinary and + extension location (EC/derived IDs, user IDs/data/segments, IP/IPv6, UA, + precise geo, URL/referrer/query, device IDs/fingerprints, and forwarding + headers) and prove none survives; a destination without a qualified + contextual registration produces no outbound request. +- GPP transport fixtures derive sorted unique `gpp_sid` from decoded + applicability, never copy `__gpp_sid`, and assert atomic pair omission for an + unconstructable set plus restrictive mismatch handling. - The §7 S2S authority path: **every denial reason individually** — denied rule, tightened baseline without acceptable stored evidence, expired evidence, regime-rejected grant source — plus the exempt @@ -1132,6 +1696,12 @@ further consumer if and when it proceeds. → refusal, → opt-out, → malformed, → absent — plus a mid-replacement fault proving the surviving state is the complete old **or** complete new snapshot, never a merged mixture. +- Timestamp-less opt-out clearing: GPP/USP opt-out → later bare explicit + not-opted-out presentation remains suppressed; a newer TCF `LastUpdated` or + authenticated monotonic authorization revision clears → an identical + timestamp-less opt-out presentation starts a new restrictive episode without + refreshing its original first-seen, and the following no-signal live/S2S + decisions remain suppressed; replay and equal revisions never grant. - Legacy-row withdrawal end to end (§4.3's derived family ID). - §4.3 fault-injection cases. - Policy validation tests for every §3.3 rejection, exercised through both diff --git a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md index b47bbe450..df9325d66 100644 --- a/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md +++ b/docs/superpowers/specs/2026-07-30-pluggable-providers-design.md @@ -50,11 +50,11 @@ Non-goals: ## 2. Provider taxonomy -| Concern | Trait | Built-in default | Opt-in host implementation | -| ----------- | -------------------- | --------------------------- | ----------------------------------------------------------------- | -| EC identity | `EdgeCookieProvider` | none (stateless) | `hmac` (in core; HMAC over client IP, preserves today's identity) | -| Device | `DeviceProvider` | `builtin` (User-Agent only) | `fastly` (JA4 / HTTP-2 fingerprints) | -| Geo | `GeoProvider` | none (no location) | `platform` (host geo lookup) | +| Concern | Trait | Built-in default | Opt-in host implementation | +| ----------- | -------------------- | --------------------------- | ------------------------------------------------------------------------------ | +| EC identity | `EdgeCookieProvider` | none (stateless) | `hmac` (in core; HMAC over client IP, preserves today's identity) | +| Device | `DeviceProvider` | `builtin` (User-Agent only) | `fastly` (JA4 / HTTP-2 fingerprints) — deferred; startup-rejected in this epic | +| Geo | `GeoProvider` | none (no location) | `platform` (host geo lookup) | Selection keys are strings in operator configuration: @@ -98,7 +98,7 @@ through the selected provider: | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Mint** | EC generation on first eligible request | Provider returns the identifier (and only core writes the cookie). | | **Parse / canonicalize** | Reading `ts-ec` back from the request; deciding `ec_was_present`; batch-sync ingestion | Provider parses a cookie value into its **canonical** identifier, or rejects it. Canonicalization and **equivalence are provider-declared, never imposed globally**: each provider ships equivalence fixtures naming exactly which variants are the same identity — case sensitivity is provider-specific (signed/base64-style envelopes are case-sensitive; even the built-in HMAC id is case-insensitive only in its hex prefix, with a case-preserved suffix). Declared-equivalent values parse to the same canonical identifier (satisfying #778). A value the selected provider does not recognize is treated as absent (but see §6.1 legacy readers). | -| **Canonical graph key** | KV identity-graph row reads/writes | The provider supplies a canonical key **suffix**; **core constructs the physical key** per the §6.3 key grammar (legacy-HMAC verbatim keys excepted), so cross-provider and cross-record-kind isolation is enforced by construction rather than promised by provider code. Suffixes are stable, KV-safe (length and character-set limits), and collision-free within the provider's space. Two equivalent envelopes of one identity map to one key — verbatim cookie bytes as the key would fork graph rows on canonicalization differences and discard today's batch-sync canonicalization. | +| **Canonical graph key** | KV identity-graph row reads/writes | The provider supplies a canonical key **suffix**; **core constructs the physical key** per the §6.3 key grammar (legacy-HMAC verbatim keys excepted), so cross-provider and cross-record-kind isolation is enforced by construction rather than promised by provider code. A provider either emits an injective suffix within the 123-byte portable grammar or declares the collision-detecting SHA-256 form defined in §6.3. Two equivalent envelopes of one identity map to one key — verbatim cookie bytes as the key would fork graph rows on canonicalization differences and discard today's batch-sync canonicalization. | | **Cluster prefix** (optional capability) | IP-cluster sizing (`cluster_trust_threshold`, implemented as a **KV prefix listing**), pull-sync dedupe, log redaction | A provider declaring cluster support returns a prefix that is a **literal byte prefix of the canonical graph key** — the cluster count lists keys by prefix, so an independently derived hash that is not an actual key prefix silently reports the wrong cluster size. The prefix deliberately collides across identifiers minted from the same client evidence. A provider without the capability declares so, and cluster-dependent gating follows a configured degradation policy (treat cluster size as unknown, with the KV-write decision that implies made explicit in config) instead of counting garbage. | | **Tombstone** | Withdrawal: expiring the cookie and writing revocation markers | The identifiers eligible for tombstoning are exactly those the provider parses — never a shape-gated subset. | @@ -124,6 +124,25 @@ Three global rules sit above every provider: documents reference one number instead of assuming their own — for the identifier itself, not only the graph key — enforced by core at mint and at parse, so no provider can emit a value the cookie layer or logs cannot carry. +- **Graph-key representability.** The 256-byte identifier bound does not imply + that every identifier can be injectively encoded in the 123-byte physical-key + suffix. Each provider's namespace descriptor therefore declares exactly one + graph-key mode: `injective`, whose conformance fixtures prove a one-to-one + canonical suffix within the portable grammar and length cap; or + `sha256-detect`, whose suffix is `h` plus unpadded base64url of + `SHA-256("tsgk1|" || provider_code || canonical_identifier_bytes)` (44 ASCII + bytes total). Known-answer vector: provider code `vend` and canonical + identifier bytes `AbC123` produce + `hY1DtOYYBwBBUs-ZPabyh5JOC_aqc5mdCQngW7yfcHTY`. A `sha256-detect` row stores the canonical identifier bytes in + its encrypted/identity-bearing row envelope and every read compares them in + constant time before returning the row. A mismatch is a cryptographic key + collision: the read and attempted write fail closed, the existing row is + never overwritten or joined, and a fleet-visible security alert is emitted. + Such a provider cannot declare cluster-prefix support unless its graph-key + suffix independently preserves the required literal prefix. The security + model is explicit 256-bit collision resistance plus collision detection, + not the mathematically impossible claim that arbitrary 256-byte identifiers + map collision-free into 123 bytes. - **Namespaces are declarative and core-proven.** Disjointness of two opaque `parse` functions is not provable, so every provider declares a **static namespace descriptor** in a core-owned declarative form — a set @@ -326,7 +345,7 @@ variant**. Therefore: local clock compared against a store-issued deadline could cross it while another instance's N+2 lease was still valid. Where the backend has no store clock, `not_before` = committer time + L + - S\*fleet and every local comparison subtracts S*fleet again + S\*fleet and every local comparison subtracts S\*fleet again (mint only when `now − S_fleet ≥ not_before`) — **S_fleet is the maximum pairwise fleet clock skew, a declared and monitored infrastructure bound, a separate qualification from the @@ -354,7 +373,7 @@ variant**. Therefore: absent → active → suspended → re-attested-active → **suspended** (a second rollback) → … ; CAS losers on any transition re-read and retry against the winner's epoch. Under `suspended`, rowless - \_classification* stops but **`w` consultation and enforcement + _classification_ stops but **`w` consultation and enforcement continue** (withdrawn stays withdrawn). Re-activation after roll-forward requires complete re-attestation over the gap window. The N+2 → rollback-to-N+1 → mint → roll-forward-to-N+2 schedule is a @@ -372,8 +391,8 @@ variant**. Therefore: before any live or S2S use whenever a live `w` record exists** (keyed on the record's `valid_until`, never on the flag — an earlier "while the flag is active" scope contradicted the valid_until rule - one sentence later) (a pending or saturated entry means promotion-then-denial per - the runtime matrix, §6.2), so a withdrawn suffix cannot slip into use + one sentence later) (an exact pending suffix means promotion-then-denial; + saturation alone governs only rowless admission), so a withdrawn suffix cannot slip into use through the row path. **`w` consultation is keyed on the record's `valid_until`, not the rowless-classification flag** — the flag may clear after one cookie lifetime while `w` is retained through the longer max(cookie, row, S2S) horizon, and a late row must still find a live `w`; enforcement ends only when the `w` record itself expires. **The classification decision is an ordered procedure, not a @@ -385,11 +404,12 @@ variant**. Therefore: **first match wins**: disjointness holds by construction (each step assumes every earlier step did not match) and totality by the final default. **Negative gates are release-, config-, and - flag-invariant** — steps 2–4 run before the v1 exception and before + flag-invariant** — steps 2–6 run before the v1 exception and before every positive path, so the v1 exception relaxes only the _positive_ side and an N+2-written record still binds a rolled-back N+1 (migration spec §4.4): - 1. **Any read error** (strong records, `w`, or a consulted row read) + 1. **Any read error** (safety breaker, `q`, authority/family records, + `w`, or a consulted row read) → indeterminate: no use, no mint, no expiry, and no writes whose admission depended on the failed read. Admitted negative writes still proceed: a successfully read strong authority record proves @@ -398,23 +418,30 @@ variant**. Therefore: even when the eventual row read failed — only the browser-cookie expiry waits for the commit. A row read failure must never leave S2S authority live against an already-provable withdrawal. - 2. **Live `w` entry** (matching suffix hash, or saturated prefix) → - withdrawn: denied, and promoted to a family revocation at first - sight of a real row (§6.2). If the family is already revoked the - promotion is an idempotent no-op — `w` and revocation agree on - denial, so their overlap has exactly one outcome. Applies under - every semantics, v1 included. - 3. **Family revocation present** → denied — all semantics, all flag + 2. **Global identity safety breaker active** → no positive identity mint, + use, graph access, or egress. Negative repair, withdrawal, and deletion + operations continue so the deployment can recover. + 3. **Pending family intent in `q`** → apply its negative meaning + immediately: pending revocation denies the family; pending suppression + denies only its mapped permission. Workers continue the idempotent target + write; positive evaluation never skips ahead of the intent. + 4. **Live exact suffix in a `w` entry** → withdrawn: denied, and + promoted to family revocation at first sight of a real row (§6.2). + If already revoked, promotion is idempotent. **Saturated prefix with + no exact suffix match** → deny only a rowless candidate; an + authenticated row continues to the family/suppression checks below + and is never revoked from cohort membership alone. + 5. **Family revocation present** → denied — all semantics, all flag states. - 4. **Live suppression entry for a permission** → that permission is + 6. **Live suppression entry for a permission** → that permission is denied (identity retained — non-destructive); evaluation continues below for permissions without a live suppression. - 5. **v1 semantics** (release N+1, or N+2 under old-shape config) → + 7. **v1 semantics** (release N+1, or N+2 under old-shape config) → v1 positive behavior: recognized cookies are used per pre-epic rules, rows not required — the declared v1 exception, not an outage: the pre-epic privacy posture persists until the new model - activates (matrix row 14). Steps 2–4 have already run. - 6. **New model, row found:** + activates (matrix row 14). Steps 2–6 have already run. + 8. **New model, row found:** - authority record **absent or stub-only** (no positive summary) → no egress, but not a dead end: the live-backfill path applies — a live request resolving a regime-accepted grant runs the @@ -432,11 +459,12 @@ variant**. Therefore: the strong summary alone governs (exactly the S2S posture); a live request re-runs `AuthorityRefresh` to re-commit and realign the fence, then proceeds. - 7. **New model, row not found** (successful eventual read — which by + 9. **New model, row not found** (successful eventual read — which by itself proves nothing): - strong records for the derived family **absent on the authoritative strong-class read** and flag = `active` → - **rowless**: expire-and-re-mint / withdrawal per §5 — the + **rowless**: expire-and-re-mint, request-local non-destructive denial, + or destructive `w` withdrawal per §5 — the strong-class absence is the proof; the eventual read is never the evidence; - any strong record present → **visibility lag, not absence**: @@ -446,8 +474,8 @@ variant**. Therefore: - flag absent, cleared, or suspended → indeterminate (no rowless classification without an `active` flag; suspension pauses classification pending re-attestation). - 8. **Default** — any state not matched above → indeterminate: no - use, no mint, no expiry. + 10. **Default** — any state not matched above → indeterminate: no + use, no mint, no expiry. Graphless-era cookies never got a stub because they have no row for the scan to find; no eventual read participates. The flag itself is @@ -473,7 +501,14 @@ variant**. Therefore: values, so if the expiry response is lost the cookie survives and may resurface on the old network; every re-presentation re-attempts expiry. The residual is bounded to graphless-era cookies from changed - networks and is **sign-off item 29**. + networks and by the cookie's original absolute expiry — migration never + refreshes it. Telemetry counts rowless presentations, unverifiable + issued-version cookies, expiry attempts, and the oldest still-observed + issuance version. The guide discloses the residual and gives it a sunset: + the compatibility path may be removed only after no issued graphless-era + version is observed for a quiet period at least the maximum original + cookie lifetime plus rollout skew. This acceptance is **sign-off item + 29**. - **Rowless withdrawal writes into one capped per-prefix record, then expires the cookie** — durable (cookie-only expiry is best-effort: a lost response leaves the "withdrawn" cookie usable), and **bounded in @@ -482,14 +517,14 @@ variant**. Therefore: per-variant records would be attacker-priced strong storage. The record (strong class, keyed on the verified prefix) holds a bounded list (cap 8) of withdrawn-suffix hashes; writes are admitted only for - **prefix-verified** cookies; **saturation escalates to prefix-wide - rowless revocation** — every rowless cookie under that prefix is - treated withdrawn, which harms only the abuser's own same-IP graphless - cohort and is the declared abuse response (legitimate users hold one - or two variants ever). A re-presented withdrawn variant finds its - entry (or the saturated record) and stays dead. Row-backed - withdrawal is untouched: full-graph-key family records, one derivation - everywhere. + **prefix-verified** cookies. Saturation blocks every further _rowless_ + admission under that prefix and forces cookie expiry, but it is not + evidence that a row-backed family withdrew. When a real row later + surfaces, only an exact listed suffix promotes to family revocation; + the saturated flag alone never revokes or denies an authenticated row + belonging to another member of the NAT cohort. A re-presented exact + withdrawn variant stays dead. Row-backed withdrawal remains keyed by + full-graph-key family ID. - **Negative-record creation has admission rules everywhere — and rowless identifiers get no per-family records at all.** Durable suppression and family-revocation records may be written for an @@ -497,17 +532,19 @@ variant**. Therefore: strong read) — **or for a positively observed real row**: a successful row read is safe admission evidence (an eventual not-found is not), and without this arm the first post-upgrade GPC request could not - revoke an untouched v1 row, and the promotion path could not promote a + suppress P4 for an untouched v1 row, and the promotion path could not promote a late-surfacing row (neither has a stub by definition). There are **two** permission-exempt observed-row sequences, because one shape cannot serve both signal classes (the single revocation-shaped sequence - either destroyed identities for non-destructive opt-outs or dropped - their suppression entirely). **Destructive** (GPC, sale, USP): derive - the family ID → create-if-absent a minimal strong stub carrying no - positive authority → commit the family revocation → the identity is - denied all use between discovery and revocation commit → only then - expire the browser cookie. **Non-destructive** (SharingOptOut / - TargetedAdvertisingOptOut, refusal, malformed): row read → same stub + either destroyed identities for use opt-outs or dropped their + suppression entirely). **Destructive** (explicit storage-consent + withdrawal or authenticated deletion): derive the family ID → + create-if-absent a minimal strong stub carrying no positive authority + → commit the family revocation → the identity is denied all use between + discovery and revocation commit → only then expire the browser cookie. + **Non-destructive** (GPC, SaleOptOut, USP sale opt-out, SharingOptOut, + TargetedAdvertisingOptOut, malformed, or a TCF refusal that does not + meet permission spec §4.2's destructive P1 conditions): row read → same stub create-if-absent → **CAS the per-permission suppression entry** → the suppressed permission is denied use while the sequence is incomplete → **the family and the cookie are retained** (nothing is @@ -520,9 +557,22 @@ variant**. Therefore: records would let one prefix-holder fabricate unlimited suffixes into unlimited strong-storage records (rate limits slow creation; they do not bound cardinality), and a rowless identity has no authority to - suppress anyway. **The capped per-prefix withdrawal record is the - entirety of rowless negative state.** Fabricated, unverifiable - cookies write nothing anywhere. + suppress anyway. **For the old rowless identity, a refusal, + malformed-present record, GPC, SaleOptOut, USP sale opt-out, + SharingOptOut, or TargetedAdvertisingOptOut is request-local:** it denies + the mapped permissions for this request but creates no `s`, `q`, `fam`, or + `w` state for that rowless family. The `w` class is reserved for explicit + storage withdrawal or authenticated deletion; non-destructive signals never + enter it. If P1 permits this request to expire and re-mint through the + ordinary graph-backed path, the newly created family's authority-state + commit includes the current suppression before its cookie or identity may + become usable; a failed suppression write follows the ordinary negative- + intent/breaker protocol. Durable suppression then belongs to the new + row-backed family, never the unauthenticated old suffix. If no new family is + minted, nothing durable exists and a later presentation is reevaluated from + its current signals. **The capped per-prefix withdrawal record is the + entirety of old-rowless negative state.** Fabricated, unverifiable cookies + write nothing anywhere. **Egress is typed, not policed.** The inventory-and-denylist test (permission model spec §7) is a backstop, but conventions do not survive @@ -562,31 +612,18 @@ structural: - **Geo: circularity.** The permission set is resolved _from_ jurisdiction, which is resolved _by_ the geo provider. Gating geo on the resolved set is unsatisfiable. -- **Device: a decision, not a circularity.** Device classification is not - an input to permission resolution (the inputs are jurisdiction, policy, - and signals), so ordering geo → resolution → device → EC and gating - device is perfectly implementable. This spec deliberately does not: - the shipped device providers process technical request metadata (UA, - JA4/HTTP-2 fingerprints) for **security classification** — the bot gate - protecting KV-backed identity writes — which must run precisely for - traffic that has granted nothing. The authorization for that processing - is the operator's explicit `[device] provider` selection — a statement - about the **opt-in host-fingerprint provider**; the `builtin` UA-only - default processes nothing beyond the User-Agent every request already - carries and needs no such authorization — and this spec records that as - the decision, with its privacy implication stated: a - device provider whose data use goes beyond security classification (for - example feeding fingerprints into targeting or identity) is **not - authorized by selection alone** and requires a vocabulary extension plus - a gate before it may ship. This bites immediately, not hypothetically: - today's graph rows persist the JA4 class, an HTTP/2 fingerprint hash, - and buyer-facing quality metadata — persistence and scoring that exceed - security classification. The epic therefore **stops writing - fingerprint-derived buyer-facing fields into new rows** (a declared - change, migration spec §2); the boolean security classification outcome - may be persisted. Re-adding them is the vocabulary-extension route. - The field-level graph contract itself is normative in this spec — - §6.3 — not deferred to the implementation. +- **Device: host fingerprinting is deferred, not authorized by selection.** + Device classification is not an input to permission resolution, so a + separate security/fraud design can order geo → resolution → security + classification → EC and define its own authority. This epic ships only + the `builtin` UA-only classifier. Selecting a provider that reads JA4, + HTTP/2 fingerprints, or comparable host evidence is a startup error until + a reviewed design defines the exact security purpose, lawful/permission + basis, retention, deletion, downstream visibility, and an explicit ban on + advertising or identity reuse. Existing fingerprint-derived graph fields + remain read-only for migration, never egress, and are scrubbed on rewrite; + new rows persist neither raw fingerprints nor a fingerprint-derived + classification outcome. PR #838 declared `required_permissions` on all three traits but consulted it only for the EC provider; the geo and device declarations were @@ -699,60 +736,64 @@ Startup validation (§6) covers configuration; this covers what happens when a healthy configuration meets an unhealthy runtime. Every row logs at `error` with a metric; none is silent: -| Failure | Behavior | -| ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | -| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | -| Authority-state commit fails after the row commit at mint | No cookie, no eligibility (the strong record never reported the revision); the orphan row expires by TTL; counted — **no recovery claim**, nothing can find it | -| `w` read fails (rowless path, or any HMAC row discovery while a live `w` record exists — enforcement outlives the migration window) | Fail closed: the cookie/row is **indeterminate** — no use, no mint, no expiry this request | -| `w` write fails at rowless withdrawal | Cookie retained (withdrawal did not durably occur); durable client signal retries next presentation | -| `w` saturation encountered, a real row surfaces under the prefix | **Denied, then promoted to a family revocation** — a saturated prefix means the safe assumption is "withdrawn", so any real row under it is denied all use and its family revoked, listed-hash or overflow alike. The earlier "overflow loses promotion, never blanket-denies" rule left a completed rowless withdrawal usable the instant its row surfaced; that resurrection is closed here, not left to retention. The collateral — a non-abuser row under a saturated NAT prefix is revoked — is sign-off 30 | -| Promotion (listed suffix-hash matches a discovered row) fails | The row is denied all use until the promoted family revocation commits; retried on next observation | -| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | -| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | -| Tombstone write fails | Permission model spec §4.3: family retries, readers fail closed on partial families | -| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → `default_country` (permission model spec §5.2), counted in the lookup-failure metric | -| Device provider signals unavailable at runtime (e.g. no JA4 on a request) | Classification degrades per the provider's declared fallback, never silently upgrades `looks_like_browser` | - -The **degraded-graph health signal** referenced above and by the -withdrawal contract is a defined state machine, not a vibe: it is -**per-instance and in-memory** (no shared propagation, no stored health -record whose own read could fail), entered when graph-write failures cross -a sliding-window threshold (N failures within window W), and exited with -hysteresis after M consecutive successes. While degraded: S2S partner -egress and sync updates fail closed; organic requests continue stateless. +| Failure | Behavior | +| ----------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `generate` returns an error | No identity this request; request proceeds stateless; no cookie written | +| Graph-row commit fails at mint | Mint never happened (§5): no cookie, no egress; next request retries | +| Authority-state commit fails after the row commit at mint | No cookie, no eligibility (the strong record never reported the revision); the orphan row expires by TTL; counted — **no recovery claim**, nothing can find it | +| `w` read fails (rowless path, or any HMAC row discovery while a live `w` record exists — enforcement outlives the migration window) | Fail closed: the cookie/row is **indeterminate** — no use, no mint, no expiry this request | +| `w` write fails at rowless withdrawal | Cookie retained (withdrawal did not durably occur); durable client signal retries next presentation | +| `w` saturation encountered, a real row surfaces under the prefix | Exact listed suffix: denied and promoted. No exact match: saturation alone is not withdrawal evidence; the authenticated row proceeds through its own family revocation and suppression checks. No NAT-cohort collateral revocation. | +| Promotion (listed suffix-hash matches a discovered row) fails | The row is denied all use until the promoted family revocation commits; retried on next observation | +| Graph read fails on an existing identity | Identity unusable this request (fail closed for egress); cookie untouched | +| Cluster prefix listing fails | Treated as cluster-size-unknown → `cluster_fallback` policy applies | +| Revocation or suppression write fails | Enqueue the idempotent negative intent in the durable outbox. Until its strong target record commits, every identity decision observes the intent and applies its negative scope. If outbox enqueue also fails, trip the globally visible identity safety breaker immediately (permission spec §4.3). | +| Geo provider returns invalid output (unparseable country) at runtime | Treated as lookup failure → protective failure profile (permission model spec §5.2), counted in the lookup-failure metric | +| Host-fingerprint device provider selected | Startup error in this epic; only the builtin UA-only provider is eligible (§5) | + +The **degraded-graph health signal** for ordinary row availability is a +defined per-instance state machine, entered when graph-write failures cross +a sliding-window threshold and exited with hysteresis. It is not the +negative-intent safety mechanism: revocation and suppression failures use +the durable outbox and globally visible identity breaker above. While ordinary +graph health is degraded, S2S partner egress and sync updates fail closed; +organic requests continue stateless. The thresholds ship as constants with the implementation and are printed in the startup log. -Its protection is therefore **local-only, and the spec says so**: a -backend-wide outage degrades every instance through its own observations -within one window, but an instance-local family-write failure leaves -other instances — which have no record to find, and healthy backends of -their own — serving S2S egress until the browser's durable signal retries -successfully. That residual is **unbounded for a never-returning visitor** -(sign-off item 11 — the permission and migration specs state this and -this spec must not undercut them), is counted (failed family writes are -a first-class metric), and is accepted -in place of a deployment-wide shared fail-closed channel, whose own -availability and freshness would be a harder problem than the one it -solves. +The ordinary health signal is local-only by design. Negative intent is not: +the outbox is durable, workers retry to the strong record, and a failed +enqueue trips a fleet-visible breaker. A deployment that cannot provide or +read this state is ineligible for persisted identity use; no +never-returning-visitor residual is accepted. ### 6.3 Storage contract — normative **Physical key grammar.** Core constructs every key; providers supply only the bounded suffix: -| Record class | Key | Notes | -| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Identity row (non-hmac) | `id//` | Suffix from `graph_key_suffix`, ≤ **123** bytes (128-byte total key cap minus tag and provider code — an earlier 128 here contradicted the constructor; 123/124 boundary fixtures required), portable alphabet **`[A-Za-z0-9._~-]`** (not "KV-safe per host", which would break the single cross-adapter grammar; providers with native identifiers outside it emit base64url suffixes); ≤ **123** bytes with 123/124 boundary fixtures. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | -| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | -| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | -| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | -| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | -| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS** (concurrent suffix withdrawals must not overwrite each other's entries); value: bounded suffix-hash list (cap 8) where **each entry carries its own `valid_until`** = its withdrawal time + max(cookie lifetime, row/S2S authority horizon) — one record-level lifetime either shortchanged late entries or, rolling, let an attacker keep a saturated NAT cohort withdrawn forever; the record expires when its last entry (or the saturation flag's own pinned horizon) expires; saturation flag with its own entry-time-pinned horizon; CAS version; created-at — a `w` expiring before the row horizon would let an overflow row's identity resurface (the overflow non-promotion residual is sign-off 30); readable fail-closed by N+1 after rollback (the flag implies N+2 had converged) | -| Deployment metadata | `m` + fixed metadata name (fixed-width grammar): schema floor, graphless-migration flag, policy-activation register | Write-once/CAS class. The **floor value is encoded**: integer writer-activation schema version + minimum-reader version, ordered numerically; a binary starts only if its declared reader capability ≥ the floor's minimum-reader, which is what makes "is N+1 permitted after N+2 activates" decidable (N+1 declares N+2-reader capability, so yes). The **graphless flag's** value carries schema version, state, epoch, set-at, **`not_before`, its clock domain, and `L`** (serialized, so every observer reads the same deadline) plus the attestation; lifecycle: created by the migration readiness step **after** N+2 convergence + stub-backfill completion (both attested in the value), cleared by explicit operator CAS with the quiet-period criterion recorded. The **policy-activation register** holds the current `{source_version, policy_digest, ordinal, activated_at}` **plus a bounded 16-entry activation history**, with the transition rules of permission spec §5.5 (history adoption for same-activation idempotence; same `source_version` with a different digest fails closed; stale rejected) | -| Rewrite transaction _(informative — deferred)_ | `rwx/` | One in-flight rewrite per family | -| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | `resv/…` | Deferred client-cycle draft | +| Record class | Key | Notes | +| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Identity row (non-hmac) | `id//` | Suffix from the provider's declared `injective` or `sha256-detect` graph-key mode, ≤ **123** bytes (128-byte total key cap minus tag and provider code; 123/124 boundary fixtures required), portable alphabet **`[A-Za-z0-9._~-]`**. Native identifiers outside that representable space use the assigned collision-detecting form above rather than an unqualified claim of collision-free base64url encoding. **No version segment** — the mint version lives in the row envelope; a versioned key would be circular (core would need the version to read the row that states the version) | +| Identity row (hmac, **every** version) | The identifier verbatim (`{64hex}.{6alnum}`) | Reserved grammar for the whole provider, not just v0 — keeping all HMAC versions on the verbatim scheme is what keeps the 64-hex cluster prefix a literal key prefix for every HMAC row. (Passphrase rotation still changes a given IP's HMAC, so clusters split across rotation until old identities expire — inherent to rotation, declared, not a key-scheme artifact) | +| Alias | **The source identity key itself** — the alias is a value written _at_ the replaced row's address, distinguished by a `kind` discriminator in the JSON envelope | A separate `alias/…` address could never work: lookups by the old cookie hit the source key, and a single-key CAS cannot install a record at a different address | +| Family revocation | `fam/` | Family ID from mint or the deterministic legacy derivation (permission spec §4.3) | +| Authority-state (suppression + positive-authority summary) | `s` + family ID (fixed-width grammar) | Per-permission negative entries **and** the positive summary (permission spec §4.3); permission-exempt writes; consulted by every constructor and S2S recompute | +| Negative-intent outbox | `q` + family ID (fixed-width grammar) | One strong CAS record per family in a failure domain independent of the authority/revocation target. It holds the permission spec §4.3 queue schema and idempotent pending negative transitions. Every live, cached, and S2S identity decision checks it before positive use; a pending intent denies its mapped scope until applied. Workers commit the target before CAS-acknowledging the exact intent. | +| Rowless prefix-withdrawal record | `w` + provider code + 64-hex prefix | Strong class, **linearizable CAS**; bounded exact suffix-hash list (cap 8), each with `valid_until` = withdrawal time + max(cookie, row, S2S horizon), plus a saturation flag that blocks only further rowless admission. The record expires after its last exact entry and saturation migration window. Only an exact suffix match can promote to family revocation; the prefix flag never authorizes cohort-wide row revocation. Read fail-closed for rowless classification and exact-match enforcement. | +| Deployment metadata | `m` + fixed metadata name (fixed-width grammar): schema-floor compatibility mirror, graphless-migration flag, policy/config/model-activation register, config-sequence register, global identity safety breaker | Strong-read/CAS class outside rollbackable config. The authoritative row-schema floor and binary/model epoch live in the activation register and advance in its fleet-fenced model CAS; `m00` is only a monotonic startup mirror and never enables writes. The register also carries immutable active/settings-candidate/model-candidate bindings, authoritative fleet readiness and quiescence, the activation-journal head, and a 16-entry operational history; the journal supplies the independent time-based audit/GC horizon. The config-sequence register allocates envelope `push_sequence` independently of ordinary config-store `put`; the global breaker disables every positive identity operation when neither a target negative record nor its fault-isolated outbox intent can commit, while leaving repair/deletion paths enabled. | +| Rewrite transaction _(informative — deferred)_ | **No physical key allocated in this epic** | A revived rewrite design must allocate a delimiter-free class in this registry; the old `rwx/` sketch is not valid grammar | +| Replay reservation _(informative — deferred with client-cycle; not normative surface)_ | **No physical key allocated in this epic** | A revived client-cycle design must allocate a delimiter-free class in this registry; the old `resv/…` sketch is not valid grammar | + +`m00` completion is part of the metadata wire contract, not an operator +overwrite: after the authoritative model CAS, the authenticated deployment +controller strong-reads active and `m00`; a missing or lower mirror is CAS-set +to exactly `active.row_schema_floor`, equality is an idempotent no-op, and an +unreadable mirror or failed CAS/read-verification remains closed for retry. +The controller then strong-reads and verifies exact equality. The operation +never lowers the mirror and never authorizes or changes active; +`m00 > active.row_schema_floor` is rejected before any write as a fail-closed +register/journal inconsistency, not an automatic repair case. Grammars are pairwise non-intersecting by their literal prefixes (plus the reserved hmac grammar), and every record value carries a `kind` @@ -766,9 +807,9 @@ migration, shared storage, and parity; and Fastly's prefix queries reject both `/` and `:`, so no delimiter character is safely portable). Physical keys are **delimiter-free with fixed-width segments**: a 1-character class tag — `i` row, `r` family revocation, `s` -authority-state, `x` transaction, `w` rowless prefix-withdrawal, `m` -deployment metadata (the full enumeration; earlier lists omitted `w` and -`m`), every tag chosen **outside the hex alphabet** so no +authority-state, `q` negative-intent outbox, `w` rowless prefix-withdrawal, +`m` deployment metadata (the complete normative enumeration for this epic), every +tag chosen **outside the hex alphabet** so no generated key can begin with 64 hex characters, which is what makes disjointness from the legacy `{64hex}.{6alnum}` grammar _provable_ rather than asserted (an earlier `f` tag was itself a hex digit) — then @@ -796,12 +837,14 @@ input `tsfam1|` + `i` + `vend` + `abcdef` → `w` suffix hashes are likewise assigned: SHA-256 with domain tag `tswsx1|` over the raw suffix bytes, truncated to 16 bytes, lowercase hex (32 chars) — vector: `AbC123` → `08cb55acf42929772862e82b0960c134`. -Cross-language vector suites extend these) for revocation/authority records (no provider code: the family id -already encodes derivation); `w` + provider-code(4) + prefix(64 hex) for -rowless withdrawal; `x` + family-id(64) for transactions; `m` + a **2-digit +Cross-language vector suites extend these) for revocation/authority records +(no provider code: the family id already encodes derivation); `q` + +family-id(64) for the negative-intent outbox; `w` + provider-code(4) + +prefix(64 hex) for rowless withdrawal; `m` + a **2-digit registry-assigned index** for deployment metadata (a closed name -registry in this spec: `00` schema floor, `01` graphless-migration -flag, `02` policy-activation register — padding-based names aliased `foo` and `foo-`, so names are not +registry in this spec: `00` schema-floor compatibility mirror, `01` graphless-migration +flag, `02` policy/config/model-activation register, `03` config-sequence allocator, +`04` global identity safety breaker — padding-based names aliased `foo` and `foo-`, so names are not encoded in keys at all). Maximum physical key length **128 bytes**; every class has a total parser, and segment boundaries are positional, so no segment can contain or escape a @@ -813,24 +856,38 @@ reserved exception, with the 64-hex cluster prefix at position zero.) **Wire schemas** (JSON, like identity rows; every class carries a schema version): the **alias record** (reserved-future, with rewrite) holds target key, created-at, retirement deadline, and fencing epoch; the **family revocation record** holds the -family ID, revoked-at, triggering signal class (§4.5 destructive column), +family ID, revoked-at, triggering withdrawal class (explicit storage +withdrawal, authenticated deletion, or qualifying TCF Purpose-1 refusal), and a **family epoch** bumped on every revocation-state change (the client-cycle commit CAS is conditioned on it) — deliberately no identity data, so it can outlive its members; the **authority-state record** holds, per permission — negative side: state (`suppressed`/`cleared`), cause, source class, authoritative or -observation evidence timestamp, entry `valid_until` (evidence-class TTL; -expired entries are inert), and the provenance revision a clear -references; positive side (the summary, **every field the permission +observation evidence timestamp, optional entry `valid_until` (required for +refusal/malformed/absence; absent for a persistent use opt-out), and the +provenance revision plus explicit newer-authorization evidence a clear +references. Each permission also carries a monotonic +`authorization_revision: u64`: a persistent opt-out records the current value +as its clearing floor; only an authenticated same-subject authorization CAS +may increment it, and a clear based on that channel requires a value strictly +above the floor. Overflow is a hard error. Ordinary GPP/USP/GPC presentation +never increments this field. Positive side (the summary, **every field the permission spec's absence/replay decisions consume — a reduced schema cannot reproduce them**): kind (user evidence vs policy baseline), grant basis/source class, policy revision (the §5.5 pair: digest + activation -ordinal), **resolved jurisdiction with its own `jurisdiction_observed_at`** -— set **only by live geo resolution**, never derived from evidence -timestamps (TCF `LastUpdated` can predate the live lookup, and a -policy-baseline grant has no wall-clock evidence time at all), so S2S -never reads jurisdiction from an eventually stale row and decision 25's -stored-jurisdiction age gate has a field that actually measures -jurisdiction age (the summary is self-sufficient for the recompute) — +ordinal), and **tagged jurisdiction provenance** — exactly one of +`Live { jurisdiction, provider_id, observed_at }`, +`StaticDefault { jurisdiction, config_revision, observed_at }`, or +`ProtectiveLookupFailure { provider_id, profile_revision, observed_at }`. +The first comes from successful live geo, the second only from acknowledged +static mode, and the third carries no jurisdiction and is never eligible for +context-free S2S egress. `observed_at` is written by the browser-request +resolution that produced the tag, never derived from evidence timestamps (TCF +`LastUpdated` can predate it, and a policy-baseline grant has no wall-clock +evidence time at all). S2S therefore never reads jurisdiction from an +eventually stale row; static authority is fenced to its exact active config, +lookup failure is structurally non-authorizing for S2S, and decision 25's +stored-jurisdiction age gate has a field that actually measures jurisdiction +age (the summary is self-sufficient for the recompute) — `valid_until`, provenance revision, evidence timestamp, and a **bounded replay history** whose slots are keyed by a **timestamp-independent `state_key`** — (source class, semantic result digest _excluding_ `LastUpdated`) — distinct from the _evidence @@ -840,95 +897,132 @@ source token, not an enum byte** (an earlier draft said "enum bytes: tcf=1…" while its own vector hashed the ASCII token; the vector was right, the prose wrong — enum bytes exist only in storage, never in hash input): `state_key` = SHA-256 `tsstk1|` + `` (`tcf` / -`gpp` / `usp`) + `|` + canonical semantic result covering **all enforced +`gpp` / `usp` / `gpc`) + `|` + canonical semantic result covering **all enforced permissions for that source** (slots are **per source**, not per permission·source — the result string carries every permission, as the vector shows; timestamps integer epoch-ms where present); evidence digest = SHA-256 `tsevd1|` over the same, **plus `|lu=` only for sources with an intrinsic authoritative timestamp (TCF); -timestamp-less sources (GPP, USP) omit the `|lu=` field entirely — +timestamp-less sources (GPP, USP, and GPC) omit the `|lu=` field entirely — omission is the canonical form, no sentinel value exists**. Vectors: `tsstk1|tcf|p1=grant,p4=refuse` → `a49148a0e3b486fd93a141404857868871319a7e1f2ef85b5499aed80c7e59df`; `tsevd1|tcf|p1=grant,p4=refuse|lu=1690000000000` → `67259c0247ae2b33c52d9f18193bcd622f48ad4754ffbab6998d7c293b0143b4`; timestamp-less form `tsevd1|gpp|p1=grant,p4=refuse` → -`89b08580c214070b6d1d58ad57c12bd585134f324c718cbc0802b4d82a0d72e6`: keying slots on the +`89b08580c214070b6d1d58ad57c12bd585134f324c718cbc0802b4d82a0d72e6`; +GPC vectors `tsstk1|gpc|p4=opt-out` → +`3a0ff99076bfa849edd18da99018e1993251d128c322a1d73857f6057eb3fe96` +and `tsevd1|gpc|p4=opt-out` → +`a5df50001cd9ef127365a18754e96d43527e2c3f34e94bcd03e04e0a1f65b28c`: +keying slots on the evidence digest would give every renewal a fresh key and make "updates its slot in place" impossible, the incompatibility an earlier -draft shipped. A slot stores its `state_key`, the current evidence -digest, that digest's pinned first-seen, the newest authoritative -timestamp observed, **and a per-permission `observed_at` map** (bounded -by the enforced permission set) — maintained by a defined transactional -algorithm, since the comparison base and copy-forward source were -otherwise ambiguous: the record stores a per-source -**`current_state_key` pointer** naming the active slot; on a live -resolution producing combined vector V, the comparison base is **the -current slot's stored vector** (the immediately preceding active -state — never the vector of whatever old slot `key(V)` happens to -match); per permission, a changed token takes the new observation's -timestamp and an unchanged token **copies `observed_at` forward from -the current slot**; the target slot — fresh, or a previously occupied -slot being returned to (A→B→A) — has its per-permission map **replaced -by the computed map** (its stale map is never merged; its pinned -first-seen digest handling is unchanged); then -`current_state_key := key(V)`, all inside the record's single CAS. -A→B→A and P4-only-change are named test vectors. This is what makes -the permission spec's per-permission equality digests real (a P4-only -change must never refresh P1's age) without per-permission slots; a TCF renewal (same `state_key`, newer -`LastUpdated`) updates the slot in place, while a replay (not newer) -changes nothing — replay protection derives from recency comparison, -not per-value history, so no per-digest sublists are needed. 16 slots -per source; entries live to the evidence/suppression -horizon. **Saturation is a fixed epoch, not a rolling slot** (one slot -cannot hold independent timestamps for multiple overflow digests): when -all slots hold distinct in-horizon states, the record sets a -`saturation_epoch` with `saturated_until = now + consent TTL`, **fixed -at entry and never extended by later overflow values**; while -saturated, novel values cannot grant (fail restrictive); a -**restrictive overflow** (timestamp-less opt-out or malformed) is -recorded under an **epoch-scoped restrictive marker pinned at first -restrictive overflow**: the marker takes that _first_ overflow's own -observation timestamp and a full consent-TTL `valid_until` from it — -replays and later overflow values neither advance nor extend it (the -unpinned version let repetition renew denial forever), and pinning to -the epoch's entry instead would have back-dated a genuinely new opt-out -and expired it early. Later restrictive overflows within the epoch -inherit the marker — the **declared saturation exception** (sign-offs -16 and 31 both carry it): a genuine opt-out arriving as a restrictive -overflow late in the epoch receives **less than its §4.3 full-TTL -lifetime, down to nearly zero at the epoch's end**. This is a product -choice, not an accident: per-overflow state is exactly what saturation -exists to avoid storing (unbounded slots), and refreshing the marker on -later overflows would let replays of evicted values extend denial -indefinitely (the anti-replay bound) — so the shortening is accepted, -bounded to sources presenting ≥ 16 distinct in-horizon states, and -ratified explicitly. **Epoch expiry is a complete transition, not a -hope**: at `saturated_until` the epoch ends; expired slots are lazily -garbage-collected on the next write; if a slot is then free, the record -leaves saturation and novel values occupy slots normally, each with its -own full TTL; if every slot still holds an unexpired state — TCF -renewals legitimately extend slots past `saturated_until`, since slots -are individually lived and the epoch bounds only untracked overflow — -the next novel value opens a **new epoch**. **The restrictive marker -is marker-scoped, not epoch-scoped, in lifetime**: it lives to its own -`valid_until` (first-overflow-pinned + full TTL), **outliving the -epoch that created it when the first overflow arrived late** — -discarding it at `saturated_until` would deny that first overflow its -promised lifetime, and the record holds **at most one live marker per -source**, so overlapping markers never need representing. While the -marker lives, every later restrictive overflow — same epoch or a -successor — inherits it (the anti-replay inherit rule; the declared -shortening therefore spans epoch boundaries); only after it expires -does the next restrictive overflow pin a fresh marker at its own -timestamp. "Epochs never chain timestamps" means exactly this: a -**fresh** marker derives from its own overflow, never from epoch state -or a prior marker. The grant rule at the boundary: "novel values -cannot grant" is epoch-scoped and ends at `saturated_until`; the -marker carries only the restrictive/denial state of the overflow -opt-outs, evaluated by its `valid_until` alone. Saturation is a -first-class metric — -the cap and its denial behavior are **sign-off item 31**; record level: family ID, +draft shipped. Replay transition is **source-specific**, because TCF has +an authoritative clock and GPP/USP/GPC do not: + +- **TCF:** the record stores a per-source `max_last_updated`. A value with + `LastUpdated <= max_last_updated` is replay and changes nothing, even on + A→B→A. A value with a strictly newer valid `LastUpdated` is genuine newer + evidence: it updates or returns to the semantic `state_key` slot, replaces + that slot's evidence digest, and refreshes only the permissions represented + by the new evidence. Same semantics plus newer `LastUpdated` is a valid CMP + renewal. No first-seen timestamp participates in TCF authority age. +- **GPP/USP/GPC:** a slot stores `state_key`, evidence digest, and a bounded + per-permission `observed_at` map. The record stores `current_state_key`. + For a new vector V, unchanged permission tokens copy `observed_at` from + the current slot; a changed token reuses the earliest live-slot + `observed_at` for the same `(permission, token)`, else uses the current + observation. The target slot's map is replaced, never merged. Thus + A→B→A, grant→refusal→same-grant, and equivalent encodings do not refresh + timestamp-less authority; a P4-only change leaves P1 age untouched. + `observed_at` orders acquisition and non-user suppression recovery only; it + never proves that a bare timestamp-less grant is new enough to clear a + persistent use opt-out. That clear needs TCF's authoritative + `LastUpdated` or the authenticated `authorization_revision` path above. + After such a clear, presentation of the same restrictive GPP/USP/GPC value + starts a new suppression episode: it keeps the evidence's original + `observed_at`, records the current authorization revision as the new clearing + floor, and therefore remains effective for later no-signal/S2S decisions. + Restrictive presentation can never clear any state. + +Both branches update `current_state_key` inside the authority record's one +CAS. Named vectors cover TCF same-state renewal, TCF stale A→B→A replay, +GPP/USP/GPC A→B→A, equivalent encoding, grant→refusal→same-grant, and P4-only +change, plus persistent opt-out → later bare not-opted-out (does not clear) → +authenticated authorization-revision increment (clears) → identical +timestamp-less opt-out (new restrictive episode) → no-signal/S2S remains +suppressed. + +There are 16 slots per source. Saturation is restrictive without creating +collateral withdrawal: + +1. Expired slots are reclaimed first; otherwise grant-class slots are + eviction candidates before refusal or opt-out slots. +2. A novel grant with no eligible slot cannot authorize and is rejected + fail-restrictive. Repetition cannot create more slots. +3. A restrictive overflow updates one bounded per-source, + per-permission marker. A valid use opt-out marker has no time-based + expiry and clears only on strictly newer explicit authorization or + identity deletion. A refusal/malformed marker receives its complete + evidence horizon from the new observation; it never inherits an older + marker's nearly expired clock. +4. Saturation is a first-class metric and rate-abuse signal. It never turns + one source's history pressure into revocation of another family, and it + never shortens a newly observed opt-out. + +The **negative-intent outbox record** is the other strong wire schema. Its only +physical constructor is `q` + family-id(64 lowercase hex), exactly 65 bytes; +the parser rejects every other length, alphabet, or family-id case. It holds +`schema_version`, family ID, monotonic `queue_revision`, and at most 32 entries +sorted by lowercase 64-hex `intent_id`. Each entry stores the canonical target +key, the canonical transition payload below, and `enqueued_at_unix_ms`. The +payload materializes these keys, including nullable values, before hashing: +`cause` (closed string token), `clearing_floor_authorization_revision` (integer +or null), `evidence_digest` (64 lowercase hex), `evidence_time_unix_ms` +(integer), `permission` (permission ID or null), `source_class` (closed string +token), `state` (`revoked` or `suppressed`), `transition_kind` +(`family_revocation` or `suppression_strengthen`), and +`valid_until_unix_ms` (integer or null). Integers are in `0..=2^53-1`, never +strings or floating-point values. +The closed `source_class` tokens are `tcf`, `gpp`, `usp`, `gpc`, `absence`, +`storage-withdrawal`, and `authenticated-deletion`. The closed cause tokens are +`tcf-refusal`, `gpc`, `sale-opt-out`, `sharing-opt-out`, +`targeted-advertising-opt-out`, `malformed-present`, `absence`, +`explicit-storage-withdrawal`, `authenticated-deletion`, and +`tcf-purpose-1-withdrawal`. Signal evidence uses the authority record's +canonical `tsevd1|` digest. Absence uses +`SHA-256("tsevd1|absence|" || permission-id || "=absence")`; explicit +withdrawal/deletion uses SHA-256 over `tsneg1|`, the source-class token, `|`, +and the canonical authenticated audit-event bytes defined by that endpoint. +Raw action tokens, subjects, and credentials never enter the outbox. +`permission` and `clearing_floor_authorization_revision` are non-null for +`suppression_strengthen` and null for `family_revocation`; +`valid_until_unix_ms` is null only for a non-expiring transition. Enqueue time, +target key, family ID, and queue metadata are deliberately outside the payload +and therefore outside transition identity. `intent_id` is SHA-256 over the UTF-8 +bytes `tsq1|`, the fixed-width canonical target-key bytes, and RFC 8785 JSON of +that complete materialized payload, in that order with no added separator. +Cross-field validation is total: `family_revocation` requires target +`r` + this outbox's family ID, state `revoked`, null permission/floor, and one +of the three withdrawal causes with its matching source class; +`suppression_strengthen` requires target `s` + this family ID, state +`suppressed`, a vocabulary permission, a present floor, and a compatible +signal/malformed/absence cause. Any other target family, tag, nullability, +state/transition pair, or cause/source pair is malformed and denies the family. + +Known-answer vector representing a suppression after authorization revision 7: +target key `s` + 64 zeroes and payload +`{"cause":"gpc","clearing_floor_authorization_revision":7,"evidence_digest":"0000000000000000000000000000000000000000000000000000000000000000","evidence_time_unix_ms":1785369600000,"permission":"select-personalised-ads","source_class":"gpc","state":"suppressed","transition_kind":"suppression_strengthen","valid_until_unix_ms":null}` +produce +`cafa2fadf3625ab0e6a0108224e97955b8230dc42dd589948c6ac2caf7eabbb0`. +Enqueue merge, absorbing revocation, exact-revision acknowledgment, +retention, capacity overflow, and failure-domain behavior are normative in the +permission spec §4.3. Unknown fields/schema, unsorted/duplicate ids, a payload +whose recomputed id differs, or revision regression fail closed; none is +skipped as “best effort.” + +The replay cap and fail-restrictive behavior are product sign-off item 31; authority record level: family ID, CAS version counter, schema version, stub marker (backfill, §5); unknown-field and range validation apply like every class (strong class, permission-exempt writes per the permission spec's inventory; revisions are app-level counters because @@ -950,26 +1044,36 @@ readers round-trip unknown keys **semantically** (values preserved through read-modify-write; byte-identical output is not required and not achievable through a structured serializer). -| Field | Purpose | Source | Gating permission (egress) | TTL / refresh | Rewrite | On revocation | -| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | -| key (v1: identifier verbatim; v2: core-constructed, §4) | Row identity | Provider/core | — | Row TTL (1 y today) | New canonical row; old key becomes alias | Family record governs; member tombstone as cleanup | -| `v` | Schema discriminator | Core | — | — | Written at current version | Retained | -| `created` / **`expires_at`** | Row age and the **absolute retention deadline, pinned at mint** — every update writes with the _remaining_ lifetime, never a fresh full TTL (today's full-TTL rewrite lets a frequently visited identity live forever; refreshable derived state must never rejuvenate the identity) | Core | P1 (first-party ops) | Never extended | Preserved (no rejuvenation) | Retained in tombstone | -| `consent.tcf` / `consent.gpp` | Raw signal snapshot for audit; superseded as authority by provenance | Request | Never egressed to partners | Replaced on live resolution (§7 snapshot rule, permission spec) | Fresh live values | Scrubbed | -| `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | -| New: **immutable mint tag** (`mint_provider`, `mint_version`) | Credential retirement and audit — write-once at mint (legacy backfill may populate a missing tag once); **never part of the replaceable snapshot**, or a v1 identity revisited after rotation would be restamped v2 | Mint (or one-time backfill) | — | Immutable | — | Retained | -| New: **provenance revision** (application-level monotonic counter, u64, initialized at 1, incremented by every provenance-bearing write, CAS'd with the row generation, serialized as an integer; overflow is a hard error, not a wrap) | Orders clears vs. snapshots (permission spec §4.3) | Core | Read by S2S/clears | Monotonic | — | Retained | -| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision) | **Audit mirror only — never the S2S decision input**: the strong summary carries every decision field (jurisdiction included); the row supplies identity/partner data only after the exact revision fence | Live resolution only | Not read for gating — audit and cleanup only (S2S reads the strong summary alone) | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | -| New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | -| `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | -| `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | -| `pub_properties` (origin/seen domains) | Creation context | Core at mint | P1 | Write-once | Preserved | Scrubbed | -| `device.*` (JA4 class, H2 hash, quality metadata) | **Discontinued for new rows** (§5): fingerprint-derived, buyer-facing — beyond security-classification authorization. v1 rows retain them read-only; they are never egressed post-epic and are dropped at rewrite | Fastly device provider | None grants egress | Write-once (v1) | **Dropped** | Scrubbed | -| New: security classification outcome (boolean) | Bot-gate result | Device provider | — (never egressed) | Written at mint | Fresh | Scrubbed | -| `network.*` (immutable evidence: ASN etc.) | Cluster disambiguation | Platform at mint | P1 | Write-once | Fresh | Scrubbed | -| Derived cluster state (`cluster_size`, computed-at) | Trust gating | Computed | — | **Refreshable, short validity; generation-CAS update; never touches `expires_at`** | Recomputed | Scrubbed | -| `ids` (partner → UID map) | Partner identity graph | Pixel/pull/batch sync | P1 ∧ P4 (partner egress) | Per-mapping timestamps; bounded count/length | Copied **with original timestamps/expiry** | Scrubbed | -| New: alias record kind | Rewrite indirection (§6.1) | Core | — | Retirement deadline | Is the mechanism | Family-revoked like any member | +| Field | Purpose | Source | Gating permission (egress) | TTL / refresh | Rewrite | On revocation | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| key (v1: identifier verbatim; v2: core-constructed, §4) | Row identity | Provider/core | — | Row TTL (1 y today) | New canonical row; old key becomes alias | Family record governs; member tombstone as cleanup | +| `v` | Schema discriminator | Core | — | — | Written at current version | Retained | +| New: `graph_key_canonical_identifier` | Collision witness required **only** for `sha256-detect` graph-key mode. Exact JSON string containing the provider-canonical identifier's ASCII bytes (global ≤256-byte alphabet); forbidden in `injective` and legacy-HMAC rows. Before any row read, merge, CAS update, alias action, graph use, or partner use, core compares length and bytes against the request's canonical identifier using the shared constant-time comparator. Mismatch is the §2 collision outcome: fail closed, never overwrite/join, trip the fleet-visible security alert. | Provider canonicalization at mint | Never egressed; identity-bearing row field | Immutable for the complete row/tombstone lifetime | Preserved byte-for-byte; N+1 knows and enforces the field | Retained in tombstone; family revocation remains authoritative after row cleanup | +| `created` / **`expires_at`** | Row age and the **absolute retention deadline, pinned at mint** — every update writes with the _remaining_ lifetime, never a fresh full TTL (today's full-TTL rewrite lets a frequently visited identity live forever; refreshable derived state must never rejuvenate the identity) | Core | P1 (first-party ops) | Never extended | Preserved (no rejuvenation) | Retained in tombstone | +| `consent.tcf` / `consent.gpp` | **Legacy-only raw snapshots.** New rows do not store them; a live resolution stores normalized per-permission provenance and a domain-separated evidence digest. A separately approved audit store, if deployed, is encrypted, access-controlled, and retention-bounded outside the identity row. | Legacy request data | Never egressed from the row | Read-only until scrubbed; no new writes | **Dropped** | Scrubbed | +| `consent.ok` / `consent.updated` | v1 liveness flag — **superseded by the family revocation record**; written during member cleanup for v1-reader benefit through the transition | Core | — | — | Fresh | `ok = false` written as cleanup; the family record is authoritative (v1's 24 h tombstone TTL does not bound revocation) | +| New: **immutable mint tag** (`mint_provider`, `mint_version`) | Credential retirement and audit — write-once at mint (legacy backfill may populate a missing tag once); **never part of the replaceable snapshot**, or a v1 identity revisited after rotation would be restamped v2 | Mint (or one-time backfill) | — | Immutable | — | Retained | +| New: **provenance revision** (application-level monotonic counter, u64, initialized at 1, incremented by every provenance-bearing write, CAS'd with the row generation, serialized as an integer; overflow is a hard error, not a wrap) | Orders clears vs. snapshots (permission spec §4.3) | Core | Read by S2S/clears | Monotonic | — | Retained | +| New: per-permission provenance (grant basis, authoritative timestamp, `valid_until`, jurisdiction, policy revision) | **Audit mirror only — never the S2S decision input**: the strong summary carries every decision field (jurisdiction included); the row supplies identity/partner data only after the exact revision fence | Live resolution only | Not read for gating — audit and cleanup only (S2S reads the strong summary alone) | `valid_until` per evidence class; replaced atomically, never merged | **Fresh live resolution** — never copied | Scrubbed | +| New: `family_id` | Revocation discovery | Core at mint (derived for legacy, permission spec §4.3) | — | Immutable | Shared across linked rows | Is the revocation key | +| `geo.country` / `geo.region` | Jurisdiction snapshot | Geo provider at mint | P1 | Written at mint | Fresh | Scrubbed | +| `geo.asn` / `geo.dma` | Cluster disambiguation / market signal | Platform at mint | P1; DMA additionally P4 for bidstream use | Written at mint | Fresh | Scrubbed | +| `pub_properties` (origin/seen domains) | Creation context | Core at mint | P1 | Write-once | Preserved | Scrubbed | +| `device.*` (JA4 class, H2 hash, quality metadata) | **Discontinued for new rows** (§5): fingerprint-derived, buyer-facing — beyond security-classification authorization. v1 rows retain them read-only; they are never egressed post-epic and are dropped at rewrite | Fastly device provider | None grants egress | Write-once (v1) | **Dropped** | Scrubbed | +| New: security classification outcome (boolean) | Deferred with host fingerprinting; absent in this epic | — | — | Not written | — | — | +| `network.*` (immutable evidence: ASN etc.) | Cluster disambiguation | Platform at mint | P1 | Write-once | Fresh | Scrubbed | +| Derived cluster state (`cluster_size`, computed-at) | Trust gating | Computed | — | **Refreshable, short validity; generation-CAS update; never touches `expires_at`** | Recomputed | Scrubbed | +| `ids` (partner → UID map) | Partner identity graph | Pixel/pull/batch sync | P1 ∧ P4 (partner egress) | Per-mapping timestamps; bounded count/length | Copied **with original timestamps/expiry** | Scrubbed | +| New: alias record kind | Rewrite indirection (§6.1) | Core | — | Retirement deadline | Is the mechanism | Family-revoked like any member | + +For `sha256-detect`, create-if-absent stores +`graph_key_canonical_identifier` in the same atomic row value as `v`; a +pre-existing value must match before a generation CAS is even attempted. A +missing/invalid witness in that mode, or a witness present in `injective`/HMAC +mode, is malformed row state and fails closed. N+1's reader/preserver release +parses and enforces this field rather than relying on generic unknown-field +round-tripping, which is why an N+2-created collision witness remains effective +during pre-promotion rollback. ## 7. Composition root and adapter parity @@ -991,18 +1095,19 @@ Requirements: **per-record-class consistency requirements**, because "has KV" says nothing about whether revocation is observable: - | Record class | Required semantics | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | - | Family revocation records | **Globally observable strong consistency** — every instance's read observes a committed revocation, not merely the writing session's own writes (writer-scoped read-your-writes is insufficient for a fleet). Cloudflare Workers KV is **not eligible** — "60 seconds or more" is an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A **failed or erroring revocation-record read fails closed** for egress | - | Authority-state records (suppression + positive summary) | **Globally observable strong reads AND linearizable per-key CAS** — CAS alone orders writes, but a stale successful _read_ on another instance would authorize egress after a committed suppression ("read failures fail closed" does not cover stale successes); both properties are adapter eligibility gates | - | Identity-row mutation | **Generation CAS** (conditional write on row generation) with reread/recompute on conflict — rows are heavily mutable (snapshots replaced, partner IDs merged, derived state refreshed), so unordered last-writer-wins loses newer evidence and mappings; _visibility_ may stay eventual, unordered _mutation_ may not. Fastly KV offers generation-marker conditional writes; Workers KV's documented concurrent last-write-wins is ineligible for mutation-bearing rows | - | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | - | Deployment metadata — schema floor (write-once/CAS); **graphless flag additionally requires globally observable strong reads** (N+2 lease revalidation and N+1's `not_before` barrier both need globally current reads, not just write-once); **policy-activation register additionally requires linearizable CAS + strong reads and an ordered upstream `source_version`** (backend push version, or the `ts config push` envelope sequence); **graphless deadline checks additionally require one of: store-issued current time on a strong read, or a declared bounded fleet clock skew (S_fleet)** — each its own capability cell | **Write-once/CAS**, outside ordinary config storage (migration spec §4) | - | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | - | Rowless prefix-withdrawal (`w`) records | **Globally strong reads + linearizable CAS** (same bar as authority-state), plus the **bounded listing-visibility window** the backfill scan depends on — all three are eligibility gates for the graphless migration. The strong-read obligation is **permanent for HMAC row discovery** — enforcement runs to the last entry's `valid_until`, long after the flag clears — and retention runs through the **max(cookie, row, S2S) horizon of each entry** (an earlier "cookie lifetime" cell contradicted §6.3's per-entry horizons) | - | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | - | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | + | Record class | Required semantics | + | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | Replay reservations (client-cycle) | **Linearizable CAS with fencing** (ownership epoch). Eventually-consistent KV cannot provide this — on Cloudflare that means a Durable-Object-class primitive, not Workers KV; an adapter without it fails startup for client-cycle selection | + | Family revocation records | **Globally observable strong consistency** — every instance's read observes a committed revocation, not merely the writing session's own writes (writer-scoped read-your-writes is insufficient for a fleet). Cloudflare Workers KV is **not eligible** — "60 seconds or more" is an expectation, not a bound; on Cloudflare this record class needs a Durable-Object-class primitive. An adapter without an eligible primitive fails startup for identity features. A failed/erroring read fails closed; a successful absence has no positive-use lease, while a present revocation may be cached only to deny. | + | Authority-state records (suppression + positive summary) | **Globally observable strong reads AND linearizable per-key CAS** — CAS alone orders writes, but a stale successful _read_ on another instance would authorize egress after a committed suppression ("read failures fail closed" does not cover stale successes); both properties are adapter eligibility gates. Absence and positive authority receive no success lease. A typed restrictive result may be cached only to deny and cannot construct authority, clear state, acknowledge an intent, or drive a CAS. | + | Negative-intent outbox + global identity safety breaker | Per-family outbox records require globally strong reads + linearizable CAS in a failure domain independent of the authority/revocation target. The deployment breaker follows permission §4.3's qualified failure contract. Every identity decision performs fresh checks of both before positive operation; no absence/healthy success lease is allowed. A typed pending/tripped result may be cached only to deny. A deployment lacking the independent primitive or its failure-semantics proof cannot enable persisted identity use. | + | Identity-row mutation | **Generation CAS** (conditional write on row generation) with reread/recompute on conflict — rows are heavily mutable (snapshots replaced, partner IDs merged, derived state refreshed), so unordered last-writer-wins loses newer evidence and mappings; _visibility_ may stay eventual, unordered _mutation_ may not. Fastly KV offers generation-marker conditional writes; Workers KV's documented concurrent last-write-wins is ineligible for mutation-bearing rows | + | Row creation | **Atomic create-if-absent** (fresh mints), same primitive family | + | Deployment metadata — schema-floor mirror; graphless flag; policy/config/model candidate and activation register; config-sequence allocator; global identity safety breaker | **Globally strong reads + linearizable CAS outside ordinary config storage.** Activation additionally needs immutable version-addressed publication, authenticated authoritative fleet membership/readiness/quiescence, one deployment-qualified `serve_admission_lease_bound_ms`, one authenticated nondecreasing Unix-ms time domain shared by the activation register and immutable journal-object service, atomic local admission/refcount closure, fail-closed lease renewal/restart/suspend behavior, and an immutable time-retained activation journal. The common clock enforces `promotion_not_before_unix_ms`, journal creation at/after that gate, and the 60-second publication-age ceiling; incomparable clocks are ineligible. The bounded lease amortizes only whole-settings/model admission; it never leases privacy-state success. Graphless deadlines need store-issued time or a declared bounded fleet clock skew. The config-sequence allocator assigns ordered envelope versions, and the activation register binds complete blob/data/config/policy/model identities plus the journal head; ordinary config-store `put` and the `m00` compatibility mirror activate nothing. After model promotion the authenticated controller idempotently CAS-raises and read-verifies `m00`; a higher mirror fails closed for investigation. | + | Rewrite transactions | **Linearizable fenced CAS required** (same primitive class as reservations) | + | Rowless prefix-withdrawal (`w`) records | **Globally strong reads + linearizable CAS** (same bar as authority-state), plus the **bounded listing-visibility window** the backfill scan depends on — all three are eligibility gates for the graphless migration. Successful absence has no positive-use lease; a matching restrictive result may be cached only to deny. The strong-read obligation is **permanent for HMAC row discovery** — enforcement runs to the last entry's `valid_until`, long after the flag clears — and retention runs through the **max(cookie, row, S2S) horizon of each entry** (an earlier "cookie lifetime" cell contradicted §6.3's per-entry horizons) | + | Alias installs (reserved) | Row-store **per-key CAS with read-your-writes** — recorded for the future `rewrite_legacy` spec; nothing in the epic writes an alias | + | Identity rows | Eventual **visibility** acceptable _after_ a generation-CAS mutation commits (see Identity-row mutation above) — the earlier "rows are accretive" claim is deleted: rows replace snapshots, merge partner IDs, and refresh derived state, and unordered last-writer-wins loses newer evidence | Every record class additionally declares **durability and maximum retention**: a store passing the consistency check but capping TTLs @@ -1023,19 +1128,52 @@ Requirements: them is how a "yes" cell hides an unusable feature. Feature eligibility requires wired, not merely available: - | Capability | Fastly | Axum (dev) | Cloudflare | Spin | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | - | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | - | Prefix listing (cluster) | Used today, but **ratification requires a cited platform completeness bound, pagination behavior, and failure semantics** — the graphless scan's settle window cannot be derived from "works in practice" | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | - | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | - | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | - | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic degrades per the permission spec's declared rule, which directly changes state-level US outcomes | No | - | Device host evidence (JA4/H2) | Yes | No | No | No | - | Authority-state: global strong reads + CAS | Conditional writes available (generation marker); **globally current read semantics to verify** — both required, wiring to verify | **Unavailable** (no store wired) | Workers KV: **ineligible**; Durable Objects: feasible, not wired | **Unavailable** | - | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | - | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | - | Deployment metadata — floor (write-once/CAS), **graphless flag (globally strong reads + CAS + store-clock or S_fleet branch)**, and **policy-activation register (linearizable CAS + strong reads + ordered `source_version`)** | **Not wired** — needs a primitive distinct from the config store | **Unavailable** | DO: feasible, not wired | **Unavailable** | - | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | + | Capability | Fastly | Axum (dev) | Cloudflare | Spin | + | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | + | Graph persistence (eventual OK) | KV Store: available + wired | **Not wired** — the current adapter installs `UnavailableKvStore`; an in-process store is dev-feasible but does not exist yet | Workers KV: available + wired (eventually consistent) | Key-value: **available, not wired** — Spin config is embedded and EC KV routes are unwired today | + | Prefix listing (cluster) | Used today, but **ratification requires a cited platform completeness bound, pagination behavior, and failure semantics** — the graphless scan's settle window cannot be derived from "works in practice" | **Unavailable** (no store wired — in-process feasibility is a note, not a cell) | Yes (eventual) | _verify_ | + | Strongly consistent revocation reads | _verify_ against Fastly KV semantics | **Unavailable** (no store wired) | **Workers KV: no** — needs Durable Objects, not currently wired | _verify_ | + | Linearizable fenced CAS _(informative — deferred features)_ | **Not currently available** | **Unavailable** (no store wired) | Durable Objects: possible, not wired | **No** | + | Platform geo | Yes — country + region | No | **Yes — country only, no region** (`cf-ipcountry`): regionless US traffic uses the country-wide protective US floor | No | + | Device host evidence (JA4/H2) | Platform-available but **feature deferred and startup-ineligible in this epic** | No | No | No | + | Authority-state: global strong reads + CAS | Conditional writes available (generation marker); **globally current read semantics to verify** — both required, wiring to verify | **Unavailable** (no store wired) | Workers KV: **ineligible**; Durable Objects: feasible, not wired | **Unavailable** | + | Negative-intent outbox + global identity safety breaker | **Not wired**; requires a globally strong CAS failure domain independent of authority/revocation targets plus the §4.3 failure proof | **Unavailable** | A separate DO-class domain is feasible, not wired | **Unavailable** | + | Identity-row generation-CAS mutation | Same primitive as above | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Row create-if-absent | Generation-marker create: available, **wiring to verify** | **Unavailable** | Workers KV: **ineligible**; DO: feasible, not wired | **Unavailable** | + | Deployment metadata — `m00` schema-floor compatibility mirror (monotonic CAS; never activates writes; idempotent controller repair/read-verification) | **Not wired** — needs a primitive distinct from the config store and the controller completion/repair path | **Unavailable** | DO: feasible, not wired; repair path absent | **Unavailable** | + | Graphless flag (globally strong reads + CAS, plus the store-clock **or** S_fleet deadline branch) | **Not wired**; neither clock branch qualified (no store-time primitive verified, no declared fleet-skew bound) | **Unavailable** | DO: feasible, not wired; clock branch unqualified | **Unavailable** | + | Config-sequence allocator (linearizable CAS) | **Not wired** — ordinary config-store `put` is insufficient | **Unavailable** | DO: feasible, not wired | **Unavailable** | + | Serve-admission lease bound/timebase (positive `serve_admission_lease_bound_ms`, timer error/suspend proof, shared register/journal clock, non-early deadline) | **Unqualified** — no exact bound or timer/suspend proof declared and no shared register/journal time domain or non-early gate wired | **Unavailable** | DO-class coordination is feasible, but no exact bound, timer/suspend proof, shared time domain, or non-early gate is wired | **Unavailable** | + | Immutable config publication + settings/model prepare/activation register (strong reads + CAS + bounded admission lease + store-clock not-before + membership/readiness/quiescence + atomic whole-request gate/refcount + journal retention) | **Not wired**; version-addressed publication, authoritative settings/model readiness/quiescence, lease timebase/bound, store-enforced not-before, atomic admission/refcount, and journal lifecycle are unqualified | **Unavailable** | DO metadata is feasible; immutable publication, settings/model membership/readiness/quiescence, bounded lease/not-before, atomic admission/refcount, and journal lifecycle are not wired | **Unavailable** | + | Durability / max-retention proof | KV durable; TTL ceilings **to verify** against computed horizons | **Unavailable** | Workers KV TTLs: to verify; DO storage: feasible | **Unavailable** | + + Activation qualification fixtures cover: a validation read linearizing before + and after the drain CAS; a delayed successful response whose usable lease is + shortened or expired, including a delayed old-generation response after + reopen; a last-instant admission with a long-running request (time bound + passed but early promotion still rejected until quiescence); zero-bound + candidate rejection, qualified/candidate/readiness bound mismatch, and a + bound-change attempt while traffic remains eligible; slow/fast timer and + store-clock rate, forward clock steps, incomparable register/journal clocks, + an attempted time-domain change while traffic/candidate is live, restart, + suspend/resume, expiry-boundary, activation-watcher delay, and renewal + failure; member + partition/crash/replacement, membership restage, cancel-drain/retry, and stale + acknowledgment; and background origin/vendor egress, cache publication, and + identity mutation remaining inside the atomic quiescence refcount. The model + fixture proves no `pre_epic_v1` request overlaps a v2 write. Privacy-state + fixtures prove stale restrictive caches only deny and every positive path + still performs fresh revocation, authority, outbox, `w`, and breaker reads. + Mirror fixtures cover missing, lower, equal, higher, unreadable, CAS failure, + read-verification failure, and controller crash after the authoritative model + CAS but before `m00`; only missing/lower/equal are idempotently raised or + confirmed, while higher remains fail-closed. + + Whole-settings/model activation is a universal serve capability, including + for identity-free requests. A stateless-identity selection waives only the + identity persistence/strong-state cells; it never bypasses activation, + admission, journal, or quiescence qualification. An adapter that cannot + qualify the activation rows cannot serve under this spec. - Each adapter's runtime-services setup routes through the shared builders. - Providers are constructed **once** per application instance and stored in @@ -1067,11 +1205,12 @@ Two defaults chosen for neutrality change effective behavior on existing Fastly deployments; both are called out in the migration spec and must be prominent in release notes: -- **Bot gate.** The pre-provider EC bot gate required JA4 _and_ platform - class. With `device.provider = "builtin"` the gate degrades to User-Agent - heuristics. Restoring the stronger gate requires `[device] provider = -"fastly"`; the migration guide lists this as a behavior-preserving step for - Fastly deployments. +- **Bot gate.** The pre-provider EC bot gate required JA4 and platform + class. This epic intentionally falls back to the builtin User-Agent-only + classifier; `[device] provider = "fastly"` is startup-rejected until the + separate host-fingerprinting/security design is ratified. Release notes + call out the loss of the stronger gate rather than presenting selection + alone as authorization. - **Geo.** With no geo provider, jurisdiction resolution falls to the configured default country. The permission model spec (§5.3) constrains this combination so it cannot silently grant permissions to mis-attributed @@ -1107,7 +1246,8 @@ prominent in release notes: selection defaults to `platform` (today's always-on behavior); the selector exists, only its default is held back. 5. The permission model PR: flips the geo default to none **in the same - change** that introduces the `default_country` fallback and the §5.3 + change** that introduces acknowledged static-jurisdiction + `default_country` mode, the protective provider-failure profile, and the §5.3 acknowledgment guard, and adds the EC permission-enforcement point of §5; `required_permissions()` appears on the EC trait in this step, not before (per the §4 minimalism rule). diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 4c2fcb95f..6f63cd949 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -32,30 +32,30 @@ and whether that is a preservation or a declared change. **Silent changes are defects.** PR #838 changed six of these without declaring any; each was discoverable only because a deleted test had pinned the old behavior. -| # | Decision (today) | After epic | Status | -| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | -| 2 | US-state request, GPC/GPP/USP opt-out → no EC, existing EC expired + tombstoned, **even when a consenting TCF string is present** | Same (precedence §4 of permission spec) | Preserved — **must not regress** | -| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | -| 3a | US-state request, explicit GPP `sale_opt_out = false` or a US Privacy string that is present and not opting out (including "not applicable") → EC allowed | Same: these are grant-class signals satisfying `requires_signal` (permission spec §4) | Preserved — **must not regress** | -| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | -| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | -| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | -| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | All three fields enforced per the §4.5 mapping (targeted-advertising affects P4 only, never destructive) | **New enforcement, declared** — opt-out effects are more protective; the same fields' not-opted-out values can also **newly grant P4**, which is not protective — both directions are classified in permission spec §4.5 | -| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Same: policy enumerates `US/` rules for configured privacy states; country-level `US` resolves non-regulated (permission spec §3.4) | Preserved — **must not regress** (a country-wide `US = "us-opt-out"` rule would deny all of it) | -| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | -| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | -| 5 | No country resolvable (geo unavailable) → no EC (fail-closed) | `default_country` baseline, constrained by permission spec §5.3 so the fail-open combination cannot occur silently | Declared change, guarded | -| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | -| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | -| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Revokes and withdraws globally (permission spec §4 and §4.2 trigger 1) — including tombstoning, which is irreversible | Declared change, more protective, **irreversible** — see §6.4 | -| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Only with `[device] provider = "fastly"`; the `builtin` default is UA-only | Declared change with a documented restore step (§5) | -| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | -| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | -| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | -| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | -| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | -| 13 | Cookies minted by graphless deployments have no graph row | Rowless proof via strong-class records under the graphless-migration flag (N+2 convergence + attested stub-backfill first — providers spec §5); verified cookies expire and re-mint without continuity; **rowless withdrawal writes into the capped per-prefix `w` record, then expires** (exact-cookie family records are superseded); unverifiable roaming cookies get disclosed cookie-only expiry (sign-off 29) | **Declared** — pre-existing identities restart rather than carry over | +| # | Decision (today) | After epic | Status | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | EU/GDPR request, no TCF consent → no EC | Same (opt-in baseline) | Preserved | +| 2 | US-state request, GPC/GPP/USP sale opt-out → no EC, existing EC expired + tombstoned, even when a consenting TCF string is present | Opt-out denies P4 globally but does not delete P1 or the first-party identity. A separately valid P1 grant may retain/mint an identity that cannot enter partner egress; auction dispatch may continue only through permission §7's positive `ContextualAuctionView` | **Changed deliberately:** sale/sharing opt-out is separated from deletion/storage withdrawal | +| 3 | US-state request, no signals at all → no EC (fail-closed) | Preserved by the recipe: the US rule is `requires_signal`, and the extended grant-signal class (permission spec §4) is what makes that possible — `granted` would allow no-signal traffic; a TCF-only grant class could not grant from GPP/USP values | Preserved under the recipe | +| 3a | US-state request, explicit GPP/USP not-opted-out value (including today's N/A-as-allow behavior) → EC allowed | Explicit applicable not-opted-out may grant P4 only; it does not grant P1. N/A, absent, reserved, and unknown values grant nothing. P1 requires its own accepted evidence or baseline | **Changed deliberately:** N/A is not affirmative permission and sale state is not storage authority | +| 3b | US-state request, TCF record present and refusing Purpose 1 (no US opt-out signal) → no EC | Same: refusal beats coexisting non-TCF grant signals (permission spec §4, precedence 3–4) | Preserved | +| 3c | Consent-record conflict modes (restrictive/permissive/newest), expiry, KV fallback, proxy mode | Each row of the normalization matrix (permission spec §4.4) is individually marked preserved or changed there; changed rows: malformed-present now blocks acquisition | Per §4.4 matrix | +| 3d | Valid + expired consent records: conflict resolution runs first and can select the expired record | Expired sources drop **before** conflict resolution (permission spec §4.4 pipeline) | **Changed (declared)** | +| 3e | Only the GPP sale field (and USP) is consulted; `SharingOptOut` / `TargetedAdvertisingOptOut` are ignored | Sale, sharing, and targeted-advertising opt-outs deny P4; their explicit applicable not-opted-out values may grant P4 under a US opt-out regime. None affects P1 or destroys identity | **New enforcement, declared** | +| 3f | Non-privacy-state US traffic (e.g. Wyoming) is non-regulated → EC allowed | Country-level `US` is the protective `us-opt-out` floor; region-specific rules may be stricter. Regionless traffic never degrades to non-regulated | **Changed deliberately** to make country-only geo safe | +| 3g | Graph rows persist JA4 class, H2 fingerprint hash, and buyer-facing quality metadata | Discontinued for new rows; v1 values retained read-only, never egressed, dropped at rewrite (providers spec §6.3) | **Changed (declared)** — more protective | +| 4 | UK request, no TCF record → no EC | Same, unless the policy deliberately adopts a `granted` storage baseline for GB, with citation and sign-off | Declared change (if made) | +| 5 | No country resolvable (geo provider failure) → no EC (fail-closed) | Protective failure profile: both permissions require signal and dispatch uses GDPR-class handling; `default_country` is reserved for acknowledged static-jurisdiction mode | **Changed deliberately:** deny-all becomes signal-required/GDPR-class; absent or invalid grants remain denied, while valid accepted grants are newly possible (sign-off 18) | +| 6 | Non-regulated country, TCF record refusing Purpose 1 → EC still created, existing identity never tombstoned | Refusal now blocks _new_ grants everywhere (permission spec §4, precedence 3) — a declared, more-protective change. Existing identity is still **never tombstoned** where the baseline is `granted` (permission spec §4.2) | Split: creation is a declared change; no-tombstone is preserved | +| 7 | Country resolved but not in any regulation list ("non-regulated") → EC created, EIDs pass through | Governed by the policy's `rules.default` entry (permission spec §5.4). The §5 recipe sets it to a `granted` baseline to preserve today's behavior; the protective example policy instead requires a signal worldwide — a declared operator choice between the two | Preserved under the recipe; declared change under the protective default | +| 8 | Opt-out signal (GPC/GPP/USP) **outside** US states → ignored today | Its mapped use restriction is honored globally; sale/sharing/GPC deny P4 but do not tombstone P1 identity | Declared global privacy hardening without irreversible overreach | +| 9 | Fastly bot gate requires JA4 + platform class before KV-backed EC writes | Host fingerprinting is deferred and `[device] provider = "fastly"` startup-fails; builtin UA-only classification is the only shipped behavior | Declared loss of the stronger gate pending a separate security/fingerprinting design | +| 10 | Fastly always resolves geo per request | Only with `[geo] provider = "platform"`. The neutral geo default flips **only** in the permission model PR, together with the §5.3 guard — never in an intermediate step where absent geo would fail closed and zero EC issuance (providers spec §11) | Declared change, sequenced, with a documented restore step (§5) | +| 11a | Raw EC egress on paths gated by the jurisdiction gate today (OpenRTB `user.id`, derived request IDs, page bids, EIDs, identify, pull sync — pull checks the live `EcContext` today) | Gated by the egress inventory (permission spec §7): bidstream and partner egress require both purposes, revocation exempt — at least as strict as today for every path | Preserved (strengthened); **must not regress** — PR #838 gated only EIDs and left `user.id` reachable | +| 11b | Proxy / click / Testlight forwarding extract the raw EC cookie/header **without** today's jurisdiction gate | Gated by the egress inventory (both purposes) | **New privacy hardening, declared change** — not preservation | +| 11c | Batch sync today only authenticates the S2S caller and checks live/tombstoned row state — it is **not** jurisdiction-gated | Gated by stored-provenance recompute (permission spec §7); legacy rows fail closed until backfilled | **New privacy hardening, declared change** — not preservation | +| 12 | EC generation succeeds without a configured identity-graph store | A minting provider requires an openable graph store at startup (providers spec §5, §6); pre-N+1 readiness step provisions it | **Breaking, declared** — graphless deployments must provision storage before upgrading | +| 13 | Cookies minted by graphless deployments have no graph row | Rowless proof via strong-class records under the graphless-migration flag (N+2 convergence + attested stub-backfill first — providers spec §5); verified cookies expire and re-mint without continuity; **rowless destructive withdrawal writes into the capped per-prefix `w` record, then expires** (exact-cookie family records are superseded); non-destructive signals are request-local for the old rowless identity, while any newly minted row-backed family commits the current suppression before use; unverifiable roaming cookies get disclosed cookie-only expiry (sign-off 29) | **Declared** — pre-existing identities restart rather than carry over | Rows 3, 4, and 7 are policy decisions, not code decisions: they belong in the `[permissions]` policy review, made explicitly by maintainers — not @@ -121,46 +121,56 @@ Requirements: convergence gate, then `ts config push`. A config mixing old and new fields (`[ec] passphrase` alongside `[ec] provider`) is **rejected** by N+1, not reconciled. **N+1 is a full semantic reader and - enforcer for every N+2 record kind — not a field preserver.** + fail-restrictive enforcer for every N+2 negative record kind — not a + field preserver — but it does not originate the new use-suppression + model.** Preserving unknown JSON does not chase aliases, consult family revocations, honor suppression records, or fail closed on provenance; an N+1 that merely preserved would, after rollback, treat revoked identities as live (aliases are reserved-future with the rewrite deferral, providers spec §6.1). N+1 must also **write family revocation records** — a withdrawal arriving on a - rolled-back N+1 fleet must still revoke. These negative gates are - **config-shape- and graphless-flag-invariant** (providers spec §5 - total state table): no config shape, flag state, or v1-semantics - row disables reading and enforcing family revocations, suppression - entries, or live `w` records. **Authority-state: N+1 may create - stubs and write negative entries, but never positive commits or - clears.** The observed-row admission sequences (providers spec §5) - need an s-class stub before revoking an untouched v1 row and a - suppression CAS for a non-destructive signal — both are - **negative/stub writes N+1 is permitted**, so a first post-upgrade - GPC or SharingOptOut executes fully on N+1 (revocation, or a - persisted suppression, not a deny-and-forget). What N+1 must **not** - do is commit or clear **positive** authority (that needs the - `AuthorityRefresh` revision protocol a v1 writer cannot produce): - suppression created under N+2 stays in force through a rollback and - its **clearing waits for roll-forward** — a protective, declared - limitation. This is one contract, resolving the earlier - "neither creates nor clears" wording that made first-upgrade - withdrawal unexecutable. + rolled-back N+1 fleet must still revoke. Existing negative gates are + **config-shape- and graphless-flag-invariant** (providers spec §5 total + state table): no config shape, flag state, or v1-semantics row disables + reading and enforcing an already committed family revocation, + suppression entry, pending outbox intent, or live `w` record. + + N+1's write boundary is narrower. It may create the minimum s-class stub + needed to admit an observed untouched v1 row and may write a family + revocation or its durable outbox intent for an explicit storage + withdrawal/authenticated deletion that the pre-epic lifecycle already + recognizes. It **does not create or strengthen a use-suppression entry** + from GPC, SharingOptOut, SaleOptOut, TargetedAdvertisingOptOut, malformed + input, or absence. A signal already recognized by the pre-epic gate keeps + only that request-local behavior; a newly mapped signal is telemetry on + N+1. None creates durable suppression. Persistence begins only on N+2 after + active new-shape configuration **and** the fleet-wide `permissions_v2` + model promotion. N+1 also never commits positive authority and + never clears N+2 state because it cannot produce the ordered + `AuthorityRefresh` revision. A suppression created by N+2 therefore stays + fail-restrictive through binary rollback and clearing waits for + roll-forward. This asymmetry preserves rollback safety without describing + a first-observation P4 suppression as "pre-epic behavior." **N+1's identity-write behavior is v1, explicitly** — this resolves what was an impossible trilemma (write rows without provenance, violating active-after-commit; write provenance, violating the N+2-only writer boundary; or stop minting, an undeclared outage): - N+1 **keeps minting v1 rows with today's semantics**, and the new - active-after-commit/provenance contract activates **with the N+2 - writer**, not before. Likewise the permission model itself: + N+1 **keeps minting v1 rows with today's semantics**. Merely starting an + N+2 binary does not activate the new contract: both N+1 and N+2 read the + permission spec §5.5 `model_epoch` from the strong activation register, + and N+2 must emulate N+1 while it is `pre_epic_v1`. The + active-after-commit/provenance contract activates only through the + fleet-wide `permissions_v2` model promotion, not from binary version. + Likewise the permission model itself: **old-shape config on N+1 runs the pre-epic consent gate unchanged** — dual-read means dual-behavior — so the compiled protective fallback cannot flip behavior mid-convergence before the operator pushes the new-shape policy; the new model's **live gating** - engages only with the N+2 writer _and_ new-shape config together — - on N+1, new-shape config engages validation, telemetry, and the + engages only with active new-shape config **and** the promoted + `permissions_v2` epoch together — before promotion, new-shape config on + either binary engages validation, telemetry, and the batch fail-closed boundary below, never live-request gating. The interim is declared as sign-off item 20 — with one boundary that does **not** wait for N+2: once new-shape config is active, **context-free partner egress (batch sync) on N+1 fails @@ -168,32 +178,32 @@ Requirements: spec's legacy rule requires. Otherwise N+1 would mint a P1-only v1 row under the new model and then release it through today's row-state-only batch check — the fail-closed rule cannot activate - later than the model it protects. Live-request paths keep v1 - semantics until N+2. + later than the model it protects. Live-request paths keep v1 semantics + until model promotion. The interim is **one matrix, not competing prose** — per release × config shape, each dimension separately (a new-shape P1 denial on N+1 has exactly one meaning: telemetry, never gating): - | Dimension | N+1, old shape | N+1, new shape | N+2, new shape | - | ----------------------------------------------------------- | ---------------------- | --------------------------------------------------------- | --------------- | - | Policy resolution | not parsed | parsed, validated, logged — **never gates live requests** | gates | - | Live-request permission gating | pre-epic gate | pre-epic gate (a new-shape denial is telemetry only) | new model | - | Negative gates (revocation, suppression, `w`), read + write | **active** (invariant) | **active** (invariant) | active | - | Identity-row writes | v1 rows | v1 rows | v2 + provenance | - | Positive authority commits / clears | forbidden | forbidden | N+2 writer | - | Context-free batch (S2S) egress | pre-epic row check | **fails closed for provenance-less rows** | full recompute | + | Dimension | N+1, old shape | N+1 or N+2, new shape + `pre_epic_v1` | N+2, new shape + `permissions_v2` | + | ---------------------------------------------------------- | ------------------------------- | --------------------------------------------------------- | --------------------------------- | + | Policy resolution | not parsed | parsed, validated, logged — **never gates live requests** | gates | + | Live-request permission gating | pre-epic gate | pre-epic gate (a new-shape denial is telemetry only) | new model | + | Existing negative gates (`r`, `s`, `q`, `w`), read/enforce | **active** (rollback invariant) | **active** (rollback invariant) | active | + | Explicit withdrawal/deletion family revocation + outbox | active | active | active | + | Fresh use-suppression creation/strengthening | **forbidden** | **forbidden** | active | + | Identity-row writes | v1 rows | v1 rows | v2 + provenance | + | Positive authority commits / clears | forbidden | forbidden | N+2 writer | + | Context-free batch (S2S) egress | pre-epic row check | **fails closed for provenance-less rows** | full recompute | Rollback tests therefore mirror the one contract exactly: - family-revocation read **and write**; authority-state **stub - creation and negative suppression entries, read and write** (the - earlier "read-and-fail-closed only / N+1 writes none" test text - contradicted the required observed-row sequences — a rolled-back - N+1 receiving SharingOptOut must persist the suppression, not deny - once and forget); **positive-authority commits and clears asserted - forbidden**; and v1-minting behavior — all on N+1 against - N+2-written data. **Rollback is binaries-first too, in the other direction** — - N+2 → N+1 binaries roll back keeping the new config (N+1 reads it + family-revocation read and explicit-withdrawal write; minimum stub + creation; existing suppression/outbox read and enforcement; fresh GPC, + sale, sharing, and targeted-advertising opt-outs asserted to create no + `s`/`q` record on N+1; **positive-authority commits and every clear + asserted forbidden**; and v1-minting behavior — all on N+1 against + N+2-written data. **Before model promotion, rollback is binaries-first too, + in the other direction** — N+2 → N+1 binaries roll back keeping the new config (N+1 reads it fully; reverting config first would hand the old shape to N+2 binaries that reject it) — **with one structural rule that makes it possible at all**: providers are compiled into the composition root — there is no @@ -202,7 +212,11 @@ Requirements: must ship compiled-in (dormant: registered, parseable, configurable, not selectable as writer) in R−1**; adopting a genuinely new provider gets its own reader-first rollout, exactly - like the epic itself. With that rule, **schema rollback and provider rollback are + like the epic itself. After the `permissions_v2` promotion, the active + minimum binary generation and row schema floor bar N+1 from startup and + per-request serve admission; rollback below N+2 then requires a separately + designed forward-compatible recovery release, never an N+1 binary. With + that rule, **schema rollback and provider rollback are distinct sequences**: schema rollback is binaries-first (above); **provider rollback is config-first** — a fleet whose config _selects_ the new provider as writer cannot roll binaries first, @@ -231,14 +245,17 @@ Requirements: before at least one adapter qualifies risks an epic with no selectable identity provider, so Fastly qualification (or an explicit decision to proceed without it) gates ratification — - together with **filling the PSL snapshot reference** - (`psl-snapshot-ref.md` is a placeholder; ratification cannot - reproduce the cookie-domain computation it approves until the - vendored commit is recorded), **filling the GPP registry snapshot** - (`gpp-registry-snapshot.md` equally lacks its immutable registry - commit and per-section conformance vectors; ratifying §4.5 field - mappings that cannot be reproduced against a pinned registry is the - same defect), and creating the §8 decision records. + together with **vendoring the pinned PSL artifact** + (`psl-snapshot-ref.md` now records the upstream commit, but the required + list bytes and checked hash are not yet present; the vendoring manifest/PR + must record the upstream commit tree, source blob OID, vendored SHA-256, and + byte-for-byte verification), **completing the pinned + GPP corpus and decoder** (`gpp-registry-snapshot.md` now records the + immutable official commit and accepted versions, but the per-section + conformance corpus and complete decoder are still prerequisites; its + manifest/PR must likewise record the commit tree and blob OID/SHA-256 for + each of the four state specifications and `Section Information`), and + creating the §8 decision records. 3. **Revocation-eligible storage is a per-adapter gate, and ungated adapters migrate stateless.** Identity features require the adapter's strong-consistency rows in the capability matrix (providers spec §7) @@ -288,18 +305,32 @@ Requirements: read-modify-write (values round-trip; byte-identical JSON is neither required nor achievable through a structured serializer — and a genuinely pre-N+1 worker cannot preserve at all, which is exactly why - the floor exists); after the **fleet-convergence gate**, **N+2 - activates the writer** and begins emitting the new fields. **The rollback floor is crossed at N+2 writer activation itself** — an - observable deploy event, recorded in the **deployment-metadata - primitive** (providers spec §7 capability row; the existing - config-store interface exposes ordinary put/delete and cannot express - a monotonic floor) with a specified protocol, not an assertion: the - marker lives in a dedicated namespace outside rollbackable config; - the **first N+2 instance to activate creates/advances it via - create-or-CAS** (the creation race resolves to one winner), **reads - it back, and only then enables new-format writes**; every binary - reads the floor at startup and a binary below the floor **fails - startup**; an unreadable floor fails closed (writer stays disabled). + the floor exists). After authoritative fleet convergence on N+2, the + controller runs permission §5.5's model prepare/commit: every + traffic-eligible snapshot member proves N+2 generation and the bound active + new-shape tuple, then the fleet stops admission and proves no pre-epic + request remains in flight before one CAS changes `model_epoch`, minimum + binary generation, and row schema floor together. **The rollback floor is crossed + by that fleet-wide CAS**, not by the first N+2 process to start. N+2 keeps + N+1 writer/gating behavior before the CAS; afterward every N+1 request fails + serve admission and only then may N+2 emit new fields. + + The authoritative floor lives in deployment metadata outside rollbackable + config and every request reads it through the activation fence. The legacy + `m00` key is a monotonic startup mirror updated after promotion, never an + activation source; a lower or unreadable mirror fails startup but a higher + mirror cannot enable writes without the authoritative active tuple. + **Mirror completion and repair are an owned runbook step:** after the model + CAS, the authenticated deployment controller strong-reads active and + `m00`; a missing or lower mirror is CAS-set to exactly + `active.row_schema_floor`, equality is an idempotent no-op, and an + unreadable mirror or failed CAS/read-verification remains closed for retry. + The controller then strong-reads and verifies exact equality before + declaring the cutover complete. A crash between model CAS and mirror write + reruns the same idempotent operation; it never lowers `m00` or changes + active. A higher mirror is rejected before any write rather than repaired + by lowering it — startup remains closed while operators investigate the + register/journal inconsistency. Floor-in-rollbackable-config would let "restore the previous config version" erase the marker after new-format rows exist — exactly the state it guards — and "any new-format row exists" is a fact no @@ -311,11 +342,16 @@ Requirements: permission spec §7) — and, critically, **withdrawal never depends on backfill**: the family ID for an untouched v1 row is derived deterministically (permission spec §4.3), so a first-post-upgrade - GPC request withdraws correctly with zero migrated state. Mixed-version tests with stated expected results: + explicit storage-withdrawal request revokes correctly with zero migrated + state. Once the N+2 writer and new-shape policy are active, a GPC request + the active N+2 writer instead persists a P4 use suppression without deleting the identity; N+1 + only applies its pre-epic request-local result as specified above. + Mixed-version tests with stated expected results: N+1-reader/old-row → full function; old-reader/new-row → v1 semantics, new fields untouched if read-only, preserved semantically if read-modify-write on N+1, **test-proven lost on pre-N+1** (documenting why the floor is a floor); N+2-reader/N+1-written-row → full function. + 6. **Half-migrated fails loud.** A `[ec.providers.hmac]` block with no `provider = "hmac"` selector is a startup error (providers spec §6). In PR #838 this configuration — the exact state an operator following the @@ -333,10 +369,12 @@ Requirements: documents the switch sequence and the retirement/cleanup step that ends it. 9. **The example config ships the migrated happy path**, uncommented: - `provider = "hmac"` with its block, `[geo] default_country`, and (for - Fastly) the behavior-preserving `[device] provider = "fastly"` and - `[geo] provider = "platform"` lines present with a comment stating what - removing them changes. PR #838's example shipped the passphrase block + `provider = "hmac"` with its block and `[geo] provider = "platform"` + where the adapter qualifies. Host fingerprinting is not a migration + compatibility option: `[device] provider = "fastly"` is rejected until + a separate security/fingerprinting design is approved. Static-geo + examples use `default_country` only together with + `assume_single_jurisdiction = true`. PR #838's example shipped the passphrase block uncommented with the selector commented out — steering operators directly into the silent-stateless state. 10. Every misconfiguration in the providers spec §6 table fails at @@ -356,11 +394,11 @@ Requirements: "Keep exactly today's behavior" is not fully achievable, and the recipe's name says so. The unavoidable divergences, enumerated (each also a matrix -row): global opt-out honoring (row 8); refusal blocking new grants -everywhere (row 6); newly enforced GPP sharing/targeted fields, which can -also **grant** P4 where nothing granted before (row 3e); the FR -unresolved-geo fallback, where valid TCF consent can grant while today's -unresolved-geo path always denies (row 5); malformed-present blocking +row): global P4 opt-out honoring (row 8); refusal blocking new grants +everywhere (row 6); newly enforced GPP sharing/targeted fields, where only +an explicit applicable non-opt-out can grant P4 (row 3e); the protective +geo-failure profile (row 5); country-wide protective US handling for +country-only and regionless results (row 3f); malformed-present blocking acquisition (§4.4); proxy-mode opt-out extraction; and the batch-sync provenance gate (row 11c). Everything else the recipe preserves. @@ -374,12 +412,11 @@ HMAC requirement contradicts that: The recipe is a **complete, valid TOML fixture per adapter, committed to the repository** (e.g. `docs/guide/fixtures/migration-preserving-fastly.toml` and siblings) and included in the guide verbatim — never described as a -textual delta against the example file. Per-adapter because a single -fixture cannot be: `[device] provider = "fastly"` is Fastly-only, and +textual delta against the example file. Per-adapter because `[geo] provider = "platform"` varies by host — Cloudflare **does** support platform geo but resolves **country only, no region** (per the -providers spec adapter matrix), which changes state-level US privacy -outcomes and engages the declared regionless degradation; Axum and Spin +providers spec adapter matrix), which engages the country-wide protective +US rule rather than degrading to non-regulated; Axum and Spin have no platform geo and reject the selection (providers spec §6). Each adapter's fixture carries the selections valid for it, and each is CI-validated against its adapter. (An earlier draft said "copy the example table, @@ -392,17 +429,20 @@ fixture contains, in one document: identity-graph store configuration** — selecting a minting provider without an openable graph store is a startup error (providers spec §6), so a fixture omitting it would not start; -- `[device] provider = "fastly"` (Fastly deployments: preserves the JA4 - bot gate); -- `[geo] provider = "platform"` and `default_country = "FR"` (per-request - jurisdiction detection preserved; the FR default is a **protective - opt-in fallback**, not fail-closed — valid TCF consent still grants, - where today's unresolved-geo path always denies); +- no host-fingerprinting provider selection; the shipped builtin + classification is UA-only, and selecting `[device] provider = "fastly"` + fails startup pending a separate approved design; +- `[geo] provider = "platform"` where supported. A selected provider's + lookup failure uses the compiled-in protective failure profile and never + `default_country`; a static deployment instead sets `default_country` + together with `assume_single_jurisdiction = true` and receives a + separately validated fixture; - (Fastly fixture; other adapters substitute their valid selections) the full `gdpr-eu` / `gdpr-uk` / `us-opt-out` groups and country rules from the example policy (US as `requires_signal` with the grant-signal - class — §2 rows 3–3b), plus the `non-regulated` group with - `rules.default = "non-regulated"` (row 7). Operators who prefer the + class — §2 rows 3–3b), a country-wide `US = "us-opt-out"` floor, plus + the `non-regulated` group with `rules.default = "non-regulated"` (row + 7). Operators who prefer the protective worldwide default use the example file itself instead. A partial policy is a trap the first draft of this spec fell into: a @@ -427,7 +467,57 @@ global honoring of opt-out signals is unconditional. providers first with the geo default held at today's behavior, the permission model PR flipping it together with its guard); each PR is reviewable against §2 in isolation and states which rows it touches. -2. Before/after deploy, operators watch **EC issuance rate** and EID +2. **Adapter qualification is a release gate.** Every adapter that serves any + request, including identity-free traffic, first qualifies immutable + publication, config-sequence allocation, the bounded admission-lease + timebase, shared register/journal clock, store-enforced + promotion-not-before, fleet readiness/quiescence, and the atomic local + admission gate. An adapter missing any of those universal activation cells + cannot serve under this spec; stateless identity does not bypass them. + Stateful/context-free identity use additionally stays disabled until every + required providers-spec §7 identity cell is backed by a platform artifact + and fault/concurrency tests: global strong reads plus CAS, row + generation-CAS, negative outbox and safety breaker, and retention ceilings. + An adapter that qualifies universal activation but not those identity-state + cells takes the declared stateless-identity path. The response hook likewise remains startup-disabled where its §3 + artifact/IR, atomic-commit, Vary-rekey, secret, or header-ceiling cells are + pending. +3. **Every policy publication uses staged activation.** `ts config push` + allocates a never-reused `push_sequence`, writes and verifies an immutable + version-addressed envelope, and CAS-installs the complete blob/data/config/ + policy tuple as candidate. Every authoritative fleet member acknowledges + that exact tuple and the deployment-qualified + `serve_admission_lease_bound_ms`. The draining CAS sets a trusted-store + `promotion_not_before_unix_ms`; an old-generation validation may admit only + until its hard bound, after which an all-request admission stop and + authenticated quiescence barrier proves no request or background effect + remains on the displaced tuple. Time expiry never substitutes for member + quiescence; only the controller's CAS + promotion makes the entire settings snapshot active. Instances serve the + prior active snapshot while readiness is incomplete, stop during the commit + drain, and reopen only after verifying and leasing new active; staged or stale revisions + may not perform destructive work. Rollback republishes old content under a new sequence and + follows the same prepare/commit path. Settings promotion preserves the + active model fields. The later N+2 writer cutover is a distinct unanimous + model prepare/commit on the same register: all traffic-eligible members + prove N+2 readiness against the bound active tuple, wait out the same + admission-lease bound, drain all pre-epic + requests and attest quiescence, then one CAS advances model epoch, minimum + binary generation, and row schema floor together. The controller then runs + §4 requirement 5's idempotent `m00` raise/read-verify step; cutover + completion and below-floor startup remain closed until it succeeds. + + **Operational consequence:** every settings promotion — including an + ordinary configuration-only push — deliberately creates a scheduled + fleet-wide deployment-unavailable interval while admission is closed, + displaced-generation work quiesces, the promotion CAS completes, and + members load the new tuple. This is not a zero-downtime configuration + protocol. A controller failure can extend the outage until authenticated + cancellation or successful promotion; scoping the stop to selected settings + or providing blue/green overlap requires a separate effect-classification + design. + +4. Before/after deploy, operators watch **EC issuance rate** and EID attachment rate; the migration guide names these as the canary metrics, because the failure mode of a bad migration is a silent drop to zero (or a silent grant to everyone), not an error rate. The full metric set, each @@ -443,18 +533,18 @@ global honoring of opt-out signals is unconditional. threshold, an evaluation window, and a named action** (pause rollout / roll back / block retirement) in the migration guide — a metric with a "healthy range" but no action is dashboard decoration; the two already - specified (legacy-reader quiet period) are the - pattern the rest follow. -3. Startup logs always print: selected provider per concern, whether geo is + specified (legacy-reader quiet period) are the pattern the rest follow. +5. Startup logs always print: selected provider per concern, whether geo is live, the effective default baseline, and the count of granted-without- signal permissions. One line, greppable, stable format. -4. **The batch-sync coverage dip is a gated rollout stage, not a +6. **The batch-sync coverage dip is a gated rollout stage, not a notification.** Provenance-coverage thresholds are normative gate criteria: the guide defines a target coverage level and evaluation window; recovery stalling below threshold for the window triggers the **pause action** — investigate backfill (traffic mix, dormant rows), never disable the gate; and staging is explicit: provenance - **writing** begins the moment N+2 activates, enforcement is already + **writing** begins only when the `permissions_v2` model epoch activates, + enforcement is already in force (there is no fail-open stage), so the only stageable knob is partner communication and the cleanup cadence for rows that never recover. @@ -466,22 +556,24 @@ global honoring of opt-out signals is unconditional. of the transient rejection rate. There is no fail-open shortcut — the alternative (grandfathering pre-epic identities past the permission model) is rejected in the permission spec. -5. Rollback is config-only where possible: reverting to the previous - config version restores the previous behavior on the previous binary. The - irreversible artifacts are enumerated — not "one": **family - revocation records and member tombstones** (no recovery; that is - their purpose), the **schema-floor marker** (write-once by design; - no administrative clear), and — corrected from the former "sticky - timestamp-less suppression" entry, which the permission spec's - TTL-sticky rule supersedes — nothing suppression-shaped: - timestamp-less opt-out suppression is **TTL-bounded and goes inert - automatically** (administrative clear is an optional early exit, not - a requirement, and the guide's cleanup and expiry tests follow the - TTL rule). The irreversibility of revocation is also why the - withdrawal triggers (permission spec §4.2) are exhaustive, why partial - withdrawal failure has an explicit tombstones-first, browser-retries - contract (permission spec §4.3), and why §2 rows 6 and 8 call out - tombstoning explicitly. Two operational procedures are documented in the +7. Rollback is config-only where possible: reverting to the previous + config version restores the previous behavior on the previous binary. + The durable artifacts are enumerated — not "one": **family revocation + records and member tombstones** created by explicit storage withdrawal + or authenticated deletion (no recovery; that is their purpose), the + **authoritative `permissions_v2` model epoch/minimum-generation/schema-floor + tuple** (monotonic by design; no administrative clear) and its `m00` + compatibility mirror, + persistent per-family use-opt-out suppressions, and any pending negative + intent in the durable outbox. A use-opt-out suppression does not expire + merely because a consent TTL elapses; only strictly newer explicit + authorization for that same use, or identity deletion, clears it. An + outbox entry remains until the target negative write is confirmed, and + the global identity safety breaker remains closed until the outbox is healthy + and drained. The irreversibility of identity revocation is why the + destructive triggers (permission spec §4.2) are exhaustive and why + partial withdrawal failure has a family-record-first, cleanup-retries + contract (permission spec §4.3). Two operational procedures are documented in the guide, not automated: cleanup of identities minted before a policy tightening (permission spec §4.2 trigger 3), and retirement of a legacy reader after a provider switch (providers spec §6.1), which is the @@ -503,46 +595,49 @@ global honoring of opt-out signals is unconditional. ## 8. Product decisions requiring explicit sign-off -These are decisions this spec set makes that #838 had not already made (or -made differently). **Implementation is blocked while any row is `open`**; -each row is a **decision, not an assignment**: the table tracks the -decision and its record; _who_ decided is captured inside the record +These are product decisions this spec set needs that #838 had not already +made (or made differently). The table records the recommended resolution +approved for this spec revision, not a final product decision. +**Implementation is blocked while any row is `open`**; each row is a +**decision, not an assignment**: _who_ decided is captured inside the record itself (`docs/superpowers/specs/decisions/NN-title.md` — the decision, the deciders, the date). The Decision-record column holds the link (`—` while open); an unratified row reverts to open, not to silently implemented. -| # | Decision | Where | Decision record | Status | -| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------- | --------------------------------------------------------------------------------------- | -| 1 | Opt-outs honored globally; destructive ones irreversibly withdraw outside the defining jurisdiction | permission §4, §4.2 | — | open | -| 2 | Sale opt-outs (GPP, USP) control both P1 and P4 and destroy the identity | permission §4.5 | — | open | -| 3 | Sharing / targeted-advertising fields: opt-outs remove P4 and retain the stored identity; the same fields' not-opted-out values **newly grant P4** | permission §4.5 | — | open | -| 4 | US contextual auctions continue during opt-out, identity removed | permission §7 | — | open | -| 5 | Regionless US traffic treated as non-regulated unless the operator opts into country-wide gating | permission §3.4 | — | open | -| 6 | Full consent strings continue downstream; raw consent snapshots retained in rows for audit | providers §6.3 | — | open | -| 7 | Legacy batch-sync traffic rejected until live-browser provenance backfill | this spec §6.4; permission §7 | — | open | -| 8 | Proxy / click / Testlight forwarding newly gated by P1 ∧ P4 | §2 row 11b | — | open | -| 9 | Integration cookie operations — deferred out of the v1 hook with the full read/use/withdraw model as entry bar | hook §3 | — | open (descope ratification still required; record-less ⇒ open per the decisions README) | -| 10 | Session-cookie exemption question | hook §3 | — | open (deferred, but record-less ⇒ open per the decisions README) | -| 11 | A single failed destructive-revocation **or suppression** write may leave S2S identity use live **indefinitely** for a never-returning visitor (no durable external retry queue) | permission §4.3 | — | open | -| 12 | Adapters without revocation-eligible storage migrate **stateless** rather than blocking the release | §4.2 of this spec | — | open | -| 13 | Batch-sync acceptance dropping toward zero at cutover, with dormant identities having no automatic recovery path | §6.4 of this spec | — | open | -| 14 | Policy tightening never reuses stored refusals destructively — a fresh, live post-change refusal is required (the spec decides this; ratify it) | permission §4.2 trigger 2 | — | open | -| 15 | **Epic descope**: client-cycle spec demoted to deferred-informative; `rewrite_legacy` cut to a recorded deferral; hook ships headers-only | client spec status; providers §6.1; hook §3 | — | open | -| 16 | Timestamp-less opt-out suppression is **TTL-sticky**: within its consent-TTL lifetime only a newer timestamped grant clears it; at `valid_until` it goes inert automatically (not user-sticky-forever, not administratively sticky — administrative clear is an optional early exit) — **with the declared saturation exception**: an opt-out arriving as a restrictive overflow during a saturation epoch inherits the epoch marker's earlier `valid_until` and may get less than a full lifetime (providers wire schema; ratified here and in row 31) | permission §4.3 | — | open | -| 17 | Explicit N/A values are grant-class and can newly authorize personalized advertising | permission §4.5 | — | open | -| 18 | Permissive `default_country` remains in effect during prolonged geo-provider failure (metered residual) | permission §5.2 | — | open | -| 19 | Mixed policy revisions during rollout can produce irreversible destructive outcomes on part of the fleet | permission §5.5 | — | open | -| 20 | N+1 interim: v1 minting semantics and pre-epic gating persist under old-shape config until N+2/new-shape | migration §4.4 | — | open | -| 21 | Rowless legacy cookies are expired and re-minted **without continuity** (prefix-only verification cannot authenticate the suffix; adoption would let suffix variants mint unbounded rows) | providers §5 | — | open | -| 22 | Device fingerprint (JA4/H2) processing authorized by operator selection, with the boolean classification persisted — collection purpose, retention, downstream visibility, and the vocabulary-extension boundary | providers §5 | — | open | -| 23 | **Open question, not ratified**: may DataDome's security identifier (tag injection, `datadome` cookie, `X-DataDome-ClientID` read and vendor egress) operate outside the permission model? The decision must enumerate exactly which consumers may observe the cookie/ClientID — the enumerated observers are the security channel **and the publisher origin, which receives `X-DataDome-ClientID` via the upstream overlay** — "owner-scoped overlay" names the mechanism, this row names the recipient — plus retention, whether TS withdrawal expires it, **the browser-side observers the vendor's design implies — every same-origin page script (the cookie must not be HttpOnly per vendor guidance), vendor challenge pages executing with publisher-origin access, and (if header mode is ever opted into) the JavaScript/local-storage observer** — and challenge redirect targets | hook §4a; permission §7 | — | open | -| 24 | Malformed/absence-caused suppression clears on any newer valid grant (non-sticky; opt-out stickiness applies only to opt-out causes) — and an active suppression **overrides a `granted` baseline**: one malformed request denies later no-signal requests until valid evidence clears it, and a policy-baseline grant alone never clears | permission §4.3, §4.1 | — | open | -| 25 | Batch-sync stored-jurisdiction maximum age (consent-TTL horizon): a mover into GDPR stops old-rule egress at the horizon; a mover out is denied until a live visit | permission §7 | — | open | -| 26 | Embedded GPP GPC maps to the destructive global opt-out (header-OR-embedded aggregation) | permission §4.5 | — | open | -| 27 | Proxy-mode minimal opt-out extraction (decode only §4.5-mapped opt-out fields; no grants) | permission §4.4 | — | open | -| 28 | DataDome integration is deliberately reduced relative to vendor defaults (spec-pinned pointer allowlist starting at ClientID-only; hardened cookie attributes) — requires product **and vendor** acceptance — including CSP interaction with vendor challenge pages, same-origin vendor code on the publisher origin (or an origin-isolation/sandboxing requirement), the **specific `X-DD-B` divergence** (DataDome's documented cookie-mode allow example directs `Set-Cookie` **and** `X-DD-B` to the client, while TS drops `X-DD-B` per the allowlist matrix — vendor acceptance must name that exact field), and the fail-open consequence of batch invalidation | hook §4a; `datadome-header-allowlist.md` | — | open | -| 29 | Unverifiable roaming rowless cookies receive best-effort cookie-only expiry (admission rules forbid durable records for unverified values); a lost response can leave the cookie usable on the old network until re-presented | providers §5 | — | open | -| 30 | Prefix-wide rowless saturation: the per-prefix cap (8) and its escalation treat every rowless cookie behind one IP-derived prefix as withdrawn — NAT cohorts can be affected by one actor; cap value, threat assumptions, expected cohort size, reset/retention, observability, **and the saturation collateral: under a saturated prefix, any real row (listed or overflow) is denied and revoked immediately on surfacing — a non-abuser NAT-cohort row can be revoked; `w` is retained through the max(cookie, row, S2S) horizon and consulted by `valid_until`, not the flag, so this is deterministic, not a retention accident** — all in scope | providers §5 | — | open | -| 31 | Replay-history capacity (16 per-source semantic-state slots + a saturation epoch whose restrictive marker is pinned at the **first** restrictive overflow with its own full TTL): while saturated, fresh consent cannot grant until the epoch expires, and **later restrictive overflows inherit the first marker — a shortening of their lifetime, down to nearly zero near the marker's expiry (the marker outlives its epoch and can span epoch boundaries)**; ratification chooses this knowingly — the alternatives (per-overflow state; refreshing the marker on later overflows) were rejected for unbounded storage and replay-extension respectively | permission §4.3; providers wire schema | — | open | -| 32 | GPP sections 24–27 (MD/IN/KY/RI) are reserved with **national-section-only** handling until an official binary layout can be vendored — a state-specific opt-out expressed only in an undecodable state section is not honored | permission §4.5; `gpp-registry-snapshot.md` | — | open | +| # | Recommended resolution | Where | Decision record | Status | +| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | --------------- | ------ | +| 1 | Honor mapped use opt-outs globally. Destructive identity effects are limited to explicit storage withdrawal, authenticated deletion, or a qualifying live TCF Purpose 1 refusal. | permission §4, §4.2 | — | open | +| 2 | GPP/USP sale opt-outs suppress P4 only; they neither revoke P1 nor delete the identity. | permission §4.5 | — | open | +| 3 | Sharing/targeted-advertising opt-outs suppress P4; an explicit applicable not-opted-out value may grant P4. Neither affects P1. | permission §4.5 | — | open | +| 4 | US auction dispatch may continue while P4 is unset only through permission §7's positive `ContextualAuctionView` and its sole normative `contextual-openrtb-v1-allowlist.json` manifest. Unknown, unlisted, ill-typed, or untraceable leaves cause no dispatch; there is no client IP/UA, precise geo, page/referrer URL, user identifiers/data/segments, arbitrary extensions, or client forwarding headers. A separately authorized P1 identity may remain stored but cannot enter auction or partner egress. A destination that cannot consume the exact projection receives no request. | permission §7 | — | open | +| 5 | Country-only and regionless US traffic use a protective country-wide `us-opt-out` floor; state rules may be stricter. | permission §3.4 | — | open | +| 6 | Raw regulatory strings reach only the positively registered OpenRTB field that requires each source; all other destinations default deny. Identity rows retain normalized provenance/digests, not raw consent snapshots. | permission §7; providers §6.3 | — | open | +| 7 | Reject legacy batch-sync traffic until live-browser provenance backfill makes the row re-evaluable. | rollout §6 item 6; permission §7 | — | open | +| 8 | Gate proxy, click, and Testlight identity forwarding on P1 ∧ P4. | §2 row 11b | — | open | +| 9 | Defer integration-owned cookie operations from the v1 response hook; require a complete read/use/withdraw lifecycle before admission. | hook §3 | — | open | +| 10 | Do not create a blanket session-cookie exemption; every cookie must be covered by an approved permission or narrowly defined security-use authority. | hook §3 | — | open | +| 11 | Require a durable per-family negative-intent outbox in a failure domain independent of its strong target and checked freshly by every identity consumer. If neither target nor outbox can commit, close a globally visible breaker over all positive identity operations until audited recovery; adapters must prove the §4.3 failure contract rather than merely expose three CAS keys. This rejects the prior alternative of accepting an indefinite S2S/use residual when a negative target write fails and the visitor never returns. | permission §4.3 | — | open | +| 12 | Adapters that cannot meet the revocation-storage contract migrate stateless rather than weakening the contract. | rollout §6 item 2; recipe §5 | — | open | +| 13 | Keep batch sync fail-closed at cutover; stage partner communication and cleanup using explicit coverage thresholds, windows, and pause actions. | rollout §6 item 6 | — | open | +| 14 | Policy tightening does not reinterpret historical refusal as a destructive event; destructive withdrawal requires fresh, live qualifying evidence. | permission §4.2 trigger 2 | — | open | +| 15 | Descope the client cycle and `rewrite_legacy`; ship the v1 integration hook as headers-only. | client spec status; providers §6.1; hook §3 | — | open | +| 16 | Persist use-opt-out suppression until ordered explicit authorization for that use or identity deletion. TCF `LastUpdated` or an authenticated monotonic authorization revision proves order; bare timestamp-less GPP/USP does not. A currently presented identical timestamp-less opt-out starts a new restrictive episode after a clear without refreshing its original age. TTL and saturation never shorten it. This rejects the prior TTL-sticky alternative under which suppression became inert at consent-TTL expiry, as well as administrative clear without newer authorization and saturation-based shortening. | permission §4.3 | — | open | +| 17 | N/A, absent, reserved, unknown, and unsupported values never grant processing. | permission §4.5 | — | open | +| 18 | A selected geo provider's lookup failure uses the compiled-in protective profile; `default_country` is only for acknowledged static-jurisdiction mode. | permission §5.2 | — | open | +| 19 | Use immutable version-addressed whole-config publication plus prepare/commit activation of the complete blob/data/config/policy tuple with authenticated authoritative fleet membership/readiness, a deployment-qualified bounded admission lease, a shared authenticated register/journal time domain, store-enforced promotion-not-before, an all-request quiescence barrier, and a time-retained immutable activation journal. Use a second unanimous, lease-drained and quiescent model transition on the same register to advance model epoch, minimum binary generation, and row schema floor atomically. Every ordinary settings promotion intentionally causes a scheduled fleet-wide deployment-unavailable interval; this is not a zero-downtime protocol, and controller failure may extend the outage until authenticated cancellation or promotion. A mutable “latest” blob never activates settings or the writer; membership changes restage the candidate; no request admitted under a displaced logical `activation_generation` remains able to produce effects after either promotion. The activation fence is universal, including stateless-identity and identity-free traffic; an adapter that cannot qualify it cannot serve under this spec. The lease amortizes only whole-settings admission: positive authority, revocation, outbox, `w`, and breaker decisions retain fresh strong reads. | permission §5.5; CLI §5 | — | open | +| 20 | N+1 keeps v1 minting and pre-epic live gating. It reads/enforces N+2 negative state for rollback safety and can persist an explicit pre-epic withdrawal/deletion, but it does not originate durable P4 use suppression. New-shape settings alone do not activate the new writer/model: N+2 emulates N+1 until the fleet-wide `permissions_v2` model CAS, after which the register's minimum binary generation bars N+1 from serving. The N+1 batch boundary already fails closed as soon as new-shape settings are active. | migration §4.4 | — | open | +| 21 | Expire and re-mint rowless legacy cookies without continuity; a prefix match cannot authenticate the cookie suffix. Non-destructive signals are request-local and create no negative record for the old rowless identity; if ordinary P1-gated re-mint succeeds, the new row-backed family commits the current suppression before use. | providers §5 | — | open | +| 22 | Defer host JA4/H2 fingerprint processing to a separate approved design; reject `[device] provider = "fastly"` at startup and do not persist fingerprint-derived classifications. | providers §5 | — | open | +| 23 | Permit a narrow `SecurityUse` authority for DataDome only: exact request-scoped security fields may reach the fixed HTTPS Protection API host/path, with redirects disabled, but no TS-controlled ad identity, graph, partner egress, persistence, or ordinary logs. `Request` is path-only and `Referer` origin-only; the remaining publisher path is an explicit vendor retention/DSR disclosure, not described as identity-free. Publisher-origin ClientID exposure is disabled by default; the cookie has one immutable configured domain/path scope so deletion is total for TS-created cookies, and its lost-response residual is explicit. | hook §4a; permission §7 | — | open | +| 24 | Malformed/absence suppression overrides a permissive baseline but clears on newer valid evidence; it is not sticky like an explicit use opt-out. | permission §4.3, §4.1 | — | open | +| 25 | Enforce a stored-jurisdiction/provenance horizon for batch sync: moves into stricter regimes fail closed by the horizon, and moves out require a live visit. | permission §7 | — | open | +| 26 | Aggregate embedded GPP GPC with `Sec-GPC` by OR as a global, non-destructive P4 use opt-out. | permission §4.5 | — | open | +| 27 | In proxy mode, decode only mapped opt-out fields and derive no grants. | permission §4.4 | — | open | +| 28 | Require product and written vendor conformance approval for the reduced DataDome surface: fixed `api-fastly.datadome.co/validate-request` egress, path-only Request/origin-only Referer, trusted connection IP/port with raw forwarding headers omitted, exact repeated-header normalization and request-field byte limits (including omitted `CookiesList`, `TlsCipher`, and `H2Fingerprint`, and opt-in JA4), a 24,576-byte encoded request ceiling, bounded fixed-scope cookie lifecycle, CSP/challenge behavior, hardened headers, reserved security budget, and security-owned replace-all forwarding of documented `X-DD-B` exactly once. | hook §4a; `datadome-header-allowlist.md` | — | open | +| 29 | Accept rowless roaming-cookie expiry as a bounded residual only with telemetry, an explicit maximum lifetime, operator documentation, and a removal/sunset criterion. | providers §5 | — | open | +| 30 | Saturation blocks rowless admission for that prefix but never revokes an authenticated real row without its exact suffix; monitor NAT-cohort pressure. | providers §5 | — | open | +| 31 | Keep replay history bounded by evicting expired/grant entries first and retaining restrictive state for its full horizon; saturation never shortens a later opt-out. | permission §4.3; providers wire schema | — | open | +| 32 | Accept official GPP sections 24–27 version 1, pin their layouts to the vendored IAB commit, and treat complete decoder/fixture support as a release prerequisite. The vendoring evidence records the commit tree and per-source blob OID/SHA-256 for all four state layouts plus `Section Information`; a commit string alone does not close the gate. | permission §4.5; `gpp-registry-snapshot.md` | — | open | +| 33 | Treat any malformed or unsupported-version **mapped** GPP section as a global blocker for grants to the permissions its schema maps (P4 in v1), while still honoring decodable opt-outs elsewhere and never deriving withdrawal from malformed bytes; unknown unmapped section IDs remain non-contributing. | permission §4.5 | — | open | +| 34 | Permit providers whose canonical identifiers cannot fit an injective 123-byte graph suffix to use the providers §2/§6.3 `sha256-detect` mode: 256-bit domain-separated collision resistance plus stored canonical-identifier comparison, fail-closed collision handling, no overwrite/join, and no cluster capability unless a literal prefix is independently preserved. | providers §2, §6.3 | — | open | diff --git a/docs/superpowers/specs/activation-journal-vectors.json b/docs/superpowers/specs/activation-journal-vectors.json new file mode 100644 index 000000000..cb1df334d --- /dev/null +++ b/docs/superpowers/specs/activation-journal-vectors.json @@ -0,0 +1,98 @@ +{ + "schema_version": 1, + "canonicalization": "RFC 8785 (JCS)", + "digest": "SHA-256", + "domain_prefix_utf8": "tsactj1|", + "numeric_profile": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "vectors": [ + { + "name": "genesis-config-promotion", + "journal": { + "schema_version": 1, + "attempt_id": "00000000000000000000000000000000", + "candidate_incarnation": "11111111111111111111111111111111", + "previous_journal_id": null, + "pruned_through_journal_id": null, + "expected_activation_generation": 0, + "drain_attempt": 1, + "serve_admission_lease_bound_ms": 1000, + "promotion_not_before_unix_ms": 1700000001000, + "transition_kind": "config", + "displaced_active": { + "logical_root": "builtin", + "immutable_blob_id": "builtin", + "source_version": 0, + "data_hash": "0000000000000000000000000000000000000000000000000000000000000000", + "config_revision": "0000000000000000000000000000000000000000000000000000000000000000", + "policy_digest": "0000000000000000000000000000000000000000000000000000000000000000", + "ordinal": 0, + "model_epoch": "pre_epic_v1", + "minimum_binary_generation": 1, + "row_schema_floor": 1, + "activation_generation": 0 + }, + "activated_active": { + "logical_root": "app_config", + "immutable_blob_id": "app_config/1", + "source_version": 1, + "data_hash": "1111111111111111111111111111111111111111111111111111111111111111", + "config_revision": "2222222222222222222222222222222222222222222222222222222222222222", + "policy_digest": "3333333333333333333333333333333333333333333333333333333333333333", + "ordinal": 1, + "model_epoch": "pre_epic_v1", + "minimum_binary_generation": 1, + "row_schema_floor": 1, + "activation_generation": 1 + }, + "membership_epoch": 7, + "ready_members": ["edge-a", "edge-b"], + "quiesced_members": ["edge-a", "edge-b"], + "controller_id": "deploy-controller", + "retain_for_ms": 2592000000 + }, + "canonical_json_utf8": "{\"activated_active\":{\"activation_generation\":1,\"config_revision\":\"2222222222222222222222222222222222222222222222222222222222222222\",\"data_hash\":\"1111111111111111111111111111111111111111111111111111111111111111\",\"immutable_blob_id\":\"app_config/1\",\"logical_root\":\"app_config\",\"minimum_binary_generation\":1,\"model_epoch\":\"pre_epic_v1\",\"ordinal\":1,\"policy_digest\":\"3333333333333333333333333333333333333333333333333333333333333333\",\"row_schema_floor\":1,\"source_version\":1},\"attempt_id\":\"00000000000000000000000000000000\",\"candidate_incarnation\":\"11111111111111111111111111111111\",\"controller_id\":\"deploy-controller\",\"displaced_active\":{\"activation_generation\":0,\"config_revision\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"data_hash\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"immutable_blob_id\":\"builtin\",\"logical_root\":\"builtin\",\"minimum_binary_generation\":1,\"model_epoch\":\"pre_epic_v1\",\"ordinal\":0,\"policy_digest\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"row_schema_floor\":1,\"source_version\":0},\"drain_attempt\":1,\"expected_activation_generation\":0,\"membership_epoch\":7,\"previous_journal_id\":null,\"promotion_not_before_unix_ms\":1700000001000,\"pruned_through_journal_id\":null,\"quiesced_members\":[\"edge-a\",\"edge-b\"],\"ready_members\":[\"edge-a\",\"edge-b\"],\"retain_for_ms\":2592000000,\"schema_version\":1,\"serve_admission_lease_bound_ms\":1000,\"transition_kind\":\"config\"}", + "sha256_hex": "7af3934b4e5500903ef77ad8a1367db83b03fc62a0bb83bd0849289c685d2e88" + } + ], + "rejection_vectors": [ + { + "name": "unsafe-top-level-u64", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/expected_activation_generation", + "raw_json_number": "9007199254740992", + "error": "integer exceeds portable JCS profile" + }, + { + "name": "unsafe-embedded-active-u64", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/activated_active/source_version", + "raw_json_number": "9007199254740992", + "error": "integer exceeds portable JCS profile" + }, + { + "name": "fractional-journal-number", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/retain_for_ms", + "raw_json_number": "2592000000.5", + "error": "journal number is not an integer" + }, + { + "name": "unsafe-admission-lease-bound", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/serve_admission_lease_bound_ms", + "raw_json_number": "9007199254740992", + "error": "integer exceeds portable JCS profile" + }, + { + "name": "zero-promotion-admission-lease-bound", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/serve_admission_lease_bound_ms", + "raw_json_number": "0", + "error": "config/model admission lease bound is not positive" + } + ] +} diff --git a/docs/superpowers/specs/contextual-openrtb-v1-allowlist.json b/docs/superpowers/specs/contextual-openrtb-v1-allowlist.json new file mode 100644 index 000000000..2707ff96a --- /dev/null +++ b/docs/superpowers/specs/contextual-openrtb-v1-allowlist.json @@ -0,0 +1,668 @@ +{ + "schema_version": 1, + "openrtb_version": "2.6", + "path_grammar": "dot-separated exact JSON member names; [] denotes each array element", + "default": "deny", + "unknown_or_unlisted_behavior": "serialization_error_no_dispatch", + "container_rules": { + "implicit_parents_only": true, + "omit_empty_optional_arrays": true, + "minimum_imp_elements": 1, + "site_app": "exactly_one", + "supported_imp_media": ["banner", "video"], + "each_imp_media": "exactly_one_of_banner_video", + "unsupported_imp_media_behavior": "serialization_error_no_dispatch" + }, + "cardinalities": { + "required_single": "present exactly once in its object", + "optional_single": "absent or present exactly once in its object", + "required_array": "present non-empty scalar array; every element matches the rule", + "optional_array": "absent or present non-empty scalar array; every element matches the rule", + "required_array_member": "present exactly once in every nearest enclosing object-array element", + "optional_array_member": "absent or present exactly once in every nearest enclosing object-array element" + }, + "cross_field_rules": { + "all_or_none": [ + ["regs.ext.gpp", "regs.ext.gpp_sid[]"], + ["imp[].banner.w", "imp[].banner.h"] + ], + "required_nonempty_object_arrays": [ + { + "when_parent_present": "source.ext.schain", + "array_path": "source.ext.schain.nodes[]" + } + ], + "at_least_one_complete_group": [ + { + "when_parent_present": "imp[].banner", + "groups": [ + ["imp[].banner.w", "imp[].banner.h"], + ["imp[].banner.format[]"] + ] + } + ] + }, + "derivations": { + "fresh_transaction": "fresh CSPRNG request value, never derived from request/user/security state", + "inventory": "validated publisher inventory configuration only", + "request_coarse": "typed coarse request value named by the rule", + "privacy": "permission resolver or admitted raw regulatory transport only", + "constant": "literal value named by the rule" + }, + "rules": [ + { + "path": "id", + "type": "string", + "cardinality": "required_single", + "derivation": "fresh_transaction" + }, + { + "path": "at", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "tmax", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "test", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "allimps", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "cur[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "bcat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "badv[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "wseat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + + { + "path": "source.fd", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "source.tid", + "type": "string", + "cardinality": "optional_single", + "derivation": "fresh_transaction" + }, + { + "path": "source.pchain", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "source.ext.schain.complete", + "type": "integer", + "cardinality": "required_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "source.ext.schain.ver", + "type": "string", + "cardinality": "required_single", + "derivation": "inventory" + }, + { + "path": "source.ext.schain.nodes[].asi", + "type": "string", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "source.ext.schain.nodes[].sid", + "type": "string", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "source.ext.schain.nodes[].hp", + "type": "integer", + "cardinality": "required_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "source.ext.schain.nodes[].rid", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "fresh_transaction" + }, + { + "path": "source.ext.schain.nodes[].name", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "source.ext.schain.nodes[].domain", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + + { + "path": "imp[].id", + "type": "string", + "cardinality": "required_array_member", + "derivation": "fresh_transaction" + }, + { + "path": "imp[].tagid", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].instl", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "imp[].bidfloor", + "type": "number", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].bidfloorcur", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].secure", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "imp[].exp", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + + { + "path": "imp[].banner.w", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].banner.h", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].banner.format[].w", + "type": "integer", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].banner.format[].h", + "type": "integer", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].banner.pos", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].banner.topframe", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "imp[].banner.btype[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].banner.battr[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].banner.mimes[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].banner.api[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + + { + "path": "imp[].video.mimes[]", + "type": "string", + "cardinality": "required_array", + "derivation": "inventory" + }, + { + "path": "imp[].video.minduration", + "type": "integer", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.maxduration", + "type": "integer", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.protocols[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].video.w", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.h", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.startdelay", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.placement", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.plcmt", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.linearity", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.skip", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "imp[].video.playbackmethod[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].video.api[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].video.battr[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].video.pos", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + + { + "path": "imp[].pmp.private_auction", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "imp[].pmp.deals[].id", + "type": "string", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].pmp.deals[].bidfloor", + "type": "number", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].pmp.deals[].bidfloorcur", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].pmp.deals[].at", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].pmp.deals[].wseat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].pmp.deals[].wadomain[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + + { + "path": "site.id", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.name", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.domain", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.cat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "site.sectioncat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "site.pagecat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "site.mobile", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "site.privacypolicy", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "site.publisher.id", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.publisher.name", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.publisher.domain", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.content.cat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "site.content.language", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + + { + "path": "app.id", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.name", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.bundle", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.domain", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.ver", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.paid", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "app.cat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "app.sectioncat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "app.pagecat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "app.privacypolicy", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "app.publisher.id", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.publisher.name", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.publisher.domain", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.content.cat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "app.content.language", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + + { + "path": "device.devicetype", + "type": "integer", + "cardinality": "optional_single", + "derivation": "request_coarse" + }, + { + "path": "device.os", + "type": "string", + "cardinality": "optional_single", + "derivation": "request_coarse" + }, + { + "path": "device.language", + "type": "string", + "cardinality": "optional_single", + "derivation": "request_coarse" + }, + { + "path": "device.lmt", + "type": "integer", + "cardinality": "required_single", + "derivation": "constant", + "constant": 1 + }, + { + "path": "device.geo.country", + "type": "string", + "cardinality": "optional_single", + "derivation": "request_coarse" + }, + + { + "path": "regs.coppa", + "type": "integer", + "cardinality": "optional_single", + "derivation": "privacy", + "allowed": [0, 1] + }, + { + "path": "regs.ext.gdpr", + "type": "integer", + "cardinality": "optional_single", + "derivation": "privacy", + "allowed": [0, 1] + }, + { + "path": "regs.ext.us_privacy", + "type": "string", + "cardinality": "optional_single", + "derivation": "privacy" + }, + { + "path": "regs.ext.gpp", + "type": "string", + "cardinality": "optional_single", + "derivation": "privacy" + }, + { + "path": "regs.ext.gpp_sid[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "privacy" + }, + { + "path": "user.ext.consent", + "type": "string", + "cardinality": "optional_single", + "derivation": "privacy" + } + ] +} diff --git a/docs/superpowers/specs/datadome-header-allowlist.md b/docs/superpowers/specs/datadome-header-allowlist.md index 698553e1f..a6c5bf5c6 100644 --- a/docs/superpowers/specs/datadome-header-allowlist.md +++ b/docs/superpowers/specs/datadome-header-allowlist.md @@ -1,9 +1,206 @@ -# DataDome header allowlist (normative, checked-in — request and response directions) +# DataDome field allowlists (normative, checked-in — vendor request and response directions) Adding or changing any name here is a reviewed commit to this file and -a spec change. This file holds the **only** normative pointer lists; -the hook spec §4a references it and carries no duplicate or -per-decision lists of its own. +a spec change. This file holds the **only** normative Protection API +request-field and response-pointer lists; the hook spec §4a references it +and carries no duplicate or per-decision lists of its own. + +## Protection API request fields (browser request → DataDome only) + +`SecurityUse` admits only the fields below to the configured DataDome +Protection API endpoint. They are request-scoped and are never persisted in +the identity graph, copied to publisher upstream or another integration, or +logged as raw values. + +The endpoint is the fixed core-owned +`https://api-fastly.datadome.co/validate-request`; “configured endpoint” in +this file means that DataDome protection is enabled, not that an operator may +supply an authority. Redirect following is disabled. No TS-controlled +advertising identifier, consent-store key, graph value, request query, or full +referrer is admitted. The normalized publisher URL path remains disclosed and +may itself contain publisher-chosen data; sign-offs 23/28 must classify that +surface, its retention, and DSR handling rather than calling the entire URL +identity-free. + +Core-derived fields: + +- `Key`, `IP`, `Method`, `Protocol`, `Host`, `ServerHostname`, `Request` +- `RequestModuleName`, `ModuleVersion`, `TimeRequest`, `Port` +- `ServerName`, `ServerRegion` +- `ClientID` from the single unambiguous `datadome` cookie only. The form key + is always present because the Protection API declares it mandatory; its + value is the empty string when no unambiguous cookie exists +- `CookiesLen`, `AuthorizationLen`, and `PostParamLen` as lengths only +- `HeadersList`, containing only the source header names admitted by the + next list plus `authorization`, `content-length`, and `cookie` (whose values + remain length-only/ClientID-only above). Names are lowercased, comma-separated, + and retain received field-line order, including repeated admitted names; + arbitrary/custom header names are excluded. An adapter that cannot preserve + received header order does not qualify this integration until DataDome + approves a canonical replacement order in sign-off 28 + +Core derives those fields identically on every qualified adapter: + +- `Key` is the resolved DataDome server secret and is never obtained from + request/config text; `IP` and `Port` are the remote address and TCP source + port from trusted connection metadata. Missing `Key`, `IP`, or `Port` skips + the call through the metered fail-open path; no sentinel is synthesized +- `Method` is the validated HTTP method token; `Protocol` is exactly `http` or + `https` from the adapter request URI; `Host` is the normalized ASCII request + authority with a non-default port retained; `ServerHostname` is trusted TLS + SNI/local-host metadata, omitted when unavailable +- `Request` is only the URL path. Empty path becomes `/`; dot segments are + removed, percent escapes are preserved without percent-decoding and + normalized to uppercase hex, and the complete query and fragment are + discarded before the security view exists +- `RequestModuleName` is the literal `trusted-server`; `ModuleVersion` is the + build's checked-in Trusted Server version; `TimeRequest` is the request-ingress + Unix timestamp in decimal microseconds, captured once before integration + processing and constrained to `0..=2^53-1` +- `ServerName` is the adapter-qualified deployment/service name and + `ServerRegion` is its adapter-qualified region code; either is omitted when + the platform cannot supply it without request input +- `CookiesLen`, `AuthorizationLen`, and `PostParamLen` are decimal byte counts + of the received field/body surfaces before redaction. Overflow beyond an + unsigned 64-bit count skips the call; it never wraps or truncates + +Exact request-header value mappings: + +- `Accept` ← `accept`; `AcceptCharset` ← `accept-charset`; + `AcceptEncoding` ← `accept-encoding`; `AcceptLanguage` ← `accept-language` +- `CacheControl` ← `cache-control`; `Connection` ← `connection`; + `ContentType` ← `content-type`; `From` ← `from`; `Origin` ← a successfully + parsed `origin` serialized as scheme + ASCII host + non-default port only; + `Pragma` ← `pragma`; `Referer` ← a successfully parsed `referer` reduced to + scheme + ASCII host + non-default port only; `UserAgent` ← `user-agent`; + `Via` ← `via` +- `SecCHDeviceMemory` ← `sec-ch-device-memory`; `SecCHUA` ← `sec-ch-ua`; + `SecCHUAArch` ← `sec-ch-ua-arch`; `SecCHUAFullVersionList` ← + `sec-ch-ua-full-version-list`; `SecCHUAMobile` ← `sec-ch-ua-mobile`; + `SecCHUAModel` ← `sec-ch-ua-model`; `SecCHUAPlatform` ← + `sec-ch-ua-platform` +- `SecFetchDest` ← `sec-fetch-dest`; `SecFetchMode` ← `sec-fetch-mode`; + `SecFetchSite` ← `sec-fetch-site`; `SecFetchStorageAccess` ← + `sec-fetch-storage-access`; `SecFetchUser` ← `sec-fetch-user` +- `X-Requested-With` ← `x-requested-with` + +Request-field multiplicity is normalized **before** parsing, truncation, and +form encoding, and adapters expose every received field line rather than a +preselected first/last value. Every admitted value line must contain valid HTTP +field-value octets **and** valid UTF-8 after OWS removal; otherwise the vendor +call is skipped through the metered fail-open path, because adapter-specific +byte-to-string replacement is forbidden: + +- The list-valued source fields are exactly `accept`, `accept-charset`, + `accept-encoding`, `accept-language`, `cache-control`, `connection`, + `pragma`, `via`, `sec-ch-ua`, and `sec-ch-ua-full-version-list`. Core removes + leading and trailing optional whitespace from each field value, rejects a + value containing invalid field-value octets, and combines all field lines + (including empty values) in received order with the two literal bytes `, `. + This one normalized value is then parsed where the mapping above requires + parsing and is then bounded. Commas inside an individual value are not split + and reserialized. +- Every other admitted value-bearing source header in the exact mapping above + is singleton. Zero lines means omit the DataDome field. Exactly one valid + line is OWS-normalized and processed. Two or more lines — even identical — + are ambiguous and skip the vendor call through the metered fail-open path; + core never chooses first, last, or comma-joined. In particular this applies + to `origin`, `referer`, `user-agent`, `content-type`, `from`, every remaining + `sec-ch-*`/`sec-fetch-*` field, and `x-requested-with`. +- `authorization` and `content-length` are security singletons for this view. + Repetition skips the vendor call before either length or `HeadersList` is + constructed. `AuthorizationLen` is the byte length of the one + OWS-normalized value. `PostParamLen` is always the byte length of the body + actually presented to core, not the numeric `content-length` value; a + malformed or body-inconsistent `content-length` is rejected by the shared + HTTP request boundary before integrations run. +- Multiple `cookie` field lines are permitted. Core OWS-normalizes them and + joins them in received order with the literal bytes `; ` for the shared RFC + cookie parser. `CookiesLen` is the byte length of that canonical joined + value. `ClientID` is populated only when the parsed result contains exactly + one syntactically valid `datadome` pair; malformed cookie syntax or duplicate + `datadome` pairs produces the required empty `ClientID` value without + exposing another cookie. The original cookie values never enter the vendor + payload. +- After successful normalization, `HeadersList` records the lowercased name of + every admitted received field line in original line order, so repeated list + fields and cookie lines remain repeated. A rejected request produces no + `HeadersList` and no vendor call. Per-field caps apply to the single + normalized value; the 24,576-byte cap applies after complete form encoding. + +For an optional mapped value, zero received lines omits both source and mapped +field; one or more lines whose OWS-normalized values are all empty omits the +mapped form field but retains each received source name in `HeadersList`. If at +least one list-valued line is nonempty, empty siblings remain represented in +the exact received-order `, ` join. Mandatory `ClientID` and the three length +fields follow their explicit rules instead of this optional-field omission. + +Adapter qualification fixtures feed the same ordered repeated-field corpus to +every host and assert byte-identical form fields, lengths, `HeadersList`, and +reject/omit outcomes. The corpus includes repeated list fields, identical and +different singleton duplicates, multiple cookies, duplicate `datadome` +cookies, empty values, invalid octets, and headers whose individual values +contain commas; invalid UTF-8 is a skip, never replacement decoding. + +`true-client-ip`, `x-forwarded-for`, and `x-real-ip` are not admitted in v1. +The trusted `IP` field already supplies connection provenance; copying raw +forwarding headers would let a client or unqualified proxy manufacture vendor +evidence. A future adapter-normalized forwarding chain requires a separately +named typed field and vendor sign-off, never reuse of the raw header mapping. + +Platform host evidence: + +- `TlsProtocol`, capped by TS at 32 bytes +- `JA4`, capped by TS at 128 bytes, only when the operator explicitly sets + `[integrations.datadome] expose_host_fingerprints_to_vendor = true`; + the default is `false`, omission is represented by absence rather than an + empty field, and startup logs the additional vendor disclosure +- `TlsCipher` is omitted in v1: DataDome defines it as the ordered list of + cipher suites offered by the client, while `RuntimeServices::client_info()` + exposes only the negotiated cipher. Substituting that value would silently + change the field's meaning +- `H2Fingerprint` is omitted in v1 because the current Protection API contract + does not define such a request field + +`X-DataDome-ClientID` is never a Protection API source in cookie-mode v1. +No wildcard (`Sec-CH-*`, `Sec-Fetch-*`, `X-*`, or otherwise) expands this +list. + +The following limits are bytes of the decoded field value before form +encoding. Truncation is UTF-8-boundary-safe. `XForwardedForIP` alone truncates +from the end; every other bounded field retains its prefix: + +- 8 bytes: `SecCHDeviceMemory`, `SecCHUAMobile`, + `SecFetchStorageAccess`, `SecFetchUser` +- 16 bytes: `SecCHUAArch` +- 32 bytes: `SecCHUAPlatform`, `SecFetchDest`, `SecFetchMode`, and the TS cap + on `TlsProtocol` +- 64 bytes: `ContentType`, `SecFetchSite`, and the TS cap on `ServerRegion` +- 128 bytes: `AcceptCharset`, `AcceptEncoding`, `CacheControl`, `Connection`, + `From`, `Pragma`, `SecCHUA`, `SecCHUAModel`, `X-Requested-With`, and the TS + cap on opt-in `JA4` +- 256 bytes: `AcceptLanguage`, `SecCHUAFullVersionList`, `Via` +- 512 bytes: `Accept`, `ClientID`, `HeadersList`, `Host`, `Origin`, + origin-only `Referer`, `ServerHostname`, and `ServerName` +- 768 bytes: `UserAgent` +- 2,048 bytes: path-only `Request` + +`Key`, `AuthorizationLen`, `CookiesLen`, `IP`, `Method`, `ModuleVersion`, +`Port`, `PostParamLen`, `Protocol`, `RequestModuleName`, and `TimeRequest` are +unbounded per-field by the vendor table but remain subject to the total bound. +The complete `application/x-www-form-urlencoded` body, including field names, +`=`/`&` separators, and percent-encoding expansion, must be at most **24,576 +bytes**. Core constructs and measures the whole payload before issuing the +request. It does not silently drop optional fields to fit: overflow skips the +vendor call and takes the same metered fail-open `Continue` path as a transport +failure. + +This is deliberately narrower than DataDome's currently documented required +surface: notably, it withholds `CookiesList` and omits empty source-header +fields. Product/vendor sign-off 28 therefore requires written confirmation +that this exact reduced profile is supported. Until that confirmation and +adapter conformance fixtures exist, the DataDome integration is not +release-qualified. ## Request-direction pointer (vendor response → publisher-upstream overlay) @@ -12,9 +209,9 @@ channel (hook spec §4a) may copy into the owner-scoped publisher-upstream overlay. Every `X-DataDome-*` name not listed here is rejected. -| Header | Direction | Scope | -| --------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------- | -| `X-DataDome-ClientID` | response → upstream overlay | Owner-scoped overlay only; never the shared request view; vendor egress governed by sign-off item 23 | +| Header | Direction | Scope | +| --------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `X-DataDome-ClientID` | response → upstream overlay | Disabled by default; admitted only with `expose_client_id_to_origin = true`. Owner-scoped publisher overlay only, never the shared request view or another integration | ## The single pointer matrix (normative — decision × session mode × pointer) @@ -24,25 +221,35 @@ column is added by the sign-off-23 opt-in, never implicitly). No wildcard rows exist — every accepted name is enumerated, and **every cell terminates in exactly one outcome**. -| Pointer | Respond (cookie mode) | Continue (cookie mode) | -| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------ | -| `Set-Cookie` | typed `datadome` cookie operation (hook spec §4a parser; exactly one `datadome` field) | typed `datadome` cookie operation | -| `X-Set-Cookie` | **invalidate the batch → Continue** (mode mismatch: TS never requests header sessions in v1; a vendor response asserting one must not half-apply) | **invalidate the batch** → effects dropped | -| `Location` | forward on a 3xx Respond, replace; **on a non-3xx Respond → invalidate the batch → Continue** (a redirect field without a redirect status is a malformed decision) | **invalidate the batch** (effects dropped) | -| `Content-Type` | forward (owns its body), replace | **invalidate the batch** (effects dropped) | -| `Cache-Control` | restricted merge; invariant pass last | **invalidate the batch** (effects dropped) | -| `Pragma` | drop-individually, logged (response `Pragma` has no standardized meaning, RFC 9111 §5.4) | drop-individually, logged | -| `X-DataDome` | forward, owner-scoped typed telemetry | forward, owner-scoped typed telemetry | -| `X-DD-B` | drop-individually, logged (header-session artifact; the vendor's cookie-mode allow example emits it — the drop is the named divergence in sign-off 28) | drop-individually, logged | -| anything not listed | invalidate the batch → Continue | invalidate → effects dropped | +| Pointer | Respond (cookie mode) | Continue (cookie mode) | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------- | +| `Set-Cookie` | typed `datadome` cookie operation (hook spec §4a parser; exactly one `datadome` field) | typed `datadome` cookie operation | +| `X-Set-Cookie` | **invalidate the batch → Continue** (mode mismatch: TS never requests header sessions in v1; a vendor response asserting one must not half-apply) | **invalidate the batch** → effects dropped | +| `Location` | forward on a 3xx Respond, replace; **on a non-3xx Respond → invalidate the batch → Continue** (a redirect field without a redirect status is a malformed decision) | **invalidate the batch** (effects dropped) | +| `Content-Type` | forward (owns its body), replace | **invalidate the batch** (effects dropped) | +| `Cache-Control` | restricted merge; invariant pass last | **invalidate the batch** (effects dropped) | +| `Pragma` | drop-individually, logged (response `Pragma` has no standardized meaning, RFC 9111 §5.4) | drop-individually, logged | +| `X-DataDome` | forward, owner-scoped typed telemetry | forward, owner-scoped typed telemetry | +| `X-DD-B` | forward as a browser-response security signal; never copy to publisher-upstream or another integration | forward as a browser-response security signal | +| anything not listed | invalidate the batch → Continue | invalidate → effects dropped | Singleton-field multiplicity (`Location`, `Content-Type`, `X-DataDome`, `X-DD-B`, `X-Set-Cookie` appearing more than once) invalidates the batch atomically; list-valued fields (`Cache-Control`, `Pragma`) join per RFC 9110 §5.3 before their cell applies (hook spec §4a). +`X-DD-B` is security-owned when DataDome is enabled. Before applying the fresh +security batch, core removes every pre-existing instance from the origin, +cached ordinary artifact, 304 metadata update, core response, or ordinary +mutator. A valid pointed vendor value then uses **replace-all** and the final +response cardinality must be exactly one; if the fresh vendor batch does not +point to it, final cardinality is zero. Append is never allowed. Fixtures cover +origin collision, cache-hit collision, 304 collision, repeated vendor fields, +and one valid fresh value, proving “exactly once” at final emission rather than +merely inside the vendor batch. + **Fixtures**: DataDome's documented challenge response (`Set-Cookie`, `Pragma`, `X-DataDome`, `Cache-Control`) asserts the decision stays **Respond** with exactly the mapped fields; the documented allow example (`Set-Cookie X-DD-B`) asserts Continue proceeds with the cookie applied -and `X-DD-B` dropped-and-logged — neither fixture may fail open. +and `X-DD-B` forwarded exactly once — neither fixture may fail open. diff --git a/docs/superpowers/specs/gpp-registry-snapshot.md b/docs/superpowers/specs/gpp-registry-snapshot.md index 0eac72a20..88ed4c335 100644 --- a/docs/superpowers/specs/gpp-registry-snapshot.md +++ b/docs/superpowers/specs/gpp-registry-snapshot.md @@ -25,40 +25,54 @@ here is treated as malformed-present (permission spec §4.4). | 21 | usnj | 1 | | 22 | ustn | 1 | | 23 | usmn | 1 | +| 24 | usmd | 1 | +| 25 | usin | 1 | +| 26 | usky | 1 | +| 27 | usri | 1 | -Version values for sections 6–23 were captured from the IAB registry at -the time of writing and are re-verified against the official registry as -part of ratification review. Sections 24–27 have assigned IDs but no -reproducibly published binary layouts in the official sources as of this -snapshot; they are reserved and inert until an official layout can be -vendored here. Any change is a reviewed change to this file. +At the pinned commit below, the official section registry assigns IDs 24–27 +to MD, IN, KY, and RI and each named state specification defines accepted +version 1. That commit-backed statement, rather than an unverified publication +month, is the authority for admitting them. Treating them as national-only +would discard a state-specific choice. Unknown IDs outside the accepted table +still contribute nothing and are flagged for snapshot review. -## Reserved sections — NOT accepted, no version +## Provenance and vectors -These state sections have assigned IDs but no reproducibly published -official binary layout as of this snapshot. They are **not** in the -accepted-version table above: an implementation MUST NOT decode them, -and a request carrying one behaves national-section-only (permission -spec §4.5, sign-off 32). A reserved ID is _expected-inert_; an unknown -ID (outside both tables) is _flagged for snapshot review_ — the only -observable difference is logging. +The immutable authority is the official +`InteractiveAdvertisingBureau/Global-Privacy-Platform` commit: -| GPP section ID | State | Status | -| -------------- | --------- | ----------------------------- | -| 24 | usmd (MD) | reserved — no official layout | -| 25 | usin (IN) | reserved — no official layout | -| 26 | usky (KY) | reserved — no official layout | -| 27 | usri (RI) | reserved — no official layout | +`00ffaefe91513785e886c83877e9b56a4ec8e88c` -## Provenance and vectors +Normative upstream paths for the newly admitted layouts are: + +- `Sections/US-States/MD/Maryland Privacy Technical Specification.md` +- `Sections/US-States/IN/Indiana Privacy Technical Specification.md` +- `Sections/US-States/KY/Kentucky Privacy Technical Specification.md` +- `Sections/US-States/RI/Rhode Island Privacy Technical Specification.md` +- `Sections/Section Information.md` + +The implementation vendors decoder fixtures under +`crates/trusted-server-core/testdata/gpp/00ffaefe91513785e886c83877e9b56a4ec8e88c/`. +That directory contains a `manifest.json` object with: -Supported sections (6–23) pin to the official IAB GPP registry revision -recorded by the implementation PR (immutable upstream commit hash), with -per-section encoded conformance vectors vendored alongside. A date is not -a revision; the commit hash is the reproducible authority. +- `upstream_commit_oid` and `upstream_commit_tree_oid`; +- a sorted `sources` array containing `{path, blob_oid, sha256_hex}` for all + five normative paths above — the four state specifications and + `Sections/Section Information.md`; and +- a sorted `cases` array whose entries are + `{section_id, version, case, encoded, expected}`. -**Status: placeholder until ratification.** Neither the immutable -registry commit nor the conformance vectors are recorded yet; like the -PSL snapshot, filling them is a pre-ratification prerequisite -(migration spec §4) — the §4.5 field mappings cannot be reproduced -against a pinned registry until they land. +The vendoring PR description quotes the same commit/tree/blob values and the +independent command output used to verify every raw source SHA-256 and the +byte-for-byte copy. A commit OID without its tree and source-blob witnesses is +not accepted as completed provenance. `expected` uses the +permission spec's normalized P1/P4/GPC tokens, not decoder-library enums. +Fixture encodings must be constructed from the pinned bit layouts by an +independent generator or hand-checked vector, never emitted and consumed only +by the decoder under test. For every accepted section/version the corpus must +contain: minimum valid core-only string, core + GPC true, each mapped opt-out +value, each explicit not-opted-out value, explicit N/A, malformed/truncated +input, unsupported version, and a mixed known/unknown-section string. CI +refuses to update this file unless the complete corpus for the new commit is +present. diff --git a/docs/superpowers/specs/policy-canonicalization-vectors.json b/docs/superpowers/specs/policy-canonicalization-vectors.json new file mode 100644 index 000000000..0df065de3 --- /dev/null +++ b/docs/superpowers/specs/policy-canonicalization-vectors.json @@ -0,0 +1,77 @@ +{ + "schema_version": 1, + "canonicalization": "RFC 8785 (JCS)", + "digest": "SHA-256", + "domain_prefix_utf8": "tspol1|", + "vectors": [ + { + "name": "minimal-gdpr-policy", + "effective_policy": { + "rules": { + "default": "gdpr" + }, + "groups": { + "gdpr": { + "regime": "gdpr", + "default": "requires_signal" + } + } + }, + "canonical_json_utf8": "{\"groups\":{\"gdpr\":{\"default\":\"requires_signal\",\"regime\":\"gdpr\"}},\"rules\":{\"default\":\"gdpr\"}}", + "sha256_hex": "68f72cc004bd59df7f799be24685b85cbbf5d5fbd1ba3069c8bf51adf4a88e6b" + }, + { + "name": "us-country-floor-and-state-override", + "effective_policy": { + "rules": { + "default": "non-regulated", + "US/CA": { + "overrides": { + "select-personalised-ads": "requires_signal" + }, + "group": "us-opt-out" + }, + "US": "us-opt-out" + }, + "groups": { + "us-opt-out": { + "regime": "us-privacy", + "default": "requires_signal" + }, + "non-regulated": { + "regime": "none", + "default": "granted" + } + } + }, + "canonical_json_utf8": "{\"groups\":{\"non-regulated\":{\"default\":\"granted\",\"regime\":\"none\"},\"us-opt-out\":{\"default\":\"requires_signal\",\"regime\":\"us-privacy\"}},\"rules\":{\"US\":\"us-opt-out\",\"US/CA\":{\"group\":\"us-opt-out\",\"overrides\":{\"select-personalised-ads\":\"requires_signal\"}},\"default\":\"non-regulated\"}}", + "sha256_hex": "6c578c849c323936dc6d492449214c19e16e968b01a962c6ef99e9bbe3a08553" + }, + { + "name": "explicit-permission-map-without-default", + "effective_policy": { + "rules": { + "default": "explicit" + }, + "groups": { + "explicit": { + "regime": "none", + "permissions": { + "store-on-device": "granted", + "select-personalised-ads": "requires_signal" + } + } + } + }, + "canonical_json_utf8": "{\"groups\":{\"explicit\":{\"permissions\":{\"select-personalised-ads\":\"requires_signal\",\"store-on-device\":\"granted\"},\"regime\":\"none\"}},\"rules\":{\"default\":\"explicit\"}}", + "sha256_hex": "47745dbb0b5cf113e4d2eb9dda48e6fc9c6c8c18dc39ee715e9838edfd57727b" + } + ], + "rejection_vectors": [ + { + "name": "null-is-not-a-materialized-policy-value", + "raw_json": "{\"rules\":{\"default\":null}}", + "error": "null value" + } + ] +} diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index 916397385..b719340f1 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -631,3 +631,255 @@ and the cookie citations updated to RFC 10025 (obsoleting RFC 6265). Ratification state: unchanged — 32 open rows, decisions/ empty beyond the README, no adapter qualified for the identity protocol or the header ceilings, GPP/PSL snapshots placeholders. All user-side gates. + +## Post-decision pass 1 (author assessment; formerly internal Round 21) + +The product posture was re-reviewed and the recommended resolutions were +applied to the normative text while ratification remains open. This round +supersedes Round 20 where the two conflict; historical rows above are not +current requirements. + +- GPC and GPP/USP sale, sharing, and targeted-advertising choices are + persistent P4 use opt-outs, not identity deletion or P1 withdrawal. + Destruction is limited to the exhaustive permission §4.2 triggers. +- N/A, absence, reserved, unknown, and unsupported values never grant. + Regionless/country-only US traffic uses the protective country floor; + selected-provider geo failure uses the protective failure profile. +- Use opt-outs do not expire at a consent TTL and replay saturation never + shortens them. Expired/grant history is evicted first; restrictive state + retains its complete horizon. +- Prefix saturation blocks rowless admission only. It never revokes an + authenticated real row without an exact withdrawn-suffix match. +- Failed negative writes use a durable per-family outbox checked by every + identity consumer; failure of both target and outbox closes a globally + visible breaker over all positive identity operations while repair remains + enabled. +- Host JA4/H2 fingerprinting is deferred and startup-rejected; the shipped + classifier is UA-only and no fingerprint-derived value is newly stored. +- Policy distribution uses a separate monotonic push-sequence allocator and + prepare/commit fleet activation. Mixed-revision destructive execution is + prohibited. JCS digest vectors are now checked in. +- DataDome has a narrow `SecurityUse` lifecycle, no publisher-origin ClientID + exposure by default, a required bounded cookie lifetime, reserved response + budget, and exactly-once browser forwarding of documented `X-DD-B`. +- GPP sections 24–27 are accepted at official version 1 and pinned to the IAB + repository commit in `gpp-registry-snapshot.md`; the PSL reference is also + pinned. The generated GPP corpus and vendored PSL bytes remain release + prerequisites, not facts already present in this worktree. + +Ratification state remained user-side: all 32 rows then listed were `open` and +no decision record was fabricated. Later passes added rows 33–34; the current +table therefore has 34 open rows. Adapter qualification cells and the required +vendored artifacts remain explicit pre-implementation gates. + +## Post-decision pass 2 (internal read-only assessment; formerly Round 22) + +An internally dispatched read-only pass rechecked the then-current normative text across +activation, negative state, replay, jurisdiction, GPP aggregation, cache +security, and the DataDome lifecycle. It reported no P0. This pass supersedes +post-decision pass 1 where the implementation contracts became more precise: + +- Config publication is immutable and version-addressed. Candidate, + readiness, and active records bind the complete blob/data/config/policy + tuple; a mutable latest blob has no activation authority, and config-only + pushes stage the entire settings snapshot while retaining the policy + ordinal. +- The negative outbox is in a failure domain independent of its target and now + has a bounded, absorbing, evidence-ordered enqueue/apply/exact-ack state + machine. Capability qualification includes failure semantics, not merely + three strong key APIs. +- Timestamp-less GPP/USP/GPC replay cannot grant, but a currently presented + restrictive value reasserts suppression after an ordered clear. GPC has an + assigned source token and digest vectors. +- Authority summaries carry tagged live/static/failure jurisdiction + provenance. Protective lookup failure cannot authorize context-free S2S; + static provenance is fenced to its active config and age horizon. +- Mapped malformed GPP sections now participate in the ordered aggregator as + P4 grant blockers. Because this is a product choice, it is new open sign-off + item 33 rather than a fabricated decision. +- The DataDome filter no longer receives raw `Request` or generic header + mutations. Cookie set/delete uses one configured ownership tuple; + `datadome` and `X-DD-B` collisions are stripped under security ownership; + the config inventory, request limits, exposure defaults, and adapter gates + are single-source and exact. +- Vary-HMAC rotation has a versioned keyring grammar, deterministic key IDs, + overlap/retirement rules, refresh-on-unknown behavior, and adapter + qualification fixtures. + +At that pass 33 rows were open; the current table has 34. No decision record +exists beyond the README, no adapter currently qualifies the new stateful +identity or response-artifact protocols, and the GPP corpus plus PSL bytes/hash +remain missing release artifacts. The recommended spec posture is internally +stated; product ratification and implementation planning remain separate gates. + +The follow-up internal blocker pass reported **no remaining P0 or P1** after +the generic request-filter and sealed DataDome security-filter APIs were +physically separated and the legacy generic security-header channel was +removed. This is an internal assessment, not external verified closure. + +## Post-decision pass 3 (author whole-surface assessment; formerly Round 23) + +A fresh internal specification-only review found no P0, but found six remaining P1 +contract contradictions and six P2 completeness gaps. This pass supersedes +post-decision pass 2's “no remaining P1” conclusion. The approved conservative resolutions +are now normative: + +- The negative-intent outbox has the missing `q` physical constructor/parser, + one fully materialized JCS transition schema, closed source/cause tokens, and + a known-answer vector that includes evidence and a post-clear authorization + floor. Deferred transaction tag `x` is no longer presented as an allocated + v1 key. +- Whole-config prepare/commit now names authenticated authoritative membership, + stable member IDs, membership-change restaging, startup admission, and a + strong per-request fence over every settings consumer. A hash-linked + activation journal, atomically bound by the promotion CAS, separates the + minimum 30-day/time-horizon audit and GC clock from the 16-entry operational + history. +- N+1 remains a fail-restrictive reader of N+2 negative state and can persist + explicit pre-epic withdrawal/deletion, but it does not originate durable P4 + use suppression. Suppression creation begins only with N+2, active new-shape + configuration, and the fleet-wide `permissions_v2` model promotion, + matching the promised pre-epic live gate. +- DataDome Protection API egress is fixed to one HTTPS host/path with no + redirect following. `Request` is path-only, `Referer` origin-only, raw + forwarding-IP headers are omitted, IP/port come from trusted connection + metadata, and publisher path disclosure is named honestly in vendor + retention/DSR sign-off rather than called identity-free. +- Auction dispatch with P4 unset uses a positive `ContextualAuctionView`, not + the ordinary request minus TS IDs. The view excludes client IP/UA, precise + geo, page/referrer URL, user IDs/data/segments, arbitrary extensions, device + fingerprints, and client forwarding headers; an unqualified destination + receives no request. +- Policy groups have one exact `permissions` child-map grammar and a canonical + vector. GPP transport derives `gpp_sid` from decoded applicability and emits + the GPP/SID pair atomically. Vary-HMAC lookup uses a stable variant index that + exposes the stored key ID before digest comparison. Long provider identifiers + use an explicit collision-detecting SHA-256 mode with a vector and fail-closed + canonical-identifier comparison. +- The normative cleanup removes the config-only-push typo, uses the consistent + policy/config-activation name, and replaces invalid bare workspace test/lint + commands with the repository's target-matched aliases. + +Ratification state remains separate: product decision rows 1–34 are open, no +decision record is fabricated, adapter qualification remains pending, and the +vendored GPP corpus plus PSL bytes/hash remain release prerequisites. + +## Post-decision pass 4 (internal executable-contract assessment; formerly Round 24) + +Two internally dispatched read-only passes rechecked the edited specification +set. The first found two P1 and four P2 gaps in the post-decision pass 3 +repairs; iterative fixes and a final internal whole-surface pass reported **no +remaining P0, P1, P2, or P3 spec finding at that revision**. This is a +self-assessed result, not a claim of externally verified closure. The contracts +at that revision were: + +- Settings and writer/model activation share one strong register with an + explicit logical `activation_generation`. Both transitions require immutable + fleet membership, unanimous readiness, an all-request admission stop, and + bound quiescence before the promotion CAS. A never-reused candidate + incarnation plus incrementing drain attempt prevents delayed readiness or + quiescence acknowledgments from crossing cancel/abort/restage boundaries. + N+2 emulates N+1 until the quiescent `permissions_v2` CAS; afterward the + minimum binary generation excludes N+1 before admission. +- The activation journal now has a portable JCS object ID, exact schema, + candidate/drain/readiness/quiescence bindings, safe-integer profile, + store-clock lifecycle, snapshot-consistent listing, conservative GC, and + authenticated checkpoint pruning. Its known-answer object ID is + `2ac2ab49922f261a8eecaee64f3621da8a2f2c1061c945defd7bc75ac2d5a569`, + with three numeric rejection vectors. +- Contextual auction output is governed by the sole machine-readable + `contextual-openrtb-v1-allowlist.json`: 98 unique exact leaf rules, closed + type/cardinality/derivation vocabularies, executable container/cross-field + constraints, atomic GPP/GPP-SID transport, nonempty supply chain, and exact + banner/video shapes. Unknown, unlisted, ill-typed, or untraceable output + suppresses dispatch. +- Processed artifacts, mutation IR, variant descriptors, and indexes bind the + complete cache revision tuple, including `model_epoch` and logical + `activation_generation`; a model-only cutover cannot replay a pre-epic + artifact. `Vary` names have one lowercase/deduplicated/sorted grammar and + repeated request values retain presence, instance, length, order, and octet + identity under keyed digests. +- `sha256-detect` graph keys carry an atomic canonical-identifier collision + witness checked before every read/write/merge/use. DataDome request headers + now have exact singleton/list/cookie multiplicity, OWS/UTF-8 handling, + length, omission, and cross-adapter fixture rules. + +This was author-assessed specification completeness, not product ratification +or implementation readiness. A subsequent external reviewer independently +recomputed the then-current eleven known-answer vectors (including the +byte-exact JCS intent ID) and the 98 unique contextual allowlist leaves; those +computations are externally verified for that revision, while the broader +“no findings” statement remains internal. Product decision rows 1–34 remain +open; no decision record has been fabricated. Adapter qualification, the +generated/vendored GPP corpus, and the pinned PSL bytes/hash remain explicit +release gates. + +## Post-decision pass 5 (external consequence review, 2026-08-05) + +A subsequent external specification review found no P0 and no internal P1 +contradiction, but identified two material operational consequences that the +sign-off rows did not state plainly, plus P2/P3 audit and runbook gaps. The +approved conservative resolutions are now normative: + +- Whole-settings/model serve admission may use only the deployment-qualified + bounded activation lease. The draining CAS records a non-early store-clock + promotion-not-before; lease expiry closes old admission unless a fresh + non-draining renewal succeeds, while every authenticated member's + quiescence still proves that the last request and background effect ended. + Successful authority, revocation, outbox, `w`, and breaker reads remain fresh + for every positive identity decision; only typed restrictive results may be + cached to deny. +- Sign-off 19 now says directly that every ordinary settings promotion is a + scheduled fleet-wide deployment-unavailable interval and is not a + zero-downtime config protocol. A narrower or blue/green drain remains a + separate effect-classification design. +- The authenticated deployment controller owns an idempotent post-model-CAS + `m00` raise/read-verify step. A lower mirror retries without changing active; + a higher mirror remains a fail-closed register/journal inconsistency. +- Non-destructive signals on the old rowless identity are request-local and + create no per-family or `w` state. If the same request is permitted to mint a + new row-backed family, that family's current suppression becomes durable + before its cookie or identity is usable. +- The geo-failure behavior row now declares the signal-backed grant divergence; + decision rows 11 and 16 name their rejected unbounded-residual and TTL-sticky + alternatives; snapshot vendoring requires commit-tree, source-blob, and + byte-hash witnesses; and the unsupported publication-month assertion was + replaced by commit-backed GPP registry/layout facts. +- Internal rounds 21–24 were renamed post-decision passes and their + self-assessed versus externally verified evidence is explicit. + +The activation-journal schema and KAT gained the lease bound and +promotion-not-before witness, so pass 4's externally recomputed activation +object ID is historical. The current local KAT object ID is +`7af3934b4e5500903ef77ad8a1367db83b03fc62a0bb83bd0849289c685d2e88`; +it requires fresh external recomputation before being described as externally +verified. Ratification remains unchanged: all 34 decision rows are open, +`decisions/` contains no fabricated approval, no adapter qualifies the complete +stateful identity/activation protocol, and vendored GPP/PSL artifacts remain +release prerequisites. + +## Post-decision pass 6 (independent internal targeted assessment, 2026-08-05) + +An independent read-only pass over the consequence-review changes found no P0 +and recomputed the current activation-journal object ID exactly. It identified +five cross-spec gaps, now text-closed: + +- Stateless identity no longer appears to waive the universal activation + fence. Every serving adapter must qualify whole-settings/model activation; + stateless selection waives only identity-state capabilities. +- The register promotion gate and journal `created_at` comparison now require + one explicitly qualified authenticated time domain. Adapters with + incomparable register/object-store clocks fail qualification rather than + performing an undefined timestamp comparison. +- The exhaustive model-candidate identity list now includes the snapshotted + admission-lease bound. +- Qualification fixtures now reject a zero config/model bound, candidate or + readiness mismatch, and attempted in-traffic bound change before drain. +- The migration runbook's mirror-completion cross-reference now points to §4 + requirement 5 rather than rollout item 5. + +The independent internal recomputation produced +`7af3934b4e5500903ef77ad8a1367db83b03fc62a0bb83bd0849289c685d2e88`. +This verifies the current KAT inside the review process but is not relabeled as +the fresh external recomputation requested in pass 5. Decision rows and release +gates remain unchanged. diff --git a/docs/superpowers/specs/psl-snapshot-ref.md b/docs/superpowers/specs/psl-snapshot-ref.md index 72bb3a519..7da249d49 100644 --- a/docs/superpowers/specs/psl-snapshot-ref.md +++ b/docs/superpowers/specs/psl-snapshot-ref.md @@ -7,7 +7,23 @@ sections apply; hostnames are IDNA-mapped before matching; IP literals and single-label hosts have no registrable domain (cookie falls back to host-only). Updating the snapshot is a reviewed spec change. -| Field | Value | -| --------------- | --------------------------------------------------------- | -| Upstream commit | _recorded by the implementation PR that vendors the list_ | -| Vendored path | _recorded alongside_ | +| Field | Value | +| ---------------------- | -------------------------------------------------------------------- | +| Upstream repository | `publicsuffix/list` | +| Upstream commit | `e1b8015c3b2f0f4f8c18659c2480fc1a22c07b20` | +| Upstream source path | `public_suffix_list.dat` | +| Required vendored path | `crates/trusted-server-core/data/public_suffix_list.dat` | +| Required hash path | `crates/trusted-server-core/data/public_suffix_list.sha256` | +| Required provenance | `crates/trusted-server-core/data/public_suffix_list.provenance.json` | + +The implementation copies the source bytes at that commit without editing +and writes the lowercase 64-hex SHA-256 plus one trailing LF (no filename or +other fields) to the required hash path. CI verifies the bytes, hash, and +commit reference together; updating any one without the others fails. The +provenance file is canonical JSON with exactly +`{upstream_repository, upstream_commit_oid, upstream_commit_tree_oid, +source_path, source_blob_oid, source_sha256_hex}`. The vendoring PR description +quotes the same commit/tree/blob values and the independent command output +that verified the raw upstream SHA-256 and byte-for-byte vendored copy. A +commit OID without its tree and source-blob witness does not satisfy the +release gate. diff --git a/docs/superpowers/specs/revision-canonicalization-vectors.json b/docs/superpowers/specs/revision-canonicalization-vectors.json new file mode 100644 index 000000000..070d67d81 --- /dev/null +++ b/docs/superpowers/specs/revision-canonicalization-vectors.json @@ -0,0 +1,45 @@ +{ + "schema_version": 1, + "canonicalization": "RFC 8785 (JCS)", + "digest": "SHA-256", + "vectors": [ + { + "name": "ordered-integration-registry", + "domain_prefix_utf8": "tsreg1|", + "normalized_value": [ + { + "behavior_revision": 1, + "id": "datadome" + }, + { + "behavior_revision": 2, + "id": "prebid" + } + ], + "canonical_json_utf8": "[{\"behavior_revision\":1,\"id\":\"datadome\"},{\"behavior_revision\":2,\"id\":\"prebid\"}]", + "sha256_hex": "a2a81a6727226821d87f885c92410b5ebd2466e1060a070e027ad0eba210eff4" + }, + { + "name": "effective-config-hash-grammar-smoke", + "domain_prefix_utf8": "tscfg1|", + "normalized_value": { + "integrations": { + "datadome": { + "secret_name": "datadome-api-key", + "enabled": true + } + } + }, + "canonical_json_utf8": "{\"integrations\":{\"datadome\":{\"enabled\":true,\"secret_name\":\"datadome-api-key\"}}}", + "sha256_hex": "83c85084578ee35ddba12418c6337b7cc064b7022be5c8ffed068e94b07118d6" + }, + { + "name": "config-sequence-binding", + "domain_prefix_utf8": "tscfgseq1|", + "push_sequence": 42, + "push_sequence_u64_be_hex": "000000000000002a", + "data_hash_hex": "0000000000000000000000000000000000000000000000000000000000000000", + "sha256_hex": "70edd94cd5728550a815355a1b719f4aafb466aa228571a4a7a6e88ad5178df0" + } + ] +} From be3e8995d4211f065827a87797ffb312abecdf07 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Thu, 6 Aug 2026 14:10:48 -0700 Subject: [PATCH 23/24] docs: inline normative specification annexes Move machine-readable vectors, registry snapshots, and DataDome/PSL contracts into their owning existing specs for one-place review. --- ...-datadome-server-side-protection-design.md | 27 +- ...2026-06-16-edgezero-based-ts-cli-design.md | 119 ++- ...integration-response-header-hook-design.md | 297 +++++- .../2026-07-30-permission-model-design.md | 927 +++++++++++++++++- ...07-30-provider-migration-rollout-design.md | 10 +- .../specs/activation-journal-vectors.json | 98 -- .../contextual-openrtb-v1-allowlist.json | 668 ------------- .../specs/datadome-header-allowlist.md | 255 ----- .../specs/gpp-registry-snapshot.md | 78 -- .../policy-canonicalization-vectors.json | 77 -- docs/superpowers/specs/pr986-review-ledger.md | 8 +- docs/superpowers/specs/psl-snapshot-ref.md | 29 - .../revision-canonicalization-vectors.json | 45 - 13 files changed, 1345 insertions(+), 1293 deletions(-) delete mode 100644 docs/superpowers/specs/activation-journal-vectors.json delete mode 100644 docs/superpowers/specs/contextual-openrtb-v1-allowlist.json delete mode 100644 docs/superpowers/specs/datadome-header-allowlist.md delete mode 100644 docs/superpowers/specs/gpp-registry-snapshot.md delete mode 100644 docs/superpowers/specs/policy-canonicalization-vectors.json delete mode 100644 docs/superpowers/specs/psl-snapshot-ref.md delete mode 100644 docs/superpowers/specs/revision-canonicalization-vectors.json diff --git a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md index d262beba2..21375e9cd 100644 --- a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md +++ b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md @@ -7,8 +7,8 @@ > (`2026-07-30-integration-response-header-hook-design.md`): one global > order applies (core finalization → ordinary mutators → security > effects → final cache/privacy invariant pass, unconditionally last), -> with typed cookie/header operations, enumerated allowlists -> (`datadome-header-allowlist.md`), and owner-only identifier +> with typed cookie/header operations, the enumerated field contract in +> §4a.2 of that spec, and owner-only identifier > boundaries. Where this document conflicts, the hook spec governs. Additionally, this document's sessionByHeader requirement ("always > send `X-DataDome-X-Set-Cookie` when the header ID is used") is > **superseded for v1**: header-session mode is startup-rejected (hook @@ -240,7 +240,7 @@ integrations cannot construct or read them or recover the underlying raw request. Core strips `ts-*`, EID/identity material, `X-DataDome-ClientID`, and the `datadome` cookie before building the shared view; the security view restores only the one typed cookie value and exact -request evidence admitted by `datadome-header-allowlist.md`. Another filter +request evidence admitted by the hook spec §4a.2.1. Another filter receives only the shared redacted view and cannot inherit this owner capability. This paragraph and the hook spec §4a replace every earlier generic `&Request`/generic security-header-mutation sketch in this document. The @@ -320,8 +320,9 @@ committed via `stream_to_client()`. ### 4. Header Mutation Semantics DataDome pointer headers are internal instructions and are never forwarded. -The one normative field/pointer allowlist and decision matrix is -`datadome-header-allowlist.md`; a pointer does not authorize an unlisted name. +The one normative field/pointer contract is the hook spec §4a.2, with the +publisher-upstream overlay in §4a.2.2 and the browser-response matrix in +§4a.2.3; a pointer does not authorize an unlisted name. | Pointer header | Destination | | ---------------------------- | -------------------------------------------------- | @@ -541,9 +542,8 @@ Content-Length: X-DataDome-X-Set-Cookie: true # only when X-DataDome-ClientID is used — SUPERSEDED for v1: never sent (hook spec §4a) ``` -The exhaustive payload field set is the Protection API request-field section -of `datadome-header-allowlist.md`. The list below is informative and may not -expand that normative allowlist: +The exhaustive payload field set is the hook spec §4a.2.1. The list below is +informative and may not expand that normative allowlist: - `Key` - `IP` @@ -656,8 +656,8 @@ fail open and continue without effects. For challenge statuses: 1. Build a response using DataDome's API response status and body. -2. Validate the complete decision-scoped pointer batch against - `datadome-header-allowlist.md` and the typed-cookie contract. +2. Validate the complete decision-scoped pointer batch against the hook spec + §4a.2.3 and the typed-cookie contract. 3. Apply the accepted security batch atomically. 4. Do not contact the publisher origin. 5. Run the final cache/privacy invariant pass after the security batch. @@ -666,8 +666,8 @@ For challenge statuses: For allow status `200`: -1. Apply only the owner-scoped publisher-upstream fields admitted by - `datadome-header-allowlist.md` before route matching; the default is no +1. Apply only the owner-scoped publisher-upstream fields admitted by the hook + spec §4a.2.2 before route matching; the default is no ClientID exposure. 2. Validate and retain the decision-scoped browser security batch. 3. Continue normal route matching. @@ -948,8 +948,7 @@ passes. it unless `expose_host_fingerprints_to_vendor = true`. `TlsCipher` is omitted because the platform exposes a negotiated cipher while the vendor field means ordered offered ciphers; `H2Fingerprint` is not a documented - Protection API field. Admit no host evidence outside - `datadome-header-allowlist.md`. + Protection API field. Admit no host evidence outside the hook spec §4a.2.1. 3. **Challenge status source of truth:** follow the Protection API docs in v1: `301`, `302`, `401`, `403`, and `429` are challenge statuses when `X-DataDomeResponse` matches the HTTP status. diff --git a/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md b/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md index 3edccf807..2b8ab8f49 100644 --- a/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md +++ b/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md @@ -177,8 +177,7 @@ The envelope contains: `SHA-256("tscfgseq1|" || push_sequence.to_be_bytes() || data_hash_bytes)`, where the domain tag is UTF-8, the integer is unsigned 64-bit big-endian, and `data_hash_bytes` is the 32 decoded bytes of the preceding hash; the - known-answer vector is in - `docs/superpowers/specs/revision-canonicalization-vectors.json`; + known-answer vector is inline in the permission spec §5.5.2; - generation timestamp metadata. Runtime loading must verify both the data hash and sequence-binding hash before @@ -285,8 +284,8 @@ on a language's larger integer type. to at least 2,592,000,000 and the longest applicable artifact, cookie-scope, rollback, and audit horizon. -The cross-language known-answer vector is -`docs/superpowers/specs/activation-journal-vectors.json`; every controller, +The cross-language known-answer and rejection vectors are inline in §5.1.1; +every controller, runtime verifier, and GC must reproduce both JCS bytes and object ID and reject every numeric boundary vector. For the first promotion, `previous_journal_id` is null only when the register head is @@ -361,6 +360,118 @@ Request-signing public/private state is intentionally out of scope for this initial CLI. It will be revisited after EdgeZero exposes suitable secret-store write primitives. +#### 5.1.1 Activation-journal vectors + +The JSON object between the stable markers is the sole normative +machine-readable activation-journal fixture. Extractors exclude the markers +and code fences, parse the enclosed UTF-8 JSON, and must reject duplicate +object keys. + + + +```json +{ + "schema_version": 1, + "canonicalization": "RFC 8785 (JCS)", + "digest": "SHA-256", + "domain_prefix_utf8": "tsactj1|", + "numeric_profile": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "vectors": [ + { + "name": "genesis-config-promotion", + "journal": { + "schema_version": 1, + "attempt_id": "00000000000000000000000000000000", + "candidate_incarnation": "11111111111111111111111111111111", + "previous_journal_id": null, + "pruned_through_journal_id": null, + "expected_activation_generation": 0, + "drain_attempt": 1, + "serve_admission_lease_bound_ms": 1000, + "promotion_not_before_unix_ms": 1700000001000, + "transition_kind": "config", + "displaced_active": { + "logical_root": "builtin", + "immutable_blob_id": "builtin", + "source_version": 0, + "data_hash": "0000000000000000000000000000000000000000000000000000000000000000", + "config_revision": "0000000000000000000000000000000000000000000000000000000000000000", + "policy_digest": "0000000000000000000000000000000000000000000000000000000000000000", + "ordinal": 0, + "model_epoch": "pre_epic_v1", + "minimum_binary_generation": 1, + "row_schema_floor": 1, + "activation_generation": 0 + }, + "activated_active": { + "logical_root": "app_config", + "immutable_blob_id": "app_config/1", + "source_version": 1, + "data_hash": "1111111111111111111111111111111111111111111111111111111111111111", + "config_revision": "2222222222222222222222222222222222222222222222222222222222222222", + "policy_digest": "3333333333333333333333333333333333333333333333333333333333333333", + "ordinal": 1, + "model_epoch": "pre_epic_v1", + "minimum_binary_generation": 1, + "row_schema_floor": 1, + "activation_generation": 1 + }, + "membership_epoch": 7, + "ready_members": ["edge-a", "edge-b"], + "quiesced_members": ["edge-a", "edge-b"], + "controller_id": "deploy-controller", + "retain_for_ms": 2592000000 + }, + "canonical_json_utf8": "{\"activated_active\":{\"activation_generation\":1,\"config_revision\":\"2222222222222222222222222222222222222222222222222222222222222222\",\"data_hash\":\"1111111111111111111111111111111111111111111111111111111111111111\",\"immutable_blob_id\":\"app_config/1\",\"logical_root\":\"app_config\",\"minimum_binary_generation\":1,\"model_epoch\":\"pre_epic_v1\",\"ordinal\":1,\"policy_digest\":\"3333333333333333333333333333333333333333333333333333333333333333\",\"row_schema_floor\":1,\"source_version\":1},\"attempt_id\":\"00000000000000000000000000000000\",\"candidate_incarnation\":\"11111111111111111111111111111111\",\"controller_id\":\"deploy-controller\",\"displaced_active\":{\"activation_generation\":0,\"config_revision\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"data_hash\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"immutable_blob_id\":\"builtin\",\"logical_root\":\"builtin\",\"minimum_binary_generation\":1,\"model_epoch\":\"pre_epic_v1\",\"ordinal\":0,\"policy_digest\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"row_schema_floor\":1,\"source_version\":0},\"drain_attempt\":1,\"expected_activation_generation\":0,\"membership_epoch\":7,\"previous_journal_id\":null,\"promotion_not_before_unix_ms\":1700000001000,\"pruned_through_journal_id\":null,\"quiesced_members\":[\"edge-a\",\"edge-b\"],\"ready_members\":[\"edge-a\",\"edge-b\"],\"retain_for_ms\":2592000000,\"schema_version\":1,\"serve_admission_lease_bound_ms\":1000,\"transition_kind\":\"config\"}", + "sha256_hex": "7af3934b4e5500903ef77ad8a1367db83b03fc62a0bb83bd0849289c685d2e88" + } + ], + "rejection_vectors": [ + { + "name": "unsafe-top-level-u64", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/expected_activation_generation", + "raw_json_number": "9007199254740992", + "error": "integer exceeds portable JCS profile" + }, + { + "name": "unsafe-embedded-active-u64", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/activated_active/source_version", + "raw_json_number": "9007199254740992", + "error": "integer exceeds portable JCS profile" + }, + { + "name": "fractional-journal-number", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/retain_for_ms", + "raw_json_number": "2592000000.5", + "error": "journal number is not an integer" + }, + { + "name": "unsafe-admission-lease-bound", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/serve_admission_lease_bound_ms", + "raw_json_number": "9007199254740992", + "error": "integer exceeds portable JCS profile" + }, + { + "name": "zero-promotion-admission-lease-bound", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/serve_admission_lease_bound_ms", + "raw_json_number": "0", + "error": "config/model admission lease bound is not positive" + } + ] +} +``` + + + ## 6. Blob config pipeline `trusted-server.toml` remains the human-authored source format. The deployed diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index b2c508262..aa523d8df 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -538,7 +538,7 @@ referrers are never in the security view. Every degree of freedom is closed: exactly — TS never accepts a different domain and never rewrites one scope into another. The explicit value cannot exceed the registrable domain, computed against the **vendored - Mozilla PSL snapshot** `docs/superpowers/specs/psl-snapshot-ref.md` — + Mozilla PSL snapshot in §4a.1** — ICANN + private sections, IDNA-mapped; IP-literal or single-label hosts fall back to host-only), path `/`; `Secure` mandatory; `SameSite` configurable `Lax` (default) / `Strict` / `None` @@ -674,8 +674,8 @@ referrers are never in the security view. Every degree of freedom is closed: - **Request-header pointers are a positive, enumerated allowlist with no default publisher-origin identifier exposure.** "Documented enrichment headers" is not enforceable; the registration - enumerates the exact names from the **checked-in allowlist file - `docs/superpowers/specs/datadome-header-allowlist.md`** — spec-pinned + enumerates the exact names from the **inline DataDome field contract in + §4a.2** — spec-pinned today to exactly **`X-DataDome-ClientID`**, admitted only when the operator explicitly sets `[integrations.datadome] expose_client_id_to_origin = true` (default @@ -701,7 +701,7 @@ referrers are never in the security view. Every degree of freedom is closed: drop-individually middle path (response `Pragma: no-cache` has no standardized meaning, RFC 9111 §5.4), and the batch-invalidation default for unlisted names — is exactly one cell of the matrix in - `datadome-header-allowlist.md`. The fail-open consequence of batch + §4a.2.3. The fail-open consequence of batch invalidation stays within sign-off 28's scope, and both documented vendor responses are verbatim fixtures at the matrix. - **Representation rules are decision-scoped and narrow.** A _Respond_ @@ -739,8 +739,8 @@ referrers are never in the security view. Every degree of freedom is closed: remains superseded). Exceeding size, first-byte, or total deadline fails the batch → Continue. - **One pointer contract, one place.** The single normative - decision × session-mode × pointer matrix lives in - **`datadome-header-allowlist.md`** — this spec's earlier inline + decision × session-mode × pointer matrix lives in **§4a.2.3** — this + spec's earlier inline decision-scoped list and outcome list are deleted in its favor (duplicated lists disagreed about `X-Set-Cookie`, `X-DataDome`, `X-DD-*`, and `Pragma`, letting one conforming implementation accept @@ -778,6 +778,291 @@ referrers are never in the security view. Every degree of freedom is closed: after Respond has short-circuited routing would leave nothing to fail open _to_. +### 4a.1 Public Suffix List snapshot (normative) + +The vendored Mozilla PSL revision used for registrable-domain +computation in §4a: the implementation PR vendors the list file +and records its upstream commit hash here. Rules: ICANN **and** private +sections apply; hostnames are IDNA-mapped before matching; IP literals +and single-label hosts have no registrable domain (cookie falls back to +host-only). Updating the snapshot is a reviewed spec change. + +| Field | Value | +| ---------------------- | -------------------------------------------------------------------- | +| Upstream repository | `publicsuffix/list` | +| Upstream commit | `e1b8015c3b2f0f4f8c18659c2480fc1a22c07b20` | +| Upstream source path | `public_suffix_list.dat` | +| Required vendored path | `crates/trusted-server-core/data/public_suffix_list.dat` | +| Required hash path | `crates/trusted-server-core/data/public_suffix_list.sha256` | +| Required provenance | `crates/trusted-server-core/data/public_suffix_list.provenance.json` | + +The implementation copies the source bytes at that commit without editing +and writes the lowercase 64-hex SHA-256 plus one trailing LF (no filename or +other fields) to the required hash path. CI verifies the bytes, hash, and +commit reference together; updating any one without the others fails. The +provenance file is canonical JSON with exactly +`{upstream_repository, upstream_commit_oid, upstream_commit_tree_oid, +source_path, source_blob_oid, source_sha256_hex}`. The vendoring PR description +quotes the same commit/tree/blob values and the independent command output +that verified the raw upstream SHA-256 and byte-for-byte vendored copy. A +commit OID without its tree and source-blob witness does not satisfy the +release gate. + +### 4a.2 DataDome field contract (normative) + +Adding or changing any name here is a reviewed spec change. This subsection +holds the **only** normative Protection API request-field and response-pointer +lists; §4a carries no duplicate or per-decision lists of its own. + +#### 4a.2.1 Protection API request fields (browser request → DataDome only) + +`SecurityUse` admits only the fields below to the configured DataDome +Protection API endpoint. They are request-scoped and are never persisted in +the identity graph, copied to publisher upstream or another integration, or +logged as raw values. + +The endpoint is the fixed core-owned +`https://api-fastly.datadome.co/validate-request`; “configured endpoint” in +this subsection means that DataDome protection is enabled, not that an operator may +supply an authority. Redirect following is disabled. No TS-controlled +advertising identifier, consent-store key, graph value, request query, or full +referrer is admitted. The normalized publisher URL path remains disclosed and +may itself contain publisher-chosen data; sign-offs 23/28 must classify that +surface, its retention, and DSR handling rather than calling the entire URL +identity-free. + +Core-derived fields: + +- `Key`, `IP`, `Method`, `Protocol`, `Host`, `ServerHostname`, `Request` +- `RequestModuleName`, `ModuleVersion`, `TimeRequest`, `Port` +- `ServerName`, `ServerRegion` +- `ClientID` from the single unambiguous `datadome` cookie only. The form key + is always present because the Protection API declares it mandatory; its + value is the empty string when no unambiguous cookie exists +- `CookiesLen`, `AuthorizationLen`, and `PostParamLen` as lengths only +- `HeadersList`, containing only the source header names admitted by the + next list plus `authorization`, `content-length`, and `cookie` (whose values + remain length-only/ClientID-only above). Names are lowercased, comma-separated, + and retain received field-line order, including repeated admitted names; + arbitrary/custom header names are excluded. An adapter that cannot preserve + received header order does not qualify this integration until DataDome + approves a canonical replacement order in sign-off 28 + +Core derives those fields identically on every qualified adapter: + +- `Key` is the resolved DataDome server secret and is never obtained from + request/config text; `IP` and `Port` are the remote address and TCP source + port from trusted connection metadata. Missing `Key`, `IP`, or `Port` skips + the call through the metered fail-open path; no sentinel is synthesized +- `Method` is the validated HTTP method token; `Protocol` is exactly `http` or + `https` from the adapter request URI; `Host` is the normalized ASCII request + authority with a non-default port retained; `ServerHostname` is trusted TLS + SNI/local-host metadata, omitted when unavailable +- `Request` is only the URL path. Empty path becomes `/`; dot segments are + removed, percent escapes are preserved without percent-decoding and + normalized to uppercase hex, and the complete query and fragment are + discarded before the security view exists +- `RequestModuleName` is the literal `trusted-server`; `ModuleVersion` is the + build's checked-in Trusted Server version; `TimeRequest` is the request-ingress + Unix timestamp in decimal microseconds, captured once before integration + processing and constrained to `0..=2^53-1` +- `ServerName` is the adapter-qualified deployment/service name and + `ServerRegion` is its adapter-qualified region code; either is omitted when + the platform cannot supply it without request input +- `CookiesLen`, `AuthorizationLen`, and `PostParamLen` are decimal byte counts + of the received field/body surfaces before redaction. Overflow beyond an + unsigned 64-bit count skips the call; it never wraps or truncates + +Exact request-header value mappings: + +- `Accept` ← `accept`; `AcceptCharset` ← `accept-charset`; + `AcceptEncoding` ← `accept-encoding`; `AcceptLanguage` ← `accept-language` +- `CacheControl` ← `cache-control`; `Connection` ← `connection`; + `ContentType` ← `content-type`; `From` ← `from`; `Origin` ← a successfully + parsed `origin` serialized as scheme + ASCII host + non-default port only; + `Pragma` ← `pragma`; `Referer` ← a successfully parsed `referer` reduced to + scheme + ASCII host + non-default port only; `UserAgent` ← `user-agent`; + `Via` ← `via` +- `SecCHDeviceMemory` ← `sec-ch-device-memory`; `SecCHUA` ← `sec-ch-ua`; + `SecCHUAArch` ← `sec-ch-ua-arch`; `SecCHUAFullVersionList` ← + `sec-ch-ua-full-version-list`; `SecCHUAMobile` ← `sec-ch-ua-mobile`; + `SecCHUAModel` ← `sec-ch-ua-model`; `SecCHUAPlatform` ← + `sec-ch-ua-platform` +- `SecFetchDest` ← `sec-fetch-dest`; `SecFetchMode` ← `sec-fetch-mode`; + `SecFetchSite` ← `sec-fetch-site`; `SecFetchStorageAccess` ← + `sec-fetch-storage-access`; `SecFetchUser` ← `sec-fetch-user` +- `X-Requested-With` ← `x-requested-with` + +Request-field multiplicity is normalized **before** parsing, truncation, and +form encoding, and adapters expose every received field line rather than a +preselected first/last value. Every admitted value line must contain valid HTTP +field-value octets **and** valid UTF-8 after OWS removal; otherwise the vendor +call is skipped through the metered fail-open path, because adapter-specific +byte-to-string replacement is forbidden: + +- The list-valued source fields are exactly `accept`, `accept-charset`, + `accept-encoding`, `accept-language`, `cache-control`, `connection`, + `pragma`, `via`, `sec-ch-ua`, and `sec-ch-ua-full-version-list`. Core removes + leading and trailing optional whitespace from each field value, rejects a + value containing invalid field-value octets, and combines all field lines + (including empty values) in received order with the two literal bytes `, `. + This one normalized value is then parsed where the mapping above requires + parsing and is then bounded. Commas inside an individual value are not split + and reserialized. +- Every other admitted value-bearing source header in the exact mapping above + is singleton. Zero lines means omit the DataDome field. Exactly one valid + line is OWS-normalized and processed. Two or more lines — even identical — + are ambiguous and skip the vendor call through the metered fail-open path; + core never chooses first, last, or comma-joined. In particular this applies + to `origin`, `referer`, `user-agent`, `content-type`, `from`, every remaining + `sec-ch-*`/`sec-fetch-*` field, and `x-requested-with`. +- `authorization` and `content-length` are security singletons for this view. + Repetition skips the vendor call before either length or `HeadersList` is + constructed. `AuthorizationLen` is the byte length of the one + OWS-normalized value. `PostParamLen` is always the byte length of the body + actually presented to core, not the numeric `content-length` value; a + malformed or body-inconsistent `content-length` is rejected by the shared + HTTP request boundary before integrations run. +- Multiple `cookie` field lines are permitted. Core OWS-normalizes them and + joins them in received order with the literal bytes `; ` for the shared RFC + cookie parser. `CookiesLen` is the byte length of that canonical joined + value. `ClientID` is populated only when the parsed result contains exactly + one syntactically valid `datadome` pair; malformed cookie syntax or duplicate + `datadome` pairs produces the required empty `ClientID` value without + exposing another cookie. The original cookie values never enter the vendor + payload. +- After successful normalization, `HeadersList` records the lowercased name of + every admitted received field line in original line order, so repeated list + fields and cookie lines remain repeated. A rejected request produces no + `HeadersList` and no vendor call. Per-field caps apply to the single + normalized value; the 24,576-byte cap applies after complete form encoding. + +For an optional mapped value, zero received lines omits both source and mapped +field; one or more lines whose OWS-normalized values are all empty omits the +mapped form field but retains each received source name in `HeadersList`. If at +least one list-valued line is nonempty, empty siblings remain represented in +the exact received-order `, ` join. Mandatory `ClientID` and the three length +fields follow their explicit rules instead of this optional-field omission. + +Adapter qualification fixtures feed the same ordered repeated-field corpus to +every host and assert byte-identical form fields, lengths, `HeadersList`, and +reject/omit outcomes. The corpus includes repeated list fields, identical and +different singleton duplicates, multiple cookies, duplicate `datadome` +cookies, empty values, invalid octets, and headers whose individual values +contain commas; invalid UTF-8 is a skip, never replacement decoding. + +`true-client-ip`, `x-forwarded-for`, and `x-real-ip` are not admitted in v1. +The trusted `IP` field already supplies connection provenance; copying raw +forwarding headers would let a client or unqualified proxy manufacture vendor +evidence. A future adapter-normalized forwarding chain requires a separately +named typed field and vendor sign-off, never reuse of the raw header mapping. + +Platform host evidence: + +- `TlsProtocol`, capped by TS at 32 bytes +- `JA4`, capped by TS at 128 bytes, only when the operator explicitly sets + `[integrations.datadome] expose_host_fingerprints_to_vendor = true`; + the default is `false`, omission is represented by absence rather than an + empty field, and startup logs the additional vendor disclosure +- `TlsCipher` is omitted in v1: DataDome defines it as the ordered list of + cipher suites offered by the client, while `RuntimeServices::client_info()` + exposes only the negotiated cipher. Substituting that value would silently + change the field's meaning +- `H2Fingerprint` is omitted in v1 because the current Protection API contract + does not define such a request field + +`X-DataDome-ClientID` is never a Protection API source in cookie-mode v1. +No wildcard (`Sec-CH-*`, `Sec-Fetch-*`, `X-*`, or otherwise) expands this +list. + +The following limits are bytes of the decoded field value before form +encoding. Truncation is UTF-8-boundary-safe. `XForwardedForIP` alone truncates +from the end; every other bounded field retains its prefix: + +- 8 bytes: `SecCHDeviceMemory`, `SecCHUAMobile`, + `SecFetchStorageAccess`, `SecFetchUser` +- 16 bytes: `SecCHUAArch` +- 32 bytes: `SecCHUAPlatform`, `SecFetchDest`, `SecFetchMode`, and the TS cap + on `TlsProtocol` +- 64 bytes: `ContentType`, `SecFetchSite`, and the TS cap on `ServerRegion` +- 128 bytes: `AcceptCharset`, `AcceptEncoding`, `CacheControl`, `Connection`, + `From`, `Pragma`, `SecCHUA`, `SecCHUAModel`, `X-Requested-With`, and the TS + cap on opt-in `JA4` +- 256 bytes: `AcceptLanguage`, `SecCHUAFullVersionList`, `Via` +- 512 bytes: `Accept`, `ClientID`, `HeadersList`, `Host`, `Origin`, + origin-only `Referer`, `ServerHostname`, and `ServerName` +- 768 bytes: `UserAgent` +- 2,048 bytes: path-only `Request` + +`Key`, `AuthorizationLen`, `CookiesLen`, `IP`, `Method`, `ModuleVersion`, +`Port`, `PostParamLen`, `Protocol`, `RequestModuleName`, and `TimeRequest` are +unbounded per-field by the vendor table but remain subject to the total bound. +The complete `application/x-www-form-urlencoded` body, including field names, +`=`/`&` separators, and percent-encoding expansion, must be at most **24,576 +bytes**. Core constructs and measures the whole payload before issuing the +request. It does not silently drop optional fields to fit: overflow skips the +vendor call and takes the same metered fail-open `Continue` path as a transport +failure. + +This is deliberately narrower than DataDome's currently documented required +surface: notably, it withholds `CookiesList` and omits empty source-header +fields. Product/vendor sign-off 28 therefore requires written confirmation +that this exact reduced profile is supported. Until that confirmation and +adapter conformance fixtures exist, the DataDome integration is not +release-qualified. + +#### 4a.2.2 Request-direction pointer (vendor response → publisher-upstream overlay) + +The complete set of vendor-response header pointers the security +channel (hook spec §4a) may copy into the owner-scoped +publisher-upstream overlay. Every `X-DataDome-*` name not listed here +is rejected. + +| Header | Direction | Scope | +| --------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `X-DataDome-ClientID` | response → upstream overlay | Disabled by default; admitted only with `expose_client_id_to_origin = true`. Owner-scoped publisher overlay only, never the shared request view or another integration | + +#### 4a.2.3 The single pointer matrix (normative — decision × session mode × pointer) + +This is the one authoritative browser-response contract. Session mode +is **cookie** in v1 (sessionByHeader is startup-rejected; a header-mode +column is added by the sign-off-23 opt-in, never implicitly). No +wildcard rows exist — every accepted name is enumerated, and **every +cell terminates in exactly one outcome**. + +| Pointer | Respond (cookie mode) | Continue (cookie mode) | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------- | +| `Set-Cookie` | typed `datadome` cookie operation (hook spec §4a parser; exactly one `datadome` field) | typed `datadome` cookie operation | +| `X-Set-Cookie` | **invalidate the batch → Continue** (mode mismatch: TS never requests header sessions in v1; a vendor response asserting one must not half-apply) | **invalidate the batch** → effects dropped | +| `Location` | forward on a 3xx Respond, replace; **on a non-3xx Respond → invalidate the batch → Continue** (a redirect field without a redirect status is a malformed decision) | **invalidate the batch** (effects dropped) | +| `Content-Type` | forward (owns its body), replace | **invalidate the batch** (effects dropped) | +| `Cache-Control` | restricted merge; invariant pass last | **invalidate the batch** (effects dropped) | +| `Pragma` | drop-individually, logged (response `Pragma` has no standardized meaning, RFC 9111 §5.4) | drop-individually, logged | +| `X-DataDome` | forward, owner-scoped typed telemetry | forward, owner-scoped typed telemetry | +| `X-DD-B` | forward as a browser-response security signal; never copy to publisher-upstream or another integration | forward as a browser-response security signal | +| anything not listed | invalidate the batch → Continue | invalidate → effects dropped | + +Singleton-field multiplicity (`Location`, `Content-Type`, `X-DataDome`, +`X-DD-B`, `X-Set-Cookie` appearing more than once) invalidates the +batch atomically; list-valued fields (`Cache-Control`, `Pragma`) join +per RFC 9110 §5.3 before their cell applies (hook spec §4a). + +`X-DD-B` is security-owned when DataDome is enabled. Before applying the fresh +security batch, core removes every pre-existing instance from the origin, +cached ordinary artifact, 304 metadata update, core response, or ordinary +mutator. A valid pointed vendor value then uses **replace-all** and the final +response cardinality must be exactly one; if the fresh vendor batch does not +point to it, final cardinality is zero. Append is never allowed. Fixtures cover +origin collision, cache-hit collision, 304 collision, repeated vendor fields, +and one valid fresh value, proving “exactly once” at final emission rather than +merely inside the vendor batch. + +**Fixtures**: DataDome's documented challenge response (`Set-Cookie`, +`Pragma`, `X-DataDome`, `Cache-Control`) asserts the decision stays +**Respond** with exactly the mapped fields; the documented allow example +(`Set-Cookie X-DD-B`) asserts Continue proceeds with the cookie applied +and `X-DD-B` forwarded exactly once — neither fixture may fail open. + ## 4. Done-when (from #782, sharpened) 1. Trait + builder + registry application, each public item documented. diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 660e966e1..5a03d3e59 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -838,8 +838,8 @@ section malformed-present (blocks grants, never withdraws). `US/TX` ↔ 16, `US/DE` ↔ 17, `US/IA` ↔ 18, `US/NE` ↔ 19, `US/NH` ↔ 20, `US/NJ` ↔ 21, `US/TN` ↔ 22, `US/MN` ↔ 23, `US/MD` ↔ 24, `US/IN` ↔ 25, `US/KY` ↔ 26, and `US/RI` ↔ 27. All accepted versions - and layouts are pinned by `gpp-registry-snapshot.md`; the current - snapshot accepts version 1 for sections 24–27 from the official IAB + and layouts are pinned by the inline GPP registry snapshot in §4.5.1; the + current snapshot accepts version 1 for sections 24–27 from the official IAB registry commit named there. A truncated map silently loses opt-outs, so every accepted section is an implementation prerequisite rather than an inert placeholder. **The current decoder is an explicit @@ -849,12 +849,12 @@ section malformed-present (blocks grants, never withdraws). versions the library happens to decode but the snapshot disallows. The implementation PR cross-checks this list against both the current decoder's section set and the official registry, and the accepted version per - section is **pinned to the vendored registry snapshot - `docs/superpowers/specs/gpp-registry-snapshot.md`** — a checked-in file enumerating, per mapped section, the + section is **pinned to the vendored registry snapshot in §4.5.1** — the + inline table enumerates, per mapped section, the accepted version(s), taken from the IAB registry at ratification (a date is not an immutable identifier, and "enumerated by the implementation PR" was two-implementations-diverge territory; the - vendored file is the single reproducible authority, and updating it + inline snapshot is the single reproducible authority, and updating it is a reviewed spec change); a mapped section carrying a version outside the pinned revision is treated as malformed-present (blocks grants, never withdraws — §4.4), @@ -931,6 +931,85 @@ mapped-version failure, and unknown-unmapped omission. inputs — current code consults only the sale field — and are declared as such in the migration matrix. +#### 4.5.1 GPP registry snapshot (normative, vendored) + +The pinned per-section accepted versions for the §4.5 map. This subsection is +the single reproducible authority; updating it is a +reviewed spec change. A mapped section presenting a version not listed +here is treated as malformed-present (§4.4). + +| GPP section ID | Section | Accepted version(s) | +| -------------- | --------------------------------------------------- | ------------------- | +| 6 | US Privacy string (uspv1, carried as a GPP section) | 1 | +| 7 | usnat | 1 | +| 8 | usca | 1 | +| 9 | usva | 1 | +| 10 | usco | 1 | +| 11 | usut | 1 | +| 12 | usct | 1 | +| 13 | usfl | 1 | +| 14 | usmt | 1 | +| 15 | usor | 1 | +| 16 | ustx | 1 | +| 17 | usde | 1 | +| 18 | usia | 1 | +| 19 | usne | 1 | +| 20 | usnh | 1 | +| 21 | usnj | 1 | +| 22 | ustn | 1 | +| 23 | usmn | 1 | +| 24 | usmd | 1 | +| 25 | usin | 1 | +| 26 | usky | 1 | +| 27 | usri | 1 | + +At the pinned commit below, the official section registry assigns IDs 24–27 +to MD, IN, KY, and RI and each named state specification defines accepted +version 1. That commit-backed statement, rather than an unverified publication +month, is the authority for admitting them. Treating them as national-only +would discard a state-specific choice. Unknown IDs outside the accepted table +still contribute nothing and are flagged for snapshot review. + +##### Provenance and vectors + +The immutable authority is the official +`InteractiveAdvertisingBureau/Global-Privacy-Platform` commit: + +`00ffaefe91513785e886c83877e9b56a4ec8e88c` + +Normative upstream paths for the newly admitted layouts are: + +- `Sections/US-States/MD/Maryland Privacy Technical Specification.md` +- `Sections/US-States/IN/Indiana Privacy Technical Specification.md` +- `Sections/US-States/KY/Kentucky Privacy Technical Specification.md` +- `Sections/US-States/RI/Rhode Island Privacy Technical Specification.md` +- `Sections/Section Information.md` + +The implementation vendors decoder fixtures under +`crates/trusted-server-core/testdata/gpp/00ffaefe91513785e886c83877e9b56a4ec8e88c/`. +That directory contains a `manifest.json` object with: + +- `upstream_commit_oid` and `upstream_commit_tree_oid`; +- a sorted `sources` array containing `{path, blob_oid, sha256_hex}` for all + five normative paths above — the four state specifications and + `Sections/Section Information.md`; and +- a sorted `cases` array whose entries are + `{section_id, version, case, encoded, expected}`. + +The vendoring PR description quotes the same commit/tree/blob values and the +independent command output used to verify every raw source SHA-256 and the +byte-for-byte copy. A commit OID without its tree and source-blob witnesses is +not accepted as completed provenance. `expected` uses the +permission spec's normalized P1/P4/GPC tokens, not decoder-library enums. +Fixture encodings must be constructed from the pinned bit layouts by an +independent generator or hand-checked vector, never emitted and consumed only +by the decoder under test. For every accepted section/version the corpus must +contain: minimum valid core-only string, core + GPC true, each mapped opt-out +value, each explicit not-opted-out value, explicit N/A, malformed/truncated +input, unsupported version, and a mixed known/unknown-section string. CI +rejects an update to this subsection unless the complete corpus for the new +commit is present. + ## 5. Jurisdiction resolution ### 5.1 Order @@ -1016,8 +1095,7 @@ numbers outside the exactly representable integer range (absolute value above 2^53 − 1) unless the field is string-typed, and `null` values (materialized defaults mean `null` never appears); negative zero serializes as JCS mandates. The machine-readable, -cross-language conformance fixtures are pinned in -`docs/superpowers/specs/policy-canonicalization-vectors.json`; every +cross-language conformance fixtures are pinned inline in §5.5.1; every runtime and the push tool must reproduce both the canonical UTF-8 bytes and digest, and must reject every rejection vector before activation. A digest difference is a startup failure, so canonicalization cannot be @@ -1361,7 +1439,7 @@ registration-order array of `{id, behavior_revision}`; config revision = array order is preserved because mutator order is behavior. The config form contains secret **references**, never resolved secret bytes, and excludes runtime observations. Both emit lowercase 64-hex digests and must reproduce -`docs/superpowers/specs/revision-canonicalization-vectors.json`. Tests cover a +the inline revision fixtures in §5.5.2. Tests cover a pre/post-model-CAS cache miss as well as concurrent push allocation, publish gaps, equal-version idempotence, equal-version digest mismatch, same-digest higher-version ordinal retention, stale restart, @@ -1376,6 +1454,153 @@ policy still cannot resurrect an identity withdrawn by a valid user signal; rollback itself follows the same staged activation protocol. +#### 5.5.1 Policy canonicalization vectors + +The JSON object between the stable markers is the sole normative +machine-readable policy fixture. Extractors exclude the markers and code +fences, parse the enclosed UTF-8 JSON, and must reject duplicate object keys. + + + +```json +{ + "schema_version": 1, + "canonicalization": "RFC 8785 (JCS)", + "digest": "SHA-256", + "domain_prefix_utf8": "tspol1|", + "vectors": [ + { + "name": "minimal-gdpr-policy", + "effective_policy": { + "rules": { + "default": "gdpr" + }, + "groups": { + "gdpr": { + "regime": "gdpr", + "default": "requires_signal" + } + } + }, + "canonical_json_utf8": "{\"groups\":{\"gdpr\":{\"default\":\"requires_signal\",\"regime\":\"gdpr\"}},\"rules\":{\"default\":\"gdpr\"}}", + "sha256_hex": "68f72cc004bd59df7f799be24685b85cbbf5d5fbd1ba3069c8bf51adf4a88e6b" + }, + { + "name": "us-country-floor-and-state-override", + "effective_policy": { + "rules": { + "default": "non-regulated", + "US/CA": { + "overrides": { + "select-personalised-ads": "requires_signal" + }, + "group": "us-opt-out" + }, + "US": "us-opt-out" + }, + "groups": { + "us-opt-out": { + "regime": "us-privacy", + "default": "requires_signal" + }, + "non-regulated": { + "regime": "none", + "default": "granted" + } + } + }, + "canonical_json_utf8": "{\"groups\":{\"non-regulated\":{\"default\":\"granted\",\"regime\":\"none\"},\"us-opt-out\":{\"default\":\"requires_signal\",\"regime\":\"us-privacy\"}},\"rules\":{\"US\":\"us-opt-out\",\"US/CA\":{\"group\":\"us-opt-out\",\"overrides\":{\"select-personalised-ads\":\"requires_signal\"}},\"default\":\"non-regulated\"}}", + "sha256_hex": "6c578c849c323936dc6d492449214c19e16e968b01a962c6ef99e9bbe3a08553" + }, + { + "name": "explicit-permission-map-without-default", + "effective_policy": { + "rules": { + "default": "explicit" + }, + "groups": { + "explicit": { + "regime": "none", + "permissions": { + "store-on-device": "granted", + "select-personalised-ads": "requires_signal" + } + } + } + }, + "canonical_json_utf8": "{\"groups\":{\"explicit\":{\"permissions\":{\"select-personalised-ads\":\"requires_signal\",\"store-on-device\":\"granted\"},\"regime\":\"none\"}},\"rules\":{\"default\":\"explicit\"}}", + "sha256_hex": "47745dbb0b5cf113e4d2eb9dda48e6fc9c6c8c18dc39ee715e9838edfd57727b" + } + ], + "rejection_vectors": [ + { + "name": "null-is-not-a-materialized-policy-value", + "raw_json": "{\"rules\":{\"default\":null}}", + "error": "null value" + } + ] +} +``` + + + +#### 5.5.2 Configuration-revision canonicalization vectors + +The same extraction rule applies to this sole normative registry/config +revision fixture. + + + +```json +{ + "schema_version": 1, + "canonicalization": "RFC 8785 (JCS)", + "digest": "SHA-256", + "vectors": [ + { + "name": "ordered-integration-registry", + "domain_prefix_utf8": "tsreg1|", + "normalized_value": [ + { + "behavior_revision": 1, + "id": "datadome" + }, + { + "behavior_revision": 2, + "id": "prebid" + } + ], + "canonical_json_utf8": "[{\"behavior_revision\":1,\"id\":\"datadome\"},{\"behavior_revision\":2,\"id\":\"prebid\"}]", + "sha256_hex": "a2a81a6727226821d87f885c92410b5ebd2466e1060a070e027ad0eba210eff4" + }, + { + "name": "effective-config-hash-grammar-smoke", + "domain_prefix_utf8": "tscfg1|", + "normalized_value": { + "integrations": { + "datadome": { + "secret_name": "datadome-api-key", + "enabled": true + } + } + }, + "canonical_json_utf8": "{\"integrations\":{\"datadome\":{\"enabled\":true,\"secret_name\":\"datadome-api-key\"}}}", + "sha256_hex": "83c85084578ee35ddba12418c6337b7cc064b7022be5c8ffed068e94b07118d6" + }, + { + "name": "config-sequence-binding", + "domain_prefix_utf8": "tscfgseq1|", + "push_sequence": 42, + "push_sequence_u64_be_hex": "000000000000002a", + "data_hash_hex": "0000000000000000000000000000000000000000000000000000000000000000", + "sha256_hex": "70edd94cd5728550a815355a1b719f4aafb466aa228571a4a7a6e88ad5178df0" + } + ] +} +``` + + + ## 6. Failure-mode matrix — normative | Condition | Resolution behavior | @@ -1458,8 +1683,8 @@ Consumers of the resolved set in this epic: minus EC.”** Whenever auction dispatch is allowed while `select-personalised-ads` is unset, core serializes a `ContextualAuctionView` constructed independently from the ordinary auction - object. The **sole normative v1 output schema** is the checked-in - `docs/superpowers/specs/contextual-openrtb-v1-allowlist.json`; descriptive + object. The **sole normative v1 output schema** is the inline manifest in + §7.1; descriptive prose cannot add a field. Its path language is dot-separated exact JSON member names, with `[]` denoting every element of the immediately preceding array. Object and array containers are implicit and exist only when at least @@ -1658,6 +1883,688 @@ Consumers of the resolved set in this epic: The client-cycle resolve endpoint (own spec, currently on hold) would be a further consumer if and when it proceeds. +### 7.1 Contextual OpenRTB v1 allowlist + +The JSON object between the stable markers is the sole normative +machine-readable contextual projection. Extractors exclude the markers and +code fences, parse the enclosed UTF-8 JSON, and must reject duplicate object +keys. Descriptive prose elsewhere cannot add a field or relax a constraint. + + + +```json +{ + "schema_version": 1, + "openrtb_version": "2.6", + "path_grammar": "dot-separated exact JSON member names; [] denotes each array element", + "default": "deny", + "unknown_or_unlisted_behavior": "serialization_error_no_dispatch", + "container_rules": { + "implicit_parents_only": true, + "omit_empty_optional_arrays": true, + "minimum_imp_elements": 1, + "site_app": "exactly_one", + "supported_imp_media": ["banner", "video"], + "each_imp_media": "exactly_one_of_banner_video", + "unsupported_imp_media_behavior": "serialization_error_no_dispatch" + }, + "cardinalities": { + "required_single": "present exactly once in its object", + "optional_single": "absent or present exactly once in its object", + "required_array": "present non-empty scalar array; every element matches the rule", + "optional_array": "absent or present non-empty scalar array; every element matches the rule", + "required_array_member": "present exactly once in every nearest enclosing object-array element", + "optional_array_member": "absent or present exactly once in every nearest enclosing object-array element" + }, + "cross_field_rules": { + "all_or_none": [ + ["regs.ext.gpp", "regs.ext.gpp_sid[]"], + ["imp[].banner.w", "imp[].banner.h"] + ], + "required_nonempty_object_arrays": [ + { + "when_parent_present": "source.ext.schain", + "array_path": "source.ext.schain.nodes[]" + } + ], + "at_least_one_complete_group": [ + { + "when_parent_present": "imp[].banner", + "groups": [ + ["imp[].banner.w", "imp[].banner.h"], + ["imp[].banner.format[]"] + ] + } + ] + }, + "derivations": { + "fresh_transaction": "fresh CSPRNG request value, never derived from request/user/security state", + "inventory": "validated publisher inventory configuration only", + "request_coarse": "typed coarse request value named by the rule", + "privacy": "permission resolver or admitted raw regulatory transport only", + "constant": "literal value named by the rule" + }, + "rules": [ + { + "path": "id", + "type": "string", + "cardinality": "required_single", + "derivation": "fresh_transaction" + }, + { + "path": "at", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "tmax", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "test", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "allimps", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "cur[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "bcat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "badv[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "wseat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + + { + "path": "source.fd", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "source.tid", + "type": "string", + "cardinality": "optional_single", + "derivation": "fresh_transaction" + }, + { + "path": "source.pchain", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "source.ext.schain.complete", + "type": "integer", + "cardinality": "required_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "source.ext.schain.ver", + "type": "string", + "cardinality": "required_single", + "derivation": "inventory" + }, + { + "path": "source.ext.schain.nodes[].asi", + "type": "string", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "source.ext.schain.nodes[].sid", + "type": "string", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "source.ext.schain.nodes[].hp", + "type": "integer", + "cardinality": "required_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "source.ext.schain.nodes[].rid", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "fresh_transaction" + }, + { + "path": "source.ext.schain.nodes[].name", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "source.ext.schain.nodes[].domain", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + + { + "path": "imp[].id", + "type": "string", + "cardinality": "required_array_member", + "derivation": "fresh_transaction" + }, + { + "path": "imp[].tagid", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].instl", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "imp[].bidfloor", + "type": "number", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].bidfloorcur", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].secure", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "imp[].exp", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + + { + "path": "imp[].banner.w", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].banner.h", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].banner.format[].w", + "type": "integer", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].banner.format[].h", + "type": "integer", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].banner.pos", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].banner.topframe", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "imp[].banner.btype[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].banner.battr[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].banner.mimes[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].banner.api[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + + { + "path": "imp[].video.mimes[]", + "type": "string", + "cardinality": "required_array", + "derivation": "inventory" + }, + { + "path": "imp[].video.minduration", + "type": "integer", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.maxduration", + "type": "integer", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.protocols[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].video.w", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.h", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.startdelay", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.placement", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.plcmt", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.linearity", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].video.skip", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "imp[].video.playbackmethod[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].video.api[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].video.battr[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].video.pos", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + + { + "path": "imp[].pmp.private_auction", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "imp[].pmp.deals[].id", + "type": "string", + "cardinality": "required_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].pmp.deals[].bidfloor", + "type": "number", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].pmp.deals[].bidfloorcur", + "type": "string", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].pmp.deals[].at", + "type": "integer", + "cardinality": "optional_array_member", + "derivation": "inventory" + }, + { + "path": "imp[].pmp.deals[].wseat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "imp[].pmp.deals[].wadomain[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + + { + "path": "site.id", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.name", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.domain", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.cat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "site.sectioncat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "site.pagecat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "site.mobile", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "site.privacypolicy", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "site.publisher.id", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.publisher.name", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.publisher.domain", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "site.content.cat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "site.content.language", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + + { + "path": "app.id", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.name", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.bundle", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.domain", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.ver", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.paid", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "app.cat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "app.sectioncat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "app.pagecat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "app.privacypolicy", + "type": "integer", + "cardinality": "optional_single", + "derivation": "inventory", + "allowed": [0, 1] + }, + { + "path": "app.publisher.id", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.publisher.name", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.publisher.domain", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + { + "path": "app.content.cat[]", + "type": "string", + "cardinality": "optional_array", + "derivation": "inventory" + }, + { + "path": "app.content.language", + "type": "string", + "cardinality": "optional_single", + "derivation": "inventory" + }, + + { + "path": "device.devicetype", + "type": "integer", + "cardinality": "optional_single", + "derivation": "request_coarse" + }, + { + "path": "device.os", + "type": "string", + "cardinality": "optional_single", + "derivation": "request_coarse" + }, + { + "path": "device.language", + "type": "string", + "cardinality": "optional_single", + "derivation": "request_coarse" + }, + { + "path": "device.lmt", + "type": "integer", + "cardinality": "required_single", + "derivation": "constant", + "constant": 1 + }, + { + "path": "device.geo.country", + "type": "string", + "cardinality": "optional_single", + "derivation": "request_coarse" + }, + + { + "path": "regs.coppa", + "type": "integer", + "cardinality": "optional_single", + "derivation": "privacy", + "allowed": [0, 1] + }, + { + "path": "regs.ext.gdpr", + "type": "integer", + "cardinality": "optional_single", + "derivation": "privacy", + "allowed": [0, 1] + }, + { + "path": "regs.ext.us_privacy", + "type": "string", + "cardinality": "optional_single", + "derivation": "privacy" + }, + { + "path": "regs.ext.gpp", + "type": "string", + "cardinality": "optional_single", + "derivation": "privacy" + }, + { + "path": "regs.ext.gpp_sid[]", + "type": "integer", + "cardinality": "optional_array", + "derivation": "privacy" + }, + { + "path": "user.ext.consent", + "type": "string", + "cardinality": "optional_single", + "derivation": "privacy" + } + ] +} +``` + + + ## 8. Testing strategy - **The decision matrix is the test plan.** Every row of §4.1 × each diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index 6f63cd949..b8084d6de 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -246,11 +246,11 @@ Requirements: selectable identity provider, so Fastly qualification (or an explicit decision to proceed without it) gates ratification — together with **vendoring the pinned PSL artifact** - (`psl-snapshot-ref.md` now records the upstream commit, but the required + (hook spec §4a.1 now records the upstream commit, but the required list bytes and checked hash are not yet present; the vendoring manifest/PR must record the upstream commit tree, source blob OID, vendored SHA-256, and byte-for-byte verification), **completing the pinned - GPP corpus and decoder** (`gpp-registry-snapshot.md` now records the + GPP corpus and decoder** (permission spec §4.5.1 now records the immutable official commit and accepted versions, but the per-section conformance corpus and complete decoder are still prerequisites; its manifest/PR must likewise record the commit tree and blob OID/SHA-256 for @@ -610,7 +610,7 @@ implemented. | 1 | Honor mapped use opt-outs globally. Destructive identity effects are limited to explicit storage withdrawal, authenticated deletion, or a qualifying live TCF Purpose 1 refusal. | permission §4, §4.2 | — | open | | 2 | GPP/USP sale opt-outs suppress P4 only; they neither revoke P1 nor delete the identity. | permission §4.5 | — | open | | 3 | Sharing/targeted-advertising opt-outs suppress P4; an explicit applicable not-opted-out value may grant P4. Neither affects P1. | permission §4.5 | — | open | -| 4 | US auction dispatch may continue while P4 is unset only through permission §7's positive `ContextualAuctionView` and its sole normative `contextual-openrtb-v1-allowlist.json` manifest. Unknown, unlisted, ill-typed, or untraceable leaves cause no dispatch; there is no client IP/UA, precise geo, page/referrer URL, user identifiers/data/segments, arbitrary extensions, or client forwarding headers. A separately authorized P1 identity may remain stored but cannot enter auction or partner egress. A destination that cannot consume the exact projection receives no request. | permission §7 | — | open | +| 4 | US auction dispatch may continue while P4 is unset only through permission §7's positive `ContextualAuctionView` and the sole normative inline manifest in permission §7.1. Unknown, unlisted, ill-typed, or untraceable leaves cause no dispatch; there is no client IP/UA, precise geo, page/referrer URL, user identifiers/data/segments, arbitrary extensions, or client forwarding headers. A separately authorized P1 identity may remain stored but cannot enter auction or partner egress. A destination that cannot consume the exact projection receives no request. | permission §7.1 | — | open | | 5 | Country-only and regionless US traffic use a protective country-wide `us-opt-out` floor; state rules may be stricter. | permission §3.4 | — | open | | 6 | Raw regulatory strings reach only the positively registered OpenRTB field that requires each source; all other destinations default deny. Identity rows retain normalized provenance/digests, not raw consent snapshots. | permission §7; providers §6.3 | — | open | | 7 | Reject legacy batch-sync traffic until live-browser provenance backfill makes the row re-evaluable. | rollout §6 item 6; permission §7 | — | open | @@ -634,10 +634,10 @@ implemented. | 25 | Enforce a stored-jurisdiction/provenance horizon for batch sync: moves into stricter regimes fail closed by the horizon, and moves out require a live visit. | permission §7 | — | open | | 26 | Aggregate embedded GPP GPC with `Sec-GPC` by OR as a global, non-destructive P4 use opt-out. | permission §4.5 | — | open | | 27 | In proxy mode, decode only mapped opt-out fields and derive no grants. | permission §4.4 | — | open | -| 28 | Require product and written vendor conformance approval for the reduced DataDome surface: fixed `api-fastly.datadome.co/validate-request` egress, path-only Request/origin-only Referer, trusted connection IP/port with raw forwarding headers omitted, exact repeated-header normalization and request-field byte limits (including omitted `CookiesList`, `TlsCipher`, and `H2Fingerprint`, and opt-in JA4), a 24,576-byte encoded request ceiling, bounded fixed-scope cookie lifecycle, CSP/challenge behavior, hardened headers, reserved security budget, and security-owned replace-all forwarding of documented `X-DD-B` exactly once. | hook §4a; `datadome-header-allowlist.md` | — | open | +| 28 | Require product and written vendor conformance approval for the reduced DataDome surface: fixed `api-fastly.datadome.co/validate-request` egress, path-only Request/origin-only Referer, trusted connection IP/port with raw forwarding headers omitted, exact repeated-header normalization and request-field byte limits (including omitted `CookiesList`, `TlsCipher`, and `H2Fingerprint`, and opt-in JA4), a 24,576-byte encoded request ceiling, bounded fixed-scope cookie lifecycle, CSP/challenge behavior, hardened headers, reserved security budget, and security-owned replace-all forwarding of documented `X-DD-B` exactly once. | hook §4a.2 | — | open | | 29 | Accept rowless roaming-cookie expiry as a bounded residual only with telemetry, an explicit maximum lifetime, operator documentation, and a removal/sunset criterion. | providers §5 | — | open | | 30 | Saturation blocks rowless admission for that prefix but never revokes an authenticated real row without its exact suffix; monitor NAT-cohort pressure. | providers §5 | — | open | | 31 | Keep replay history bounded by evicting expired/grant entries first and retaining restrictive state for its full horizon; saturation never shortens a later opt-out. | permission §4.3; providers wire schema | — | open | -| 32 | Accept official GPP sections 24–27 version 1, pin their layouts to the vendored IAB commit, and treat complete decoder/fixture support as a release prerequisite. The vendoring evidence records the commit tree and per-source blob OID/SHA-256 for all four state layouts plus `Section Information`; a commit string alone does not close the gate. | permission §4.5; `gpp-registry-snapshot.md` | — | open | +| 32 | Accept official GPP sections 24–27 version 1, pin their layouts to the vendored IAB commit, and treat complete decoder/fixture support as a release prerequisite. The vendoring evidence records the commit tree and per-source blob OID/SHA-256 for all four state layouts plus `Section Information`; a commit string alone does not close the gate. | permission §4.5.1 | — | open | | 33 | Treat any malformed or unsupported-version **mapped** GPP section as a global blocker for grants to the permissions its schema maps (P4 in v1), while still honoring decodable opt-outs elsewhere and never deriving withdrawal from malformed bytes; unknown unmapped section IDs remain non-contributing. | permission §4.5 | — | open | | 34 | Permit providers whose canonical identifiers cannot fit an injective 123-byte graph suffix to use the providers §2/§6.3 `sha256-detect` mode: 256-bit domain-separated collision resistance plus stored canonical-identifier comparison, fail-closed collision handling, no overwrite/join, and no cluster capability unless a literal prefix is independently preserved. | providers §2, §6.3 | — | open | diff --git a/docs/superpowers/specs/activation-journal-vectors.json b/docs/superpowers/specs/activation-journal-vectors.json deleted file mode 100644 index cb1df334d..000000000 --- a/docs/superpowers/specs/activation-journal-vectors.json +++ /dev/null @@ -1,98 +0,0 @@ -{ - "schema_version": 1, - "canonicalization": "RFC 8785 (JCS)", - "digest": "SHA-256", - "domain_prefix_utf8": "tsactj1|", - "numeric_profile": { - "type": "integer", - "minimum": 0, - "maximum": 9007199254740991 - }, - "vectors": [ - { - "name": "genesis-config-promotion", - "journal": { - "schema_version": 1, - "attempt_id": "00000000000000000000000000000000", - "candidate_incarnation": "11111111111111111111111111111111", - "previous_journal_id": null, - "pruned_through_journal_id": null, - "expected_activation_generation": 0, - "drain_attempt": 1, - "serve_admission_lease_bound_ms": 1000, - "promotion_not_before_unix_ms": 1700000001000, - "transition_kind": "config", - "displaced_active": { - "logical_root": "builtin", - "immutable_blob_id": "builtin", - "source_version": 0, - "data_hash": "0000000000000000000000000000000000000000000000000000000000000000", - "config_revision": "0000000000000000000000000000000000000000000000000000000000000000", - "policy_digest": "0000000000000000000000000000000000000000000000000000000000000000", - "ordinal": 0, - "model_epoch": "pre_epic_v1", - "minimum_binary_generation": 1, - "row_schema_floor": 1, - "activation_generation": 0 - }, - "activated_active": { - "logical_root": "app_config", - "immutable_blob_id": "app_config/1", - "source_version": 1, - "data_hash": "1111111111111111111111111111111111111111111111111111111111111111", - "config_revision": "2222222222222222222222222222222222222222222222222222222222222222", - "policy_digest": "3333333333333333333333333333333333333333333333333333333333333333", - "ordinal": 1, - "model_epoch": "pre_epic_v1", - "minimum_binary_generation": 1, - "row_schema_floor": 1, - "activation_generation": 1 - }, - "membership_epoch": 7, - "ready_members": ["edge-a", "edge-b"], - "quiesced_members": ["edge-a", "edge-b"], - "controller_id": "deploy-controller", - "retain_for_ms": 2592000000 - }, - "canonical_json_utf8": "{\"activated_active\":{\"activation_generation\":1,\"config_revision\":\"2222222222222222222222222222222222222222222222222222222222222222\",\"data_hash\":\"1111111111111111111111111111111111111111111111111111111111111111\",\"immutable_blob_id\":\"app_config/1\",\"logical_root\":\"app_config\",\"minimum_binary_generation\":1,\"model_epoch\":\"pre_epic_v1\",\"ordinal\":1,\"policy_digest\":\"3333333333333333333333333333333333333333333333333333333333333333\",\"row_schema_floor\":1,\"source_version\":1},\"attempt_id\":\"00000000000000000000000000000000\",\"candidate_incarnation\":\"11111111111111111111111111111111\",\"controller_id\":\"deploy-controller\",\"displaced_active\":{\"activation_generation\":0,\"config_revision\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"data_hash\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"immutable_blob_id\":\"builtin\",\"logical_root\":\"builtin\",\"minimum_binary_generation\":1,\"model_epoch\":\"pre_epic_v1\",\"ordinal\":0,\"policy_digest\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"row_schema_floor\":1,\"source_version\":0},\"drain_attempt\":1,\"expected_activation_generation\":0,\"membership_epoch\":7,\"previous_journal_id\":null,\"promotion_not_before_unix_ms\":1700000001000,\"pruned_through_journal_id\":null,\"quiesced_members\":[\"edge-a\",\"edge-b\"],\"ready_members\":[\"edge-a\",\"edge-b\"],\"retain_for_ms\":2592000000,\"schema_version\":1,\"serve_admission_lease_bound_ms\":1000,\"transition_kind\":\"config\"}", - "sha256_hex": "7af3934b4e5500903ef77ad8a1367db83b03fc62a0bb83bd0849289c685d2e88" - } - ], - "rejection_vectors": [ - { - "name": "unsafe-top-level-u64", - "base_vector": "genesis-config-promotion", - "replace_json_pointer": "/expected_activation_generation", - "raw_json_number": "9007199254740992", - "error": "integer exceeds portable JCS profile" - }, - { - "name": "unsafe-embedded-active-u64", - "base_vector": "genesis-config-promotion", - "replace_json_pointer": "/activated_active/source_version", - "raw_json_number": "9007199254740992", - "error": "integer exceeds portable JCS profile" - }, - { - "name": "fractional-journal-number", - "base_vector": "genesis-config-promotion", - "replace_json_pointer": "/retain_for_ms", - "raw_json_number": "2592000000.5", - "error": "journal number is not an integer" - }, - { - "name": "unsafe-admission-lease-bound", - "base_vector": "genesis-config-promotion", - "replace_json_pointer": "/serve_admission_lease_bound_ms", - "raw_json_number": "9007199254740992", - "error": "integer exceeds portable JCS profile" - }, - { - "name": "zero-promotion-admission-lease-bound", - "base_vector": "genesis-config-promotion", - "replace_json_pointer": "/serve_admission_lease_bound_ms", - "raw_json_number": "0", - "error": "config/model admission lease bound is not positive" - } - ] -} diff --git a/docs/superpowers/specs/contextual-openrtb-v1-allowlist.json b/docs/superpowers/specs/contextual-openrtb-v1-allowlist.json deleted file mode 100644 index 2707ff96a..000000000 --- a/docs/superpowers/specs/contextual-openrtb-v1-allowlist.json +++ /dev/null @@ -1,668 +0,0 @@ -{ - "schema_version": 1, - "openrtb_version": "2.6", - "path_grammar": "dot-separated exact JSON member names; [] denotes each array element", - "default": "deny", - "unknown_or_unlisted_behavior": "serialization_error_no_dispatch", - "container_rules": { - "implicit_parents_only": true, - "omit_empty_optional_arrays": true, - "minimum_imp_elements": 1, - "site_app": "exactly_one", - "supported_imp_media": ["banner", "video"], - "each_imp_media": "exactly_one_of_banner_video", - "unsupported_imp_media_behavior": "serialization_error_no_dispatch" - }, - "cardinalities": { - "required_single": "present exactly once in its object", - "optional_single": "absent or present exactly once in its object", - "required_array": "present non-empty scalar array; every element matches the rule", - "optional_array": "absent or present non-empty scalar array; every element matches the rule", - "required_array_member": "present exactly once in every nearest enclosing object-array element", - "optional_array_member": "absent or present exactly once in every nearest enclosing object-array element" - }, - "cross_field_rules": { - "all_or_none": [ - ["regs.ext.gpp", "regs.ext.gpp_sid[]"], - ["imp[].banner.w", "imp[].banner.h"] - ], - "required_nonempty_object_arrays": [ - { - "when_parent_present": "source.ext.schain", - "array_path": "source.ext.schain.nodes[]" - } - ], - "at_least_one_complete_group": [ - { - "when_parent_present": "imp[].banner", - "groups": [ - ["imp[].banner.w", "imp[].banner.h"], - ["imp[].banner.format[]"] - ] - } - ] - }, - "derivations": { - "fresh_transaction": "fresh CSPRNG request value, never derived from request/user/security state", - "inventory": "validated publisher inventory configuration only", - "request_coarse": "typed coarse request value named by the rule", - "privacy": "permission resolver or admitted raw regulatory transport only", - "constant": "literal value named by the rule" - }, - "rules": [ - { - "path": "id", - "type": "string", - "cardinality": "required_single", - "derivation": "fresh_transaction" - }, - { - "path": "at", - "type": "integer", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "tmax", - "type": "integer", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "test", - "type": "integer", - "cardinality": "optional_single", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "allimps", - "type": "integer", - "cardinality": "optional_single", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "cur[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "bcat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "badv[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "wseat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - - { - "path": "source.fd", - "type": "integer", - "cardinality": "optional_single", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "source.tid", - "type": "string", - "cardinality": "optional_single", - "derivation": "fresh_transaction" - }, - { - "path": "source.pchain", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "source.ext.schain.complete", - "type": "integer", - "cardinality": "required_single", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "source.ext.schain.ver", - "type": "string", - "cardinality": "required_single", - "derivation": "inventory" - }, - { - "path": "source.ext.schain.nodes[].asi", - "type": "string", - "cardinality": "required_array_member", - "derivation": "inventory" - }, - { - "path": "source.ext.schain.nodes[].sid", - "type": "string", - "cardinality": "required_array_member", - "derivation": "inventory" - }, - { - "path": "source.ext.schain.nodes[].hp", - "type": "integer", - "cardinality": "required_array_member", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "source.ext.schain.nodes[].rid", - "type": "string", - "cardinality": "optional_array_member", - "derivation": "fresh_transaction" - }, - { - "path": "source.ext.schain.nodes[].name", - "type": "string", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "source.ext.schain.nodes[].domain", - "type": "string", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - - { - "path": "imp[].id", - "type": "string", - "cardinality": "required_array_member", - "derivation": "fresh_transaction" - }, - { - "path": "imp[].tagid", - "type": "string", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].instl", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "imp[].bidfloor", - "type": "number", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].bidfloorcur", - "type": "string", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].secure", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "imp[].exp", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - - { - "path": "imp[].banner.w", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].banner.h", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].banner.format[].w", - "type": "integer", - "cardinality": "required_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].banner.format[].h", - "type": "integer", - "cardinality": "required_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].banner.pos", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].banner.topframe", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "imp[].banner.btype[]", - "type": "integer", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "imp[].banner.battr[]", - "type": "integer", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "imp[].banner.mimes[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "imp[].banner.api[]", - "type": "integer", - "cardinality": "optional_array", - "derivation": "inventory" - }, - - { - "path": "imp[].video.mimes[]", - "type": "string", - "cardinality": "required_array", - "derivation": "inventory" - }, - { - "path": "imp[].video.minduration", - "type": "integer", - "cardinality": "required_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].video.maxduration", - "type": "integer", - "cardinality": "required_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].video.protocols[]", - "type": "integer", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "imp[].video.w", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].video.h", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].video.startdelay", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].video.placement", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].video.plcmt", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].video.linearity", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].video.skip", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "imp[].video.playbackmethod[]", - "type": "integer", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "imp[].video.api[]", - "type": "integer", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "imp[].video.battr[]", - "type": "integer", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "imp[].video.pos", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - - { - "path": "imp[].pmp.private_auction", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "imp[].pmp.deals[].id", - "type": "string", - "cardinality": "required_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].pmp.deals[].bidfloor", - "type": "number", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].pmp.deals[].bidfloorcur", - "type": "string", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].pmp.deals[].at", - "type": "integer", - "cardinality": "optional_array_member", - "derivation": "inventory" - }, - { - "path": "imp[].pmp.deals[].wseat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "imp[].pmp.deals[].wadomain[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - - { - "path": "site.id", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "site.name", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "site.domain", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "site.cat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "site.sectioncat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "site.pagecat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "site.mobile", - "type": "integer", - "cardinality": "optional_single", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "site.privacypolicy", - "type": "integer", - "cardinality": "optional_single", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "site.publisher.id", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "site.publisher.name", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "site.publisher.domain", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "site.content.cat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "site.content.language", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - - { - "path": "app.id", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "app.name", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "app.bundle", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "app.domain", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "app.ver", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "app.paid", - "type": "integer", - "cardinality": "optional_single", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "app.cat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "app.sectioncat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "app.pagecat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "app.privacypolicy", - "type": "integer", - "cardinality": "optional_single", - "derivation": "inventory", - "allowed": [0, 1] - }, - { - "path": "app.publisher.id", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "app.publisher.name", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "app.publisher.domain", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - { - "path": "app.content.cat[]", - "type": "string", - "cardinality": "optional_array", - "derivation": "inventory" - }, - { - "path": "app.content.language", - "type": "string", - "cardinality": "optional_single", - "derivation": "inventory" - }, - - { - "path": "device.devicetype", - "type": "integer", - "cardinality": "optional_single", - "derivation": "request_coarse" - }, - { - "path": "device.os", - "type": "string", - "cardinality": "optional_single", - "derivation": "request_coarse" - }, - { - "path": "device.language", - "type": "string", - "cardinality": "optional_single", - "derivation": "request_coarse" - }, - { - "path": "device.lmt", - "type": "integer", - "cardinality": "required_single", - "derivation": "constant", - "constant": 1 - }, - { - "path": "device.geo.country", - "type": "string", - "cardinality": "optional_single", - "derivation": "request_coarse" - }, - - { - "path": "regs.coppa", - "type": "integer", - "cardinality": "optional_single", - "derivation": "privacy", - "allowed": [0, 1] - }, - { - "path": "regs.ext.gdpr", - "type": "integer", - "cardinality": "optional_single", - "derivation": "privacy", - "allowed": [0, 1] - }, - { - "path": "regs.ext.us_privacy", - "type": "string", - "cardinality": "optional_single", - "derivation": "privacy" - }, - { - "path": "regs.ext.gpp", - "type": "string", - "cardinality": "optional_single", - "derivation": "privacy" - }, - { - "path": "regs.ext.gpp_sid[]", - "type": "integer", - "cardinality": "optional_array", - "derivation": "privacy" - }, - { - "path": "user.ext.consent", - "type": "string", - "cardinality": "optional_single", - "derivation": "privacy" - } - ] -} diff --git a/docs/superpowers/specs/datadome-header-allowlist.md b/docs/superpowers/specs/datadome-header-allowlist.md deleted file mode 100644 index a6c5bf5c6..000000000 --- a/docs/superpowers/specs/datadome-header-allowlist.md +++ /dev/null @@ -1,255 +0,0 @@ -# DataDome field allowlists (normative, checked-in — vendor request and response directions) - -Adding or changing any name here is a reviewed commit to this file and -a spec change. This file holds the **only** normative Protection API -request-field and response-pointer lists; the hook spec §4a references it -and carries no duplicate or per-decision lists of its own. - -## Protection API request fields (browser request → DataDome only) - -`SecurityUse` admits only the fields below to the configured DataDome -Protection API endpoint. They are request-scoped and are never persisted in -the identity graph, copied to publisher upstream or another integration, or -logged as raw values. - -The endpoint is the fixed core-owned -`https://api-fastly.datadome.co/validate-request`; “configured endpoint” in -this file means that DataDome protection is enabled, not that an operator may -supply an authority. Redirect following is disabled. No TS-controlled -advertising identifier, consent-store key, graph value, request query, or full -referrer is admitted. The normalized publisher URL path remains disclosed and -may itself contain publisher-chosen data; sign-offs 23/28 must classify that -surface, its retention, and DSR handling rather than calling the entire URL -identity-free. - -Core-derived fields: - -- `Key`, `IP`, `Method`, `Protocol`, `Host`, `ServerHostname`, `Request` -- `RequestModuleName`, `ModuleVersion`, `TimeRequest`, `Port` -- `ServerName`, `ServerRegion` -- `ClientID` from the single unambiguous `datadome` cookie only. The form key - is always present because the Protection API declares it mandatory; its - value is the empty string when no unambiguous cookie exists -- `CookiesLen`, `AuthorizationLen`, and `PostParamLen` as lengths only -- `HeadersList`, containing only the source header names admitted by the - next list plus `authorization`, `content-length`, and `cookie` (whose values - remain length-only/ClientID-only above). Names are lowercased, comma-separated, - and retain received field-line order, including repeated admitted names; - arbitrary/custom header names are excluded. An adapter that cannot preserve - received header order does not qualify this integration until DataDome - approves a canonical replacement order in sign-off 28 - -Core derives those fields identically on every qualified adapter: - -- `Key` is the resolved DataDome server secret and is never obtained from - request/config text; `IP` and `Port` are the remote address and TCP source - port from trusted connection metadata. Missing `Key`, `IP`, or `Port` skips - the call through the metered fail-open path; no sentinel is synthesized -- `Method` is the validated HTTP method token; `Protocol` is exactly `http` or - `https` from the adapter request URI; `Host` is the normalized ASCII request - authority with a non-default port retained; `ServerHostname` is trusted TLS - SNI/local-host metadata, omitted when unavailable -- `Request` is only the URL path. Empty path becomes `/`; dot segments are - removed, percent escapes are preserved without percent-decoding and - normalized to uppercase hex, and the complete query and fragment are - discarded before the security view exists -- `RequestModuleName` is the literal `trusted-server`; `ModuleVersion` is the - build's checked-in Trusted Server version; `TimeRequest` is the request-ingress - Unix timestamp in decimal microseconds, captured once before integration - processing and constrained to `0..=2^53-1` -- `ServerName` is the adapter-qualified deployment/service name and - `ServerRegion` is its adapter-qualified region code; either is omitted when - the platform cannot supply it without request input -- `CookiesLen`, `AuthorizationLen`, and `PostParamLen` are decimal byte counts - of the received field/body surfaces before redaction. Overflow beyond an - unsigned 64-bit count skips the call; it never wraps or truncates - -Exact request-header value mappings: - -- `Accept` ← `accept`; `AcceptCharset` ← `accept-charset`; - `AcceptEncoding` ← `accept-encoding`; `AcceptLanguage` ← `accept-language` -- `CacheControl` ← `cache-control`; `Connection` ← `connection`; - `ContentType` ← `content-type`; `From` ← `from`; `Origin` ← a successfully - parsed `origin` serialized as scheme + ASCII host + non-default port only; - `Pragma` ← `pragma`; `Referer` ← a successfully parsed `referer` reduced to - scheme + ASCII host + non-default port only; `UserAgent` ← `user-agent`; - `Via` ← `via` -- `SecCHDeviceMemory` ← `sec-ch-device-memory`; `SecCHUA` ← `sec-ch-ua`; - `SecCHUAArch` ← `sec-ch-ua-arch`; `SecCHUAFullVersionList` ← - `sec-ch-ua-full-version-list`; `SecCHUAMobile` ← `sec-ch-ua-mobile`; - `SecCHUAModel` ← `sec-ch-ua-model`; `SecCHUAPlatform` ← - `sec-ch-ua-platform` -- `SecFetchDest` ← `sec-fetch-dest`; `SecFetchMode` ← `sec-fetch-mode`; - `SecFetchSite` ← `sec-fetch-site`; `SecFetchStorageAccess` ← - `sec-fetch-storage-access`; `SecFetchUser` ← `sec-fetch-user` -- `X-Requested-With` ← `x-requested-with` - -Request-field multiplicity is normalized **before** parsing, truncation, and -form encoding, and adapters expose every received field line rather than a -preselected first/last value. Every admitted value line must contain valid HTTP -field-value octets **and** valid UTF-8 after OWS removal; otherwise the vendor -call is skipped through the metered fail-open path, because adapter-specific -byte-to-string replacement is forbidden: - -- The list-valued source fields are exactly `accept`, `accept-charset`, - `accept-encoding`, `accept-language`, `cache-control`, `connection`, - `pragma`, `via`, `sec-ch-ua`, and `sec-ch-ua-full-version-list`. Core removes - leading and trailing optional whitespace from each field value, rejects a - value containing invalid field-value octets, and combines all field lines - (including empty values) in received order with the two literal bytes `, `. - This one normalized value is then parsed where the mapping above requires - parsing and is then bounded. Commas inside an individual value are not split - and reserialized. -- Every other admitted value-bearing source header in the exact mapping above - is singleton. Zero lines means omit the DataDome field. Exactly one valid - line is OWS-normalized and processed. Two or more lines — even identical — - are ambiguous and skip the vendor call through the metered fail-open path; - core never chooses first, last, or comma-joined. In particular this applies - to `origin`, `referer`, `user-agent`, `content-type`, `from`, every remaining - `sec-ch-*`/`sec-fetch-*` field, and `x-requested-with`. -- `authorization` and `content-length` are security singletons for this view. - Repetition skips the vendor call before either length or `HeadersList` is - constructed. `AuthorizationLen` is the byte length of the one - OWS-normalized value. `PostParamLen` is always the byte length of the body - actually presented to core, not the numeric `content-length` value; a - malformed or body-inconsistent `content-length` is rejected by the shared - HTTP request boundary before integrations run. -- Multiple `cookie` field lines are permitted. Core OWS-normalizes them and - joins them in received order with the literal bytes `; ` for the shared RFC - cookie parser. `CookiesLen` is the byte length of that canonical joined - value. `ClientID` is populated only when the parsed result contains exactly - one syntactically valid `datadome` pair; malformed cookie syntax or duplicate - `datadome` pairs produces the required empty `ClientID` value without - exposing another cookie. The original cookie values never enter the vendor - payload. -- After successful normalization, `HeadersList` records the lowercased name of - every admitted received field line in original line order, so repeated list - fields and cookie lines remain repeated. A rejected request produces no - `HeadersList` and no vendor call. Per-field caps apply to the single - normalized value; the 24,576-byte cap applies after complete form encoding. - -For an optional mapped value, zero received lines omits both source and mapped -field; one or more lines whose OWS-normalized values are all empty omits the -mapped form field but retains each received source name in `HeadersList`. If at -least one list-valued line is nonempty, empty siblings remain represented in -the exact received-order `, ` join. Mandatory `ClientID` and the three length -fields follow their explicit rules instead of this optional-field omission. - -Adapter qualification fixtures feed the same ordered repeated-field corpus to -every host and assert byte-identical form fields, lengths, `HeadersList`, and -reject/omit outcomes. The corpus includes repeated list fields, identical and -different singleton duplicates, multiple cookies, duplicate `datadome` -cookies, empty values, invalid octets, and headers whose individual values -contain commas; invalid UTF-8 is a skip, never replacement decoding. - -`true-client-ip`, `x-forwarded-for`, and `x-real-ip` are not admitted in v1. -The trusted `IP` field already supplies connection provenance; copying raw -forwarding headers would let a client or unqualified proxy manufacture vendor -evidence. A future adapter-normalized forwarding chain requires a separately -named typed field and vendor sign-off, never reuse of the raw header mapping. - -Platform host evidence: - -- `TlsProtocol`, capped by TS at 32 bytes -- `JA4`, capped by TS at 128 bytes, only when the operator explicitly sets - `[integrations.datadome] expose_host_fingerprints_to_vendor = true`; - the default is `false`, omission is represented by absence rather than an - empty field, and startup logs the additional vendor disclosure -- `TlsCipher` is omitted in v1: DataDome defines it as the ordered list of - cipher suites offered by the client, while `RuntimeServices::client_info()` - exposes only the negotiated cipher. Substituting that value would silently - change the field's meaning -- `H2Fingerprint` is omitted in v1 because the current Protection API contract - does not define such a request field - -`X-DataDome-ClientID` is never a Protection API source in cookie-mode v1. -No wildcard (`Sec-CH-*`, `Sec-Fetch-*`, `X-*`, or otherwise) expands this -list. - -The following limits are bytes of the decoded field value before form -encoding. Truncation is UTF-8-boundary-safe. `XForwardedForIP` alone truncates -from the end; every other bounded field retains its prefix: - -- 8 bytes: `SecCHDeviceMemory`, `SecCHUAMobile`, - `SecFetchStorageAccess`, `SecFetchUser` -- 16 bytes: `SecCHUAArch` -- 32 bytes: `SecCHUAPlatform`, `SecFetchDest`, `SecFetchMode`, and the TS cap - on `TlsProtocol` -- 64 bytes: `ContentType`, `SecFetchSite`, and the TS cap on `ServerRegion` -- 128 bytes: `AcceptCharset`, `AcceptEncoding`, `CacheControl`, `Connection`, - `From`, `Pragma`, `SecCHUA`, `SecCHUAModel`, `X-Requested-With`, and the TS - cap on opt-in `JA4` -- 256 bytes: `AcceptLanguage`, `SecCHUAFullVersionList`, `Via` -- 512 bytes: `Accept`, `ClientID`, `HeadersList`, `Host`, `Origin`, - origin-only `Referer`, `ServerHostname`, and `ServerName` -- 768 bytes: `UserAgent` -- 2,048 bytes: path-only `Request` - -`Key`, `AuthorizationLen`, `CookiesLen`, `IP`, `Method`, `ModuleVersion`, -`Port`, `PostParamLen`, `Protocol`, `RequestModuleName`, and `TimeRequest` are -unbounded per-field by the vendor table but remain subject to the total bound. -The complete `application/x-www-form-urlencoded` body, including field names, -`=`/`&` separators, and percent-encoding expansion, must be at most **24,576 -bytes**. Core constructs and measures the whole payload before issuing the -request. It does not silently drop optional fields to fit: overflow skips the -vendor call and takes the same metered fail-open `Continue` path as a transport -failure. - -This is deliberately narrower than DataDome's currently documented required -surface: notably, it withholds `CookiesList` and omits empty source-header -fields. Product/vendor sign-off 28 therefore requires written confirmation -that this exact reduced profile is supported. Until that confirmation and -adapter conformance fixtures exist, the DataDome integration is not -release-qualified. - -## Request-direction pointer (vendor response → publisher-upstream overlay) - -The complete set of vendor-response header pointers the security -channel (hook spec §4a) may copy into the owner-scoped -publisher-upstream overlay. Every `X-DataDome-*` name not listed here -is rejected. - -| Header | Direction | Scope | -| --------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `X-DataDome-ClientID` | response → upstream overlay | Disabled by default; admitted only with `expose_client_id_to_origin = true`. Owner-scoped publisher overlay only, never the shared request view or another integration | - -## The single pointer matrix (normative — decision × session mode × pointer) - -This is the one authoritative browser-response contract. Session mode -is **cookie** in v1 (sessionByHeader is startup-rejected; a header-mode -column is added by the sign-off-23 opt-in, never implicitly). No -wildcard rows exist — every accepted name is enumerated, and **every -cell terminates in exactly one outcome**. - -| Pointer | Respond (cookie mode) | Continue (cookie mode) | -| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------- | -| `Set-Cookie` | typed `datadome` cookie operation (hook spec §4a parser; exactly one `datadome` field) | typed `datadome` cookie operation | -| `X-Set-Cookie` | **invalidate the batch → Continue** (mode mismatch: TS never requests header sessions in v1; a vendor response asserting one must not half-apply) | **invalidate the batch** → effects dropped | -| `Location` | forward on a 3xx Respond, replace; **on a non-3xx Respond → invalidate the batch → Continue** (a redirect field without a redirect status is a malformed decision) | **invalidate the batch** (effects dropped) | -| `Content-Type` | forward (owns its body), replace | **invalidate the batch** (effects dropped) | -| `Cache-Control` | restricted merge; invariant pass last | **invalidate the batch** (effects dropped) | -| `Pragma` | drop-individually, logged (response `Pragma` has no standardized meaning, RFC 9111 §5.4) | drop-individually, logged | -| `X-DataDome` | forward, owner-scoped typed telemetry | forward, owner-scoped typed telemetry | -| `X-DD-B` | forward as a browser-response security signal; never copy to publisher-upstream or another integration | forward as a browser-response security signal | -| anything not listed | invalidate the batch → Continue | invalidate → effects dropped | - -Singleton-field multiplicity (`Location`, `Content-Type`, `X-DataDome`, -`X-DD-B`, `X-Set-Cookie` appearing more than once) invalidates the -batch atomically; list-valued fields (`Cache-Control`, `Pragma`) join -per RFC 9110 §5.3 before their cell applies (hook spec §4a). - -`X-DD-B` is security-owned when DataDome is enabled. Before applying the fresh -security batch, core removes every pre-existing instance from the origin, -cached ordinary artifact, 304 metadata update, core response, or ordinary -mutator. A valid pointed vendor value then uses **replace-all** and the final -response cardinality must be exactly one; if the fresh vendor batch does not -point to it, final cardinality is zero. Append is never allowed. Fixtures cover -origin collision, cache-hit collision, 304 collision, repeated vendor fields, -and one valid fresh value, proving “exactly once” at final emission rather than -merely inside the vendor batch. - -**Fixtures**: DataDome's documented challenge response (`Set-Cookie`, -`Pragma`, `X-DataDome`, `Cache-Control`) asserts the decision stays -**Respond** with exactly the mapped fields; the documented allow example -(`Set-Cookie X-DD-B`) asserts Continue proceeds with the cookie applied -and `X-DD-B` forwarded exactly once — neither fixture may fail open. diff --git a/docs/superpowers/specs/gpp-registry-snapshot.md b/docs/superpowers/specs/gpp-registry-snapshot.md deleted file mode 100644 index 88ed4c335..000000000 --- a/docs/superpowers/specs/gpp-registry-snapshot.md +++ /dev/null @@ -1,78 +0,0 @@ -# GPP registry snapshot (normative, vendored) - -The pinned per-section accepted versions for the permission spec's §4.5 -map. This file is the single reproducible authority; updating it is a -reviewed spec change. A mapped section presenting a version not listed -here is treated as malformed-present (permission spec §4.4). - -| GPP section ID | Section | Accepted version(s) | -| -------------- | --------------------------------------------------- | ------------------- | -| 6 | US Privacy string (uspv1, carried as a GPP section) | 1 | -| 7 | usnat | 1 | -| 8 | usca | 1 | -| 9 | usva | 1 | -| 10 | usco | 1 | -| 11 | usut | 1 | -| 12 | usct | 1 | -| 13 | usfl | 1 | -| 14 | usmt | 1 | -| 15 | usor | 1 | -| 16 | ustx | 1 | -| 17 | usde | 1 | -| 18 | usia | 1 | -| 19 | usne | 1 | -| 20 | usnh | 1 | -| 21 | usnj | 1 | -| 22 | ustn | 1 | -| 23 | usmn | 1 | -| 24 | usmd | 1 | -| 25 | usin | 1 | -| 26 | usky | 1 | -| 27 | usri | 1 | - -At the pinned commit below, the official section registry assigns IDs 24–27 -to MD, IN, KY, and RI and each named state specification defines accepted -version 1. That commit-backed statement, rather than an unverified publication -month, is the authority for admitting them. Treating them as national-only -would discard a state-specific choice. Unknown IDs outside the accepted table -still contribute nothing and are flagged for snapshot review. - -## Provenance and vectors - -The immutable authority is the official -`InteractiveAdvertisingBureau/Global-Privacy-Platform` commit: - -`00ffaefe91513785e886c83877e9b56a4ec8e88c` - -Normative upstream paths for the newly admitted layouts are: - -- `Sections/US-States/MD/Maryland Privacy Technical Specification.md` -- `Sections/US-States/IN/Indiana Privacy Technical Specification.md` -- `Sections/US-States/KY/Kentucky Privacy Technical Specification.md` -- `Sections/US-States/RI/Rhode Island Privacy Technical Specification.md` -- `Sections/Section Information.md` - -The implementation vendors decoder fixtures under -`crates/trusted-server-core/testdata/gpp/00ffaefe91513785e886c83877e9b56a4ec8e88c/`. -That directory contains a `manifest.json` object with: - -- `upstream_commit_oid` and `upstream_commit_tree_oid`; -- a sorted `sources` array containing `{path, blob_oid, sha256_hex}` for all - five normative paths above — the four state specifications and - `Sections/Section Information.md`; and -- a sorted `cases` array whose entries are - `{section_id, version, case, encoded, expected}`. - -The vendoring PR description quotes the same commit/tree/blob values and the -independent command output used to verify every raw source SHA-256 and the -byte-for-byte copy. A commit OID without its tree and source-blob witnesses is -not accepted as completed provenance. `expected` uses the -permission spec's normalized P1/P4/GPC tokens, not decoder-library enums. -Fixture encodings must be constructed from the pinned bit layouts by an -independent generator or hand-checked vector, never emitted and consumed only -by the decoder under test. For every accepted section/version the corpus must -contain: minimum valid core-only string, core + GPC true, each mapped opt-out -value, each explicit not-opted-out value, explicit N/A, malformed/truncated -input, unsupported version, and a mixed known/unknown-section string. CI -refuses to update this file unless the complete corpus for the new commit is -present. diff --git a/docs/superpowers/specs/policy-canonicalization-vectors.json b/docs/superpowers/specs/policy-canonicalization-vectors.json deleted file mode 100644 index 0df065de3..000000000 --- a/docs/superpowers/specs/policy-canonicalization-vectors.json +++ /dev/null @@ -1,77 +0,0 @@ -{ - "schema_version": 1, - "canonicalization": "RFC 8785 (JCS)", - "digest": "SHA-256", - "domain_prefix_utf8": "tspol1|", - "vectors": [ - { - "name": "minimal-gdpr-policy", - "effective_policy": { - "rules": { - "default": "gdpr" - }, - "groups": { - "gdpr": { - "regime": "gdpr", - "default": "requires_signal" - } - } - }, - "canonical_json_utf8": "{\"groups\":{\"gdpr\":{\"default\":\"requires_signal\",\"regime\":\"gdpr\"}},\"rules\":{\"default\":\"gdpr\"}}", - "sha256_hex": "68f72cc004bd59df7f799be24685b85cbbf5d5fbd1ba3069c8bf51adf4a88e6b" - }, - { - "name": "us-country-floor-and-state-override", - "effective_policy": { - "rules": { - "default": "non-regulated", - "US/CA": { - "overrides": { - "select-personalised-ads": "requires_signal" - }, - "group": "us-opt-out" - }, - "US": "us-opt-out" - }, - "groups": { - "us-opt-out": { - "regime": "us-privacy", - "default": "requires_signal" - }, - "non-regulated": { - "regime": "none", - "default": "granted" - } - } - }, - "canonical_json_utf8": "{\"groups\":{\"non-regulated\":{\"default\":\"granted\",\"regime\":\"none\"},\"us-opt-out\":{\"default\":\"requires_signal\",\"regime\":\"us-privacy\"}},\"rules\":{\"US\":\"us-opt-out\",\"US/CA\":{\"group\":\"us-opt-out\",\"overrides\":{\"select-personalised-ads\":\"requires_signal\"}},\"default\":\"non-regulated\"}}", - "sha256_hex": "6c578c849c323936dc6d492449214c19e16e968b01a962c6ef99e9bbe3a08553" - }, - { - "name": "explicit-permission-map-without-default", - "effective_policy": { - "rules": { - "default": "explicit" - }, - "groups": { - "explicit": { - "regime": "none", - "permissions": { - "store-on-device": "granted", - "select-personalised-ads": "requires_signal" - } - } - } - }, - "canonical_json_utf8": "{\"groups\":{\"explicit\":{\"permissions\":{\"select-personalised-ads\":\"requires_signal\",\"store-on-device\":\"granted\"},\"regime\":\"none\"}},\"rules\":{\"default\":\"explicit\"}}", - "sha256_hex": "47745dbb0b5cf113e4d2eb9dda48e6fc9c6c8c18dc39ee715e9838edfd57727b" - } - ], - "rejection_vectors": [ - { - "name": "null-is-not-a-materialized-policy-value", - "raw_json": "{\"rules\":{\"default\":null}}", - "error": "null value" - } - ] -} diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index b719340f1..cc72ee82a 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -229,10 +229,10 @@ instead of trusting prose. | P1 N+1 cannot run suppression recovery | fixed ("N+1 neither creates nor clears" authority-state; reads fail closed; clears wait for roll-forward — declared protective limitation) | | P1 row schema missing revision / stale provider-version | fixed (provenance-revision field row added with init/overflow/CAS rules; provider/version stripped from the mutable provenance row) | | P1 summary insufficient for policy-only rule | fixed (summary carries kind, grant basis/source class, policy revision, valid_until — absence decision reproducible from the strong record) | -| P1 DataDome allowlist not enumerated | fixed (checked-in `datadome-header-allowlist.md`, spec-pinned to `X-DataDome-ClientID` alone; other `X-DataDome-*` rejected) | +| P1 DataDome allowlist not enumerated | fixed (inline hook §4a.2 field contract, spec-pinned to `X-DataDome-ClientID` alone; other `X-DataDome-*` rejected) | | P1 cookie confinement misses upstream/log surfaces | fixed (exhaustive strip inventory: origin forwarding, proxy/click/Testlight upstreams, auction serialization, logs — each a tested row) | | P1 304 not implementable | fixed (persisted final post-hook header set re-emitted; absent metadata → cache miss) | -| P2 GPP snapshot missing | fixed (`gpp-registry-snapshot.md` vendored, sections 6–27, ratification re-verification note) | +| P2 GPP snapshot missing | fixed (inline permission §4.5.1 snapshot, sections 6–27, ratification re-verification note) | | P2 field registry not enumerated | fixed (v1 admitted set enumerated in-spec; growth is a spec change) | | P2 domain/lifetime irreproducible | fixed (PSL-computed registrable domain, vendored PSL revision, Max-Age ≤ 34,214,400 s) | | P2 skew window unvalued | fixed (normative 300 s constant with rationale) | @@ -663,7 +663,7 @@ current requirements. exposure by default, a required bounded cookie lifetime, reserved response budget, and exactly-once browser forwarding of documented `X-DD-B`. - GPP sections 24–27 are accepted at official version 1 and pinned to the IAB - repository commit in `gpp-registry-snapshot.md`; the PSL reference is also + repository commit in permission §4.5.1; the PSL reference in hook §4a.1 is also pinned. The generated GPP corpus and vendored PSL bytes remain release prerequisites, not facts already present in this worktree. @@ -788,7 +788,7 @@ at that revision were: `2ac2ab49922f261a8eecaee64f3621da8a2f2c1061c945defd7bc75ac2d5a569`, with three numeric rejection vectors. - Contextual auction output is governed by the sole machine-readable - `contextual-openrtb-v1-allowlist.json`: 98 unique exact leaf rules, closed + permission §7.1's inline `contextual-openrtb-v1` manifest: 98 unique exact leaf rules, closed type/cardinality/derivation vocabularies, executable container/cross-field constraints, atomic GPP/GPP-SID transport, nonempty supply chain, and exact banner/video shapes. Unknown, unlisted, ill-typed, or untraceable output diff --git a/docs/superpowers/specs/psl-snapshot-ref.md b/docs/superpowers/specs/psl-snapshot-ref.md deleted file mode 100644 index 7da249d49..000000000 --- a/docs/superpowers/specs/psl-snapshot-ref.md +++ /dev/null @@ -1,29 +0,0 @@ -# Public Suffix List snapshot reference (normative) - -The vendored Mozilla PSL revision used for registrable-domain -computation (hook spec §4a): the implementation PR vendors the list file -and records its upstream commit hash here. Rules: ICANN **and** private -sections apply; hostnames are IDNA-mapped before matching; IP literals -and single-label hosts have no registrable domain (cookie falls back to -host-only). Updating the snapshot is a reviewed spec change. - -| Field | Value | -| ---------------------- | -------------------------------------------------------------------- | -| Upstream repository | `publicsuffix/list` | -| Upstream commit | `e1b8015c3b2f0f4f8c18659c2480fc1a22c07b20` | -| Upstream source path | `public_suffix_list.dat` | -| Required vendored path | `crates/trusted-server-core/data/public_suffix_list.dat` | -| Required hash path | `crates/trusted-server-core/data/public_suffix_list.sha256` | -| Required provenance | `crates/trusted-server-core/data/public_suffix_list.provenance.json` | - -The implementation copies the source bytes at that commit without editing -and writes the lowercase 64-hex SHA-256 plus one trailing LF (no filename or -other fields) to the required hash path. CI verifies the bytes, hash, and -commit reference together; updating any one without the others fails. The -provenance file is canonical JSON with exactly -`{upstream_repository, upstream_commit_oid, upstream_commit_tree_oid, -source_path, source_blob_oid, source_sha256_hex}`. The vendoring PR description -quotes the same commit/tree/blob values and the independent command output -that verified the raw upstream SHA-256 and byte-for-byte vendored copy. A -commit OID without its tree and source-blob witness does not satisfy the -release gate. diff --git a/docs/superpowers/specs/revision-canonicalization-vectors.json b/docs/superpowers/specs/revision-canonicalization-vectors.json deleted file mode 100644 index 070d67d81..000000000 --- a/docs/superpowers/specs/revision-canonicalization-vectors.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "schema_version": 1, - "canonicalization": "RFC 8785 (JCS)", - "digest": "SHA-256", - "vectors": [ - { - "name": "ordered-integration-registry", - "domain_prefix_utf8": "tsreg1|", - "normalized_value": [ - { - "behavior_revision": 1, - "id": "datadome" - }, - { - "behavior_revision": 2, - "id": "prebid" - } - ], - "canonical_json_utf8": "[{\"behavior_revision\":1,\"id\":\"datadome\"},{\"behavior_revision\":2,\"id\":\"prebid\"}]", - "sha256_hex": "a2a81a6727226821d87f885c92410b5ebd2466e1060a070e027ad0eba210eff4" - }, - { - "name": "effective-config-hash-grammar-smoke", - "domain_prefix_utf8": "tscfg1|", - "normalized_value": { - "integrations": { - "datadome": { - "secret_name": "datadome-api-key", - "enabled": true - } - } - }, - "canonical_json_utf8": "{\"integrations\":{\"datadome\":{\"enabled\":true,\"secret_name\":\"datadome-api-key\"}}}", - "sha256_hex": "83c85084578ee35ddba12418c6337b7cc064b7022be5c8ffed068e94b07118d6" - }, - { - "name": "config-sequence-binding", - "domain_prefix_utf8": "tscfgseq1|", - "push_sequence": 42, - "push_sequence_u64_be_hex": "000000000000002a", - "data_hash_hex": "0000000000000000000000000000000000000000000000000000000000000000", - "sha256_hex": "70edd94cd5728550a815355a1b719f4aafb466aa228571a4a7a6e88ad5178df0" - } - ] -} From ffb63e181da45e1ec0db44017730d185470179e9 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Fri, 7 Aug 2026 09:54:19 -0700 Subject: [PATCH 24/24] docs: keep baseline specs unchanged Relocate PR-specific CLI activation and DataDome deltas into the new permission, migration, and hook specs. --- ...-datadome-server-side-protection-design.md | 382 +++++----------- ...2026-06-16-edgezero-based-ts-cli-design.md | 423 ++---------------- ...integration-response-header-hook-design.md | 89 +++- .../2026-07-30-permission-model-design.md | 219 ++++++++- ...07-30-provider-migration-rollout-design.md | 82 +++- docs/superpowers/specs/pr986-review-ledger.md | 26 ++ 6 files changed, 538 insertions(+), 683 deletions(-) diff --git a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md index 21375e9cd..f8d582a4f 100644 --- a/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md +++ b/docs/superpowers/specs/2026-06-11-datadome-server-side-protection-design.md @@ -1,24 +1,5 @@ # DataDome Server-Side Protection API Integration -> **Supersession note (PR #986):** the response-effects portions of this -> document — in particular "DataDome headers/cookies apply last and win" -> and any post-finalization ordering — are **superseded** by the -> response-header hook spec's §4a security-channel contract -> (`2026-07-30-integration-response-header-hook-design.md`): one global -> order applies (core finalization → ordinary mutators → security -> effects → final cache/privacy invariant pass, unconditionally last), -> with typed cookie/header operations, the enumerated field contract in -> §4a.2 of that spec, and owner-only identifier -> boundaries. Where this document conflicts, the hook spec governs. Additionally, this document's sessionByHeader requirement ("always -> send `X-DataDome-X-Set-Cookie` when the header ID is used") is -> **superseded for v1**: header-session mode is startup-rejected (hook -> spec §4a); TS never requests it and does not forward incoming header -> ClientIDs to the vendor. This document's generic "TLS/client metadata" -> instruction is also narrowed by §4a: DataDome may receive only explicitly -> enumerated, request-scoped security fields under `SecurityUse`; the device -> provider remains deferred, no fingerprint-derived classification is stored, -> and unlisted host evidence is omitted. - **Issue:** #317 **Date:** 2026-06-11 **Status:** In Progress @@ -73,10 +54,8 @@ JavaScript SDK. default. Default-exclude Trusted Server internal routes and static assets. 2. **Endpoint default:** default to DataDome's Fastly-specific Protection API endpoint from the official Fastly Compute docs, while allowing override. -3. **Header precedence (updated by PR #986):** apply DataDome through the - core-owned security channel after ordinary mutators, then run the final - cache/privacy invariant pass unconditionally last. Security effects do not - override framing, cache safety, or privacy invariants. +3. **Header precedence:** apply DataDome downstream headers last so DataDome + cookies/cache/challenge headers are not overwritten by generic finalization. 4. **GraphQL support:** defer. 5. **Client-side tag:** auto-inject when a client-side key is configured. 6. **Methods:** protect every non-`OPTIONS` method, including `HEAD`, when the @@ -85,9 +64,6 @@ JavaScript SDK. Store using configured store/name fields. Do not store the literal key in `trusted-server.toml`. 8. **Timeout:** use `1500ms` as the default Protection API timeout for v1. - _(Superseded: `1500 ms` is the **first-byte** bound only; the - complete-response deadline is 3000 ms with defined measurement - points — hook spec §4a.)_ 9. **Duplicate tag handling:** do not attempt automatic duplicate-tag detection in v1; operators can disable injection with `inject_client_side_tag = false`. @@ -187,81 +163,43 @@ pub trait IntegrationRequestFilter: Send + Sync { pub struct RequestFilterInput<'a> { pub settings: &'a Settings, pub services: &'a RuntimeServices, - /// The only request surface generic integrations receive. - pub request: &'a RedactedRequestView<'a>, + pub request: &'a Request, } pub enum RequestFilterDecision { - Continue(OrdinaryRequestFilterEffects), + Continue(RequestFilterEffects), Respond { response: Response, - effects: OrdinaryRequestFilterEffects, + effects: RequestFilterEffects, }, } -// Separate, core-only registration path. It is not a supertrait or optional -// field on IntegrationRequestFilter, so another integration cannot receive -// the DataDome capability through the generic runner. -#[async_trait(?Send)] -pub(crate) trait DataDomeSecurityRequestFilter: sealed::Sealed + Send + Sync { - async fn filter_datadome( - &self, - input: DataDomeSecurityFilterInput<'_>, - ) -> Result>; -} - -pub(crate) struct DataDomeSecurityFilterInput<'a> { - pub settings: &'a Settings, - pub services: &'a RuntimeServices, - pub request: &'a RedactedRequestView<'a>, - /// Constructed by core from the normative field allowlist. No raw - /// Request/header map or AuthorizedIdentity is exposed. - pub security: &'a DataDomeSecurityRequestView<'a>, +#[derive(Default)] +pub struct RequestFilterEffects { + pub request_headers: Vec, + pub response_headers: Vec, } -pub(crate) enum DataDomeSecurityDecision { - Continue(DataDomeSecurityEffects), - Respond { - response: Response, - effects: DataDomeSecurityEffects, - }, +pub struct HeaderMutation { + pub name: String, + pub value: String, + pub mode: HeaderMutationMode, } -#[derive(Default)] -pub(crate) struct DataDomeSecurityEffects { - pub upstream_overlay: Vec, - pub browser_effects: Vec, +pub enum HeaderMutationMode { + Set, + Append, } ``` -`RedactedRequestView`, `DataDomeSecurityRequestView`, the security trait, and -both security operation enums are core-owned sealed surfaces. Generic -integrations cannot construct or read them or recover the underlying raw -request. Core strips `ts-*`, EID/identity material, -`X-DataDome-ClientID`, and the `datadome` cookie before building the shared -view; the security view restores only the one typed cookie value and exact -request evidence admitted by the hook spec §4a.2.1. Another filter -receives only the shared redacted view and cannot inherit this owner -capability. This paragraph and the hook spec §4a replace every earlier generic -`&Request`/generic security-header-mutation sketch in this document. The -ordinary effects type remains subject to the hook's ordinary attributed-batch -registry and cannot express `SecurityUse`, owner overlay, cookies, or reserved -security names. - Important behavior: -- Ordinary filters run in registration order over the redacted view. DataDome's - security owner view is evaluated in its dedicated security position and is - never passed to the next filter. -- On `Continue`, allowlist-validated upstream operations enter only DataDome's - owner-scoped publisher overlay; they never mutate the shared request. -- Typed browser effects are accumulated and applied through the hook spec's - single pointer matrix and security budget. +- Filters run in registration order. +- On `Continue`, request header mutations are applied immediately before the + next filter and before route matching. +- Response header mutations are accumulated and applied to the final response. - On `Respond`, routing short-circuits with that response while preserving any downstream response header effects that must be applied after finalization. - _(Superseded: one global order applies — core finalization → ordinary - mutators → security effects → invariant pass unconditionally last; - nothing applies after the invariant pass — hook spec §4a.)_ - DataDome transport/API failures should not bubble out as registry errors; DataDome should convert them to `Continue(Default::default())` to preserve fail-open behavior. @@ -291,10 +229,8 @@ pub async fn filter_request( ) -> Result> ``` -The registry outcome should contain either an immediate response plus typed -security operations, or a continue decision with accumulated typed security -operations and an owner-scoped publisher overlay. Generic header name/value -mutations are not part of this API. +The registry outcome should contain either an immediate response plus response +header mutations, or a continue decision with accumulated response header mutations. ### 3. Fastly Route Hook @@ -305,13 +241,12 @@ In `route_request()`, run filters after normal basic auth succeeds and before ```text basic auth ok → integration_registry.filter_request(...) - → Respond { response, security_effects }: validate the complete security batch - → Continue(security_effects): apply only the owner-scoped upstream overlay; route normally + → Respond { response, effects }: finalize response, apply DataDome headers last, return + → Continue(effects): request is enriched; route normally; remember response effects → route matching → EC finalize -→ ordinary response mutators -→ validated security effects -→ final cache/privacy invariant pass (always last) +→ generic finalize_response +→ apply request-filter response headers last ``` Streaming publisher responses need the same treatment before headers are @@ -319,10 +254,8 @@ committed via `stream_to_client()`. ### 4. Header Mutation Semantics -DataDome pointer headers are internal instructions and are never forwarded. -The one normative field/pointer contract is the hook spec §4a.2, with the -publisher-upstream overlay in §4a.2.2 and the browser-response matrix in -§4a.2.3; a pointer does not authorize an unlisted name. +DataDome pointer headers are internal instructions and must not be forwarded. +Only headers named by the pointers should be copied. | Pointer header | Destination | | ---------------------------- | -------------------------------------------------- | @@ -331,14 +264,14 @@ publisher-upstream overlay in §4a.2.2 and the browser-response matrix in Rules: -- `datadome` cookie effects use the hook spec's typed cookie operation; raw - `Set-Cookie` is not a generic mutation. -- Every other admitted field follows its exact decision-matrix cell; there is - no generic set/replace default. +- `Set-Cookie` mutations use append mode. +- Other headers use set/replace mode. - Pointer headers themselves are never forwarded. -- Hop-by-hop, request-target, body-framing, credential, Trusted Server - internal, and unlisted headers invalidate the applicable batch. -- Security effects run before the final invariant pass, never after it. +- Header mutations must reject hop-by-hop, request-target, body framing, and + Trusted Server internal headers such as `Connection`, `Transfer-Encoding`, + `Content-Length`, `Host`, and `x-ts-*`. +- DataDome downstream headers are applied after `ec_finalize_response()` and + `finalize_response()`. ## DataDome Protection Design @@ -361,9 +294,8 @@ rewrite_sdk = true enable_protection = false server_side_key_secret_store = "ts_secrets" server_side_key_secret_name = "datadome_server_side_key" +protection_api_origin = "https://api-fastly.datadome.co" timeout_ms = 1500 -complete_response_timeout_ms = 3000 -challenge_body_max_bytes = 65536 protection_excluded_methods = ["OPTIONS"] protection_excluded_asns = [] protection_excluded_ip_cidrs = [] @@ -371,14 +303,6 @@ protection_excluded_ip_cidr_sources = [] protection_ip_list_cache_ttl_seconds = 300 enable_graphql_support = false -# Security identity/lifecycle. No default exists for max age: enabling -# protection without an explicit value is a startup error. -security_cookie_max_age = 2592000 # example: 30 days; allowed 7d..=365d -security_cookie_domain = "host-only" # or one exact normalized ASCII domain -security_cookie_same_site = "Lax" # Lax | Strict | None -expose_client_id_to_origin = false -expose_host_fingerprints_to_vendor = false - # New client-side tag injection layer client_side_key = "" inject_client_side_tag = true @@ -391,12 +315,6 @@ type = "path_regex" patterns = ["(?i)\\.(avi|flv|mka|mkv|mov|mp4|mpeg|mpg|mp3|flac|ogg|ogm|opus|wav|webm|webp|bmp|gif|ico|jpeg|jpg|png|svg|svgz|swf|eot|otf|ttf|woff|woff2|css|less|js|map)$"] ``` -This block is the canonical v1 DataDome configuration inventory; the hook and -allowlist specs reference it rather than defining another schema. Unknown -legacy security/session fields are errors, not ignored compatibility toggles. -In particular, `sessionByHeader`, `session_by_header`, and any equivalent are -startup-rejected in v1. - Notes: - The literal server-side key is not stored in Rust config. Rust config stores @@ -409,27 +327,8 @@ Notes: - `client_side_key` is optional. Auto-injection emits a tag only when `inject_client_side_tag = true` and `client_side_key` is non-empty; an empty key is a valid no-op. -- The v1 Protection API URL is the core-owned constant - `https://api-fastly.datadome.co/validate-request`. It is not operator - configurable. Supporting another DataDome region or a publisher proxy is a - new reviewed endpoint-registry entry and product/security decision, not a - free-form URL setting. Unknown legacy `protection_api_origin` fields are - startup errors so an old override cannot silently exfiltrate the server key. -- `complete_response_timeout_ms` defaults to and may not exceed 3000; - `challenge_body_max_bytes` defaults to and may not exceed 65,536. The hook - spec §4a owns the measurement/abort semantics. -- `security_cookie_max_age` is required when protection is enabled and must be - 604,800..=31,536,000 seconds. `security_cookie_domain` defaults to - `host-only`; an explicit domain must pass the hook spec's exact-domain, - domain-match, PSL, and active-scope-change checks. -- `security_cookie_same_site` accepts exactly `Lax`, `Strict`, or `None`; - `None` is valid only with the unconditionally emitted `Secure` attribute. - DataDome cookies never carry `HttpOnly`. -- Both exposure booleans default to `false`. ClientID-to-origin requires the - exact owner-overlay capability; host fingerprints require qualified JA4 - availability and sign-offs 23/28. A selected adapter that cannot preserve - admitted request-header field-line order or enforce the request/body limits - fails startup for protection rather than synthesizing different evidence. +- `protection_api_origin` remains configurable for regional/static endpoint + selection. - Static-asset exclusion is represented as a default typed `path_regex` rule and should remain case-insensitive so uppercase file extensions such as `.PNG` are skipped. @@ -515,21 +414,15 @@ Responsibilities: 1. Decide whether a request should be protected. 2. Build the form-encoded Protection API payload. -3. Send `POST https://api-fastly.datadome.co/validate-request` through platform - services. +3. Send `POST /validate-request` through platform services. 4. Classify the API response. 5. Extract pointer-header mutations. 6. Return a request-filter decision. Use platform abstractions for the outbound call: -- Construct the URL only from the core constant and assert at startup that its - scheme is `https`, host is exactly `api-fastly.datadome.co`, port is absent - (therefore 443), path is exactly `/validate-request`, and it has no userinfo, - query, or fragment. No request/config value participates in this URL. -- Build a `PlatformBackendSpec` with `first_byte_timeout = timeout_ms` and - automatic redirect following disabled. A 3xx response is returned to the - DataDome decision parser; it is never followed to a second authority. +- Parse `protection_api_origin` with `url`. +- Build a `PlatformBackendSpec` with `first_byte_timeout = timeout_ms`. - Resolve/register backend with `RuntimeServices::backend().ensure(...)`. - Send an `edgezero_core::http::Request` through `RuntimeServices::http_client().send(...)`. @@ -539,11 +432,10 @@ Request headers: ```text Content-Type: application/x-www-form-urlencoded Content-Length: -X-DataDome-X-Set-Cookie: true # only when X-DataDome-ClientID is used — SUPERSEDED for v1: never sent (hook spec §4a) +X-DataDome-X-Set-Cookie: true # only when X-DataDome-ClientID is used ``` -The exhaustive payload field set is the hook spec §4a.2.1. The list below is -informative and may not expand that normative allowlist: +Payload fields should include the core fields from DataDome's official module: - `Key` - `IP` @@ -551,7 +443,7 @@ informative and may not expand that normative allowlist: - `Protocol` - `Host` - `ServerHostname` -- `Request` as the normalized path only; query and fragment are never disclosed +- `Request` as path plus query - `RequestModuleName` - `ModuleVersion` - `TimeRequest` @@ -568,36 +460,32 @@ informative and may not expand that normative allowlist: - `Connection` - `Content-Type` - `From` - - `Origin` as parsed origin only + - `Origin` - `PostParamLen` - `Pragma` - - `Referer` as parsed origin only + - `Referer` - `User-Agent` - `Via` + - `X-Forwarded-For` + - `X-Real-IP` - `X-Requested-With` - - only the individually enumerated Sec-CH and Sec-Fetch fields in the - normative allowlist -- only the TLS/client metadata fields explicitly admitted by the normative - allowlist and sign-offs 23/28; JA4 egress additionally requires - `expose_host_fingerprints_to_vendor = true`, and availability alone is not - authorization. `TlsCipher` and `H2Fingerprint` are omitted in v1 for the - semantic reasons recorded in that allowlist - -In cookie-mode v1, `ClientID` comes only from a single unambiguous -`datadome` cookie. `X-DataDome-ClientID` is stripped from every shared -surface and is not forwarded to the vendor, so TS never sends -`X-DataDome-X-Set-Cookie: true`. + - Sec-CH and Sec-Fetch headers supported by the official module +- TLS/client metadata when available from `RuntimeServices::client_info()` + +`ClientID` source priority: + +1. `X-DataDome-ClientID` request header +2. `datadome` cookie + +When `X-DataDome-ClientID` is used, send +`X-DataDome-X-Set-Cookie: true` to the Protection API. Encoding and size rules: - URL-encode all values. -- Omit empty source-header fields; keep mandatory `ClientID` present with an - empty value when there is no unambiguous cookie. -- Apply the exact per-field decoded-byte limits in the normative allowlist - before encoding. -- Measure the complete form-encoded body and enforce the allowlist's 24,576-byte - ceiling before issuing the call; overflow takes metered fail-open and never - triggers ad hoc field dropping. +- Omit empty fields. +- Apply per-field truncation before encoding. +- Keep the global payload under DataDome's documented limit. ### Client Metadata @@ -613,7 +501,6 @@ that adapters can populate when available: ```rust pub struct ClientInfo { pub client_ip: Option, - pub client_port: Option, pub tls_protocol: Option, pub tls_cipher: Option, pub tls_ja4: Option, @@ -624,13 +511,8 @@ pub struct ClientInfo { ``` Fastly can populate `tls_ja4` and `h2_fingerprint` from the request APIs already -used by the JA4/debug device-signal code. Other adapters may leave those -optional fingerprint fields empty. `client_ip` and `client_port` are required -for a release-qualified Protection API call and come from the adapter's trusted -connection metadata, never a request header. If either is unavailable, the -adapter skips the vendor call through the metered fail-open path and remains -unqualified until vendor sign-off explicitly accepts a different profile; it -never invents port `0` or substitutes a forwarded header. +used by the JA4/debug device-signal code. Other adapters may leave these fields +empty. ### Protection API Response @@ -656,23 +538,23 @@ fail open and continue without effects. For challenge statuses: 1. Build a response using DataDome's API response status and body. -2. Validate the complete decision-scoped pointer batch against the hook spec - §4a.2.3 and the typed-cookie contract. -3. Apply the accepted security batch atomically. +2. Copy only headers listed in `X-DataDome-headers`. +3. Append `Set-Cookie` values. 4. Do not contact the publisher origin. -5. Run the final cache/privacy invariant pass after the security batch. +5. Still run Trusted Server response finalization, then apply DataDome headers + last. ### Allowed Requests For allow status `200`: -1. Apply only the owner-scoped publisher-upstream fields admitted by the hook - spec §4a.2.2 before route matching; the default is no - ClientID exposure. -2. Validate and retain the decision-scoped browser security batch. +1. Copy headers listed in `X-DataDome-request-headers` into the request before + Trusted Server route matching. +2. Accumulate headers listed in `X-DataDome-headers` for the final browser + response. 3. Continue normal route matching. -4. Apply ordinary response mutators, then the security batch, then the final - invariant pass. +4. Apply accumulated DataDome downstream headers after EC and generic response + finalization. ## Client-Side Auto-Injection @@ -715,24 +597,17 @@ Add: - `IntegrationRequestFilter` - `RequestFilterInput` - `RequestFilterDecision` -- `OrdinaryRequestFilterEffects` -- sealed `DataDomeSecurityRequestFilter`, `DataDomeSecurityFilterInput`, and - `DataDomeSecurityRequestView` -- typed `DataDomeSecurityDecision`, `DataDomeSecurityEffects`, - `DataDomeUpstreamOperation`, and `DataDomeBrowserOperation` -- separate ordinary-filter storage and one core-owned DataDome security slot in - `IntegrationRegistryInner` -- public builder method `with_request_filter` for ordinary filters; a - crate-private `with_datadome_security_filter` callable only by the built-in - DataDome registration path -- separate registry runners; the ordinary runner's input type cannot carry the - security view +- `RequestFilterEffects` +- `HeaderMutation` +- `HeaderMutationMode` +- request-filter storage in `IntegrationRegistryInner` +- builder method `with_request_filter` +- registry method to run filters - unit-test helpers for filters ### `crates/trusted-server-core/src/integrations/mod.rs` -Re-export only the ordinary request-filter types. The sealed DataDome security -trait, input/view, and operations remain crate-private. +Re-export the new request-filter types. ### `crates/trusted-server-core/src/integrations/datadome.rs` @@ -776,8 +651,7 @@ Populate new `ClientInfo` fields from Fastly request/environment when available. - Apply request header mutations before route matching. - Carry response header mutations through all non-streaming and streaming response paths. -- Apply DataDome/filter response effects through the hook spec's security - channel, followed by the invariant pass. +- Apply DataDome/filter response headers last. ### `trusted-server.toml` @@ -800,13 +674,11 @@ Update after implementation to describe: ### Registry Tests -- ordinary filters run in registration order over `RedactedRequestView` -- DataDome alone receives the sealed typed security view -- `Continue` applies only validated owner-overlay operations before publisher - origin; another filter never observes them -- `Respond` short-circuits later filters and discards ordinary batches under the - hook's security ordering -- generic operations cannot express reserved security names or cookies +- filter runs in registration order +- `Continue` applies request headers before next filter +- response header effects accumulate +- `Respond` short-circuits later filters +- append/set header modes behave correctly ### DataDome Config Tests @@ -814,8 +686,6 @@ Update after implementation to describe: - protection disabled does not require server-side key secret store/name fields - protection enabled requires non-empty server-side key secret store/name fields - protection fails open when the configured server-side key secret cannot be read -- legacy/free-form `protection_api_origin`, session-header, and unknown security - fields fail startup - invalid regex fails startup - injection disabled allows empty `client_side_key` - injection enabled with empty `client_side_key` emits no head insert and does @@ -835,30 +705,12 @@ Update after implementation to describe: ### Payload Tests - form encoding is correct -- empty source-header fields are omitted while mandatory `ClientID` remains - present as an empty value -- the outbound authority/path is exactly the core-owned HTTPS endpoint and 3xx - responses are never followed -- `Request` contains normalized path only, with query/fragment absent, and - `Referer` contains origin only -- `IP`/`Port` come only from trusted connection metadata; their absence skips - the call, and raw `true-client-ip`, `x-forwarded-for`, and `x-real-ip` values - never enter the payload or `HeadersList` -- `ClientID` comes only from a single unambiguous `datadome` cookie -- incoming `X-DataDome-ClientID` is stripped and - `X-DataDome-X-Set-Cookie` is never sent in cookie-mode v1 +- empty fields are omitted +- `ClientID` comes from `X-DataDome-ClientID` before cookie +- `X-DataDome-X-Set-Cookie` is sent when header-based ClientID is used - `datadome` cookie is parsed safely -- repeated list-valued source headers normalize in received field-line order - with literal `, ` separators, while repeated singleton, - `authorization`, or `content-length` headers skip the call without choosing - first/last; empty and comma-containing values match the normative allowlist -- multiple cookie field lines use the normative `; ` join for `CookiesLen` and - parsing; duplicate or malformed `datadome` pairs leave mandatory `ClientID` - empty and expose no other cookie value -- long fields are truncated according to the one normative allowlist -- the cross-adapter repeated-field corpus produces byte-identical normalized - form fields, lengths, `HeadersList`, and reject/omit outcomes; an adapter - without that capability cannot enable protection +- long fields are truncated according to configured limits +- request headers list is generated deterministically enough for tests ### Response Classification Tests @@ -869,8 +721,8 @@ Update after implementation to describe: - `5xx` fails open - pointer headers are not forwarded - request enriched headers are applied to allowed requests -- admitted security fields are applied atomically before final invariants -- the typed `datadome` cookie never overwrites another cookie name +- downstream headers are applied to final responses +- `Set-Cookie` appends instead of replacing ### Route Tests @@ -878,9 +730,8 @@ Update after implementation to describe: - auth challenge short-circuits before DataDome - DataDome challenge bypasses publisher origin - allowed DataDome response enriches request before publisher origin -- DataDome security batches apply to buffered responses before final invariants -- DataDome security batches and final invariants both complete before streaming - response headers commit +- DataDome downstream headers apply to buffered responses +- DataDome downstream headers apply before streaming response headers commit ## Acceptance Criteria @@ -897,12 +748,12 @@ passes. - [x] DataDome challenge responses return without contacting the origin. Covered by an adapter route test that returns the DataDome challenge response even with no publisher-origin fallback. -- [ ] Allowed-request enrichment conforms to the owner-scoped allowlist and - default-disabled ClientID exposure in the hook spec. -- [ ] Final responses use the hook spec's atomic security batch and final - invariant ordering on every adapter/path. -- [ ] The typed `datadome` cookie contract and complete response-pointer matrix - replace generic `Set-Cookie`/header mutation. +- [x] Allowed requests receive DataDome request-enrichment headers. Covered by a + registry test that applies DataDome-style request mutations before routing. +- [x] Final responses receive DataDome downstream headers/cookies. Covered by + adapter route tests for allowed and challenged responses. +- [x] `Set-Cookie` is appended, not coalesced or overwritten. Covered by pointer + header route tests for DataDome downstream cookies. - [x] Static assets and internal Trusted Server routes are excluded by default. Covered by adapter route tests for discovery and default static-extension exclusions. @@ -913,14 +764,7 @@ passes. - [x] GraphQL body parsing is not implemented in v1 and is clearly documented. - [x] Existing DataDome first-party proxy behavior remains unchanged. Existing DataDome proxy/rewrite tests pass as part of full workspace verification. -- [ ] `cargo fmt --all -- --check` and the repository's target-matched test and - lint aliases pass after implementation: `cargo test-fastly`, - `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin`, - `cargo clippy-fastly`, `cargo clippy-axum`, - `cargo clippy-cloudflare`, `cargo clippy-cloudflare-wasm`, - `cargo clippy-spin-native`, and `cargo clippy-spin-wasm`. Do not use bare - `cargo test --workspace` or workspace-wide all-feature clippy for this - multi-WASM-target repository. +- [x] `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets --all-features -- -D warnings`, and `cargo test --workspace` pass after implementation. Verified on 2026-06-15. ## Resolved Questions @@ -929,9 +773,7 @@ passes. methods, including `HEAD`, are eligible when the URL is otherwise in scope. 2. The DataDome server-side key is loaded from runtime Secret Store in v1. The config contains only the secret store and secret name. -3. The default Protection API timeout is `1500ms` for v1. _(Superseded: - first-byte bound only; 3000 ms complete-response deadline — hook - spec §4a.)_ +3. The default Protection API timeout is `1500ms` for v1. 4. Auto-injection does not attempt duplicate-tag detection in v1. The explicit `inject_client_side_tag = false` escape hatch is sufficient. @@ -939,16 +781,14 @@ passes. 1. **Timeout semantics:** `timeout_ms = 1500` is the v1 default and maps to the dynamic backend first-byte timeout. It is not a full end-to-end response-body - deadline in v1. _(The hook spec §4a now adds the 3000 ms - complete-response deadline on the monotonic clock with defined - measurement points; both bounds apply.)_ -2. **Client metadata scope:** only JA4 may be optionally admitted to the - form-encoded Protection API payload, never publisher origin, browser, graph, - another integration, or raw logs. Availability is not authorization: omit - it unless `expose_host_fingerprints_to_vendor = true`. `TlsCipher` is omitted - because the platform exposes a negotiated cipher while the vendor field - means ordered offered ciphers; `H2Fingerprint` is not a documented - Protection API field. Admit no host evidence outside the hook spec §4a.2.1. + deadline in v1. +2. **Client metadata scope:** JA4 and H2 fingerprint values are sent only in the + form-encoded Protection API payload to DataDome. They are not forwarded to + the publisher origin or returned to the browser unless DataDome independently + returns mapped enriched headers. Include them in v1 when the platform exposes + them because DataDome recommends TLS fingerprints and these signals are + useful for distinguishing browser and automation network stacks. Omit the + fields when unavailable. 3. **Challenge status source of truth:** follow the Protection API docs in v1: `301`, `302`, `401`, `403`, and `429` are challenge statuses when `X-DataDomeResponse` matches the HTTP status. diff --git a/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md b/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md index 2b8ab8f49..82ff2a755 100644 --- a/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md +++ b/docs/superpowers/specs/2026-06-16-edgezero-based-ts-cli-design.md @@ -20,7 +20,6 @@ The command surface is: ts config init ts config validate ts config push -ts config gc ts auth login --adapter ts auth status --adapter @@ -132,10 +131,8 @@ ids = ["secrets"] default = "secrets" ``` -The initial `ts config push` writes the immutable config object and stages its -settings candidate through the deployment-metadata capabilities specified in -§5. It does not write a secret-store entry. The `secrets` store is declared for -runtime/future use but is not written by this CLI spec. +The initial `ts config push` only writes config-store entries. The `secrets` +store is declared for runtime/future use but is not written by this CLI spec. Platform store names are not stored in `trusted-server.toml`. They are resolved by EdgeZero via its environment overlay, for example: @@ -147,331 +144,37 @@ EDGEZERO__STORES__SECRETS__SECRETS__NAME=publisher-a-ts-secrets ## 5. Runtime payload contract -`ts config push` publishes one logical Trusted Server app-config snapshot by -default. It does **not** publish flattened per-setting entries; each snapshot -is a new immutable versioned object as defined below. +`ts config push` writes a single logical Trusted Server app-config blob by +default. It does **not** publish flattened per-setting entries. -| Logical root | Value | +| Key | Value | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `app_config` by default, or `--key ` when supplied | Serialized `edgezero_core::blob_envelope::BlobEnvelope` whose `data` is the validated Trusted Server settings JSON | -Publication is versioned, not an overwrite of one live blob. The adapter maps -the logical identity `(root, push_sequence)` to an immutable physical object; -the object is written once, read back, and hash-verified. The strong -policy/config/model activation register names the sole active object. Runtimes never -treat the mutable logical root's latest value as active configuration. This -indirection is required even for adapters whose native config store exposes -only `put`: a globally unique, never-reused sequence gives every publication a -new object, while the deployment-metadata CAS supplies the active pointer. - The envelope contains: - a version field owned by EdgeZero; -- `push_sequence: u64` constrained to `0..=2^53-1`, scoped once per deployment/application across all - logical config-blob keys, allocated - exactly once from Trusted Server's linearizable deployment-metadata - config-sequence register before publication; - the validated app-config JSON data; - a SHA-256 hash over EdgeZero's canonical JSON form of `data`; -- a sequence-binding hash - `SHA-256("tscfgseq1|" || push_sequence.to_be_bytes() || data_hash_bytes)`, - where the domain tag is UTF-8, the integer is unsigned 64-bit big-endian, - and `data_hash_bytes` is the 32 decoded bytes of the preceding hash; the - known-answer vector is inline in the permission spec §5.5.2; - generation timestamp metadata. -Runtime loading must verify both the data hash and sequence-binding hash before -constructing `Settings`. -The sequence is metadata, not part of `data`, and is cryptographically bound -to that exact data by `sequence_binding_hash`; a future envelope signature -signs that binding hash rather than the data hash alone. Allocation may leave a gap if publication fails but -never reuses a value. Concurrent pushes CAS the sequence register; a loser -re-reads and retries. A rollback republishes old `data` with a new sequence. -Allocation at the portable maximum is a hard deployment error, never wrap or a -larger JSON integer. -An adapter without the deployment-metadata allocator rejects multi-instance -config/policy activation; ordinary config-store `put` is not treated as CAS. +Runtime loading must verify the envelope hash before constructing `Settings`. If an adapter must split a large envelope to satisfy platform limits, the entry -for that immutable logical identity may be an adapter-owned manifest that -identifies immutable chunks. The adapter/runtime loader must reconstruct and -verify the envelope before acknowledging readiness or exposing settings to -application code. A failed candidate or aborted activation leaves only an -unreferenced immutable object; a garbage collector may remove it after the -activation register's operational history and the permission spec §5.5 -time-based retention rules both permit removal, never while `active`, -`candidate`, operational history, or a retained activation-journal record -references it. Register eviction alone is never evidence that an object is old -enough to collect. The minimum journal/blob retention is 30 days and grows to -the longest processed-artifact, cookie-scope migration, rollback, or audit -horizon. Its not-before time uses the journal store's timestamp plus the -permission spec's 60-second promotion allowance; CLI/process wall time never -shortens it. `ts config gc` obtains the qualified journal inventory, computes the -reachable set, and refuses deletion if journal listing is incomplete or its -retention clock is uncertain. - -Before installing a candidate, the deployment controller obtains the -authoritative `{membership_epoch, members[]}` snapshot defined by the -permission spec §5.5. The CLI cannot synthesize, shrink, or override that set. -A membership change aborts and restages the candidate; `--force` never bypasses -unanimous readiness, the bounded serve-admission lease drain, the all-request -quiescence barrier, or serve admission. The candidate snapshots the positive -deployment-qualified `serve_admission_lease_bound_ms`. The draining CAS -records, from its qualified store clock at the CAS linearization point, a -`promotion_not_before_unix_ms`; that clock cannot satisfy the gate before the -full real interval has elapsed. The register rejects an early promotion even -if all member acknowledgments are present. Promotion is allowed only after the -controller has written and read-verified the immutable activation-journal -entry and the active-register CAS binds its ID as the new journal head. - -`ts config push` can stage and promote only a **settings candidate** and copies -the active model epoch, minimum binary generation, and row schema floor -unchanged. It cannot construct or promote a model candidate, and no `--force` -or config value can cross that boundary. The one-time -`pre_epic_v1` → `permissions_v2` transition is an authenticated deployment- -controller operation executed by the migration runbook: it stages the exact -model candidate, collects the fleet proof, and commits the single-register CAS -specified in permission §5.5. That controller operation is deliberately not a -general-purpose initial CLI command; exposing it later requires its own typed -command and cannot be emulated by raw config-store or metadata writes. -After that CAS, the same authenticated controller owns mirror completion: it -strong-reads active and `m00`; a missing or lower mirror is CAS-set to exactly -`active.row_schema_floor`, equality is an idempotent no-op, and an unreadable -mirror or failed CAS/read-verification remains closed for retry. The controller -then strong-reads and verifies exact equality before declaring the operation -complete. Retrying after a crash is idempotent. The operation never lowers -`m00` and never changes or authorizes active; a mirror higher than active is -rejected before any write as an inconsistency that fails closed for -investigation. - -### 5.1 Activation journal object and GC protocol - -The journal uses the same qualified immutable config-object service under the -reserved logical root `ts_activation_journal`, never the mutable app-config -root or the identity graph. Its logical object ID is lowercase -`SHA-256("tsactj1|" || RFC8785-JCS-UTF8(journal))`; adapters map -`("ts_activation_journal", object_id)` to a write-once physical object. The -object materializes exactly these fields and rejects unknown/missing fields: - -Every JSON number in the journal, including every number nested in an active -tuple, is an integer in `0..=9,007,199,254,740,991` (2^53 − 1). Booleans, -floats, negative values, and larger otherwise-valid `u64` values are rejected -before JCS; implementations may use wider internal integers but cannot emit -them here. Store-supplied lifecycle timestamps use the same portable range, -and addition that would exceed it fails closed. This profile makes the JCS -object ID identical in JavaScript, Rust, and every adapter rather than relying -on a language's larger integer type. - -- `schema_version = 1`; `attempt_id` as 32 lowercase hex characters from 16 - CSPRNG bytes, allowing a timed-out attempt to publish a new object; -- `candidate_incarnation` as the exact candidate's never-reused 32 lowercase - hex CSPRNG identity for `config`/`model`, or null for `checkpoint`; -- `previous_journal_id` and `pruned_through_journal_id`, each 64 lowercase hex - or null under the link/pruning rules below; -- `expected_activation_generation: u64` and `transition_kind` exactly - `config`, `model`, or `checkpoint`; -- `drain_attempt: u64`, which is the exact nonzero candidate drain attempt for - `config`/`model` and zero for `checkpoint`; -- `serve_admission_lease_bound_ms: u64`, the exact positive - deployment-qualified bound snapshotted by the candidate, and - `promotion_not_before_unix_ms: u64`, the exact store-clock gate written by - that drain attempt; both are zero only for a `checkpoint`; -- complete `displaced_active` and `activated_active` tuples from permission - §5.5, including settings bindings, policy identity, model epoch, minimum - binary generation, row schema floor, and logical activation generation; -- `membership_epoch: u64`, sorted unique `ready_members` and - `quiesced_members` using the stable member grammar, authenticated - `controller_id`, and `retain_for_ms: u64` constrained - to at least 2,592,000,000 and the longest applicable artifact, cookie-scope, - rollback, and audit horizon. - -The cross-language known-answer and rejection vectors are inline in §5.1.1; -every controller, -runtime verifier, and GC must reproduce both JCS bytes and object ID and reject -every numeric boundary vector. For the -first promotion, `previous_journal_id` is null only when the register head is -null and expected generation is zero. Every later config/model promotion must -name the exact current head and has null `pruned_through_journal_id`; the active -register CAS rejects any link/generation mismatch. For config/model entries, -`expected_activation_generation` must equal current active's logical -activation generation, and activated active must set it to that value + 1; -both member lists must equal the candidate snapshot's complete sorted member -list, `membership_epoch` must equal that snapshot's epoch, `drain_attempt` must -equal the candidate's current attempt and every quiescence acknowledgment, and -`candidate_incarnation` must equal every readiness/quiescence binding, -`serve_admission_lease_bound_ms` and `promotion_not_before_unix_ms` must equal -the candidate's exact drain fields, the admission-lease bound must be positive, -the immutable-store `created_at` for the journal must be at or after the -promotion-not-before time and no more than 60 seconds before the promotion CAS, -and the promotion CAS must independently enforce that its register store clock -has reached that time. These comparisons are defined only because the -activation register and immutable object service expose the same qualified, -authenticated Unix-millisecond time domain; adapters with incomparable clocks -fail activation qualification rather than comparing local timestamps. -Independently, `displaced_active` must equal current active and -`activated_active` must equal the candidate's computed post-CAS tuple. Overflow -is a hard error. A checkpoint -uses the current membership epoch, empty `ready_members` and -`quiesced_members` lists, and identical displaced and activated tuples -(including unchanged activation generation), with null -`candidate_incarnation`, `drain_attempt = 0`, -`serve_admission_lease_bound_ms = 0`, and -`promotion_not_before_unix_ms = 0`; it cannot stand in for fleet readiness or -quiescence. - -The immutable store returns authenticated `created_at_unix_ms` object metadata -from the shared qualified activation time domain and maintains a separate extend-only -`delete_not_before_unix_ms` lifecycle value. On every config or journal object -write, the adapter atomically initializes deletion protection to at least store -creation time + 30 days. For a promotion journal it extends protection for the -journal and both named blobs to at least `created_at + 60 seconds + -retain_for_ms` before the active CAS may bind the journal. These lifecycle -values can only increase. Therefore failed publication, aborted candidates, -losing journal attempts, and other unreferenced objects still have a store-clock -not-before value even though no successful promotion names them. - -The object service's qualification supplies snapshot-consistent complete -listing for both logical roots: a listing returns one snapshot generation and -opaque pagination token; every page is from that generation, and mutation or -expiry of the token forces GC to restart without deleting. GC first completes -the listing, traverses and verifies the journal from the active head, and builds -the active/candidate/history/journal reachable set. Missing objects, broken -hashes/links, unknown schema, cycles, incomplete pages, or uncertain lifecycle -metadata abort the run. Deletion then uses object-ID CAS and is allowed only -when the object is unreachable and its store-enforced not-before has passed. - -Journal pruning is an authenticated controller operation, never implicit GC. -Only when every record reachable from the current head is older than its full -retention horizon may the controller publish a `checkpoint` whose displaced -and activated tuples both equal current active, whose previous ID is null, and -whose `pruned_through_journal_id` is the old head. One register CAS verifies the -unchanged active tuple/generation and replaces only the journal head. The -checkpoint names and protects the current active blob; old journal objects -remain until their individual not-before values pass. Frequent activation can -therefore retain a longer chain but can never cut a still-required segment. +at the logical key may be an adapter-owned pointer that identifies chunks. The +adapter/runtime loader must reconstruct and verify the envelope before exposing +settings to application code. Reserved future keys, not written in this initial spec: | Key | Future purpose | | --------------------- | --------------------------------------------------------------------- | -| `ts-config-signature` | Optional signature/DSSE envelope over the sequence-binding hash | +| `ts-config-signature` | Optional signature/DSSE envelope over the blob hash | | `ts-config-metadata` | Optional JSON metadata: version, published_at, valid_until, policy_id | Request-signing public/private state is intentionally out of scope for this initial CLI. It will be revisited after EdgeZero exposes suitable secret-store write primitives. -#### 5.1.1 Activation-journal vectors - -The JSON object between the stable markers is the sole normative -machine-readable activation-journal fixture. Extractors exclude the markers -and code fences, parse the enclosed UTF-8 JSON, and must reject duplicate -object keys. - - - -```json -{ - "schema_version": 1, - "canonicalization": "RFC 8785 (JCS)", - "digest": "SHA-256", - "domain_prefix_utf8": "tsactj1|", - "numeric_profile": { - "type": "integer", - "minimum": 0, - "maximum": 9007199254740991 - }, - "vectors": [ - { - "name": "genesis-config-promotion", - "journal": { - "schema_version": 1, - "attempt_id": "00000000000000000000000000000000", - "candidate_incarnation": "11111111111111111111111111111111", - "previous_journal_id": null, - "pruned_through_journal_id": null, - "expected_activation_generation": 0, - "drain_attempt": 1, - "serve_admission_lease_bound_ms": 1000, - "promotion_not_before_unix_ms": 1700000001000, - "transition_kind": "config", - "displaced_active": { - "logical_root": "builtin", - "immutable_blob_id": "builtin", - "source_version": 0, - "data_hash": "0000000000000000000000000000000000000000000000000000000000000000", - "config_revision": "0000000000000000000000000000000000000000000000000000000000000000", - "policy_digest": "0000000000000000000000000000000000000000000000000000000000000000", - "ordinal": 0, - "model_epoch": "pre_epic_v1", - "minimum_binary_generation": 1, - "row_schema_floor": 1, - "activation_generation": 0 - }, - "activated_active": { - "logical_root": "app_config", - "immutable_blob_id": "app_config/1", - "source_version": 1, - "data_hash": "1111111111111111111111111111111111111111111111111111111111111111", - "config_revision": "2222222222222222222222222222222222222222222222222222222222222222", - "policy_digest": "3333333333333333333333333333333333333333333333333333333333333333", - "ordinal": 1, - "model_epoch": "pre_epic_v1", - "minimum_binary_generation": 1, - "row_schema_floor": 1, - "activation_generation": 1 - }, - "membership_epoch": 7, - "ready_members": ["edge-a", "edge-b"], - "quiesced_members": ["edge-a", "edge-b"], - "controller_id": "deploy-controller", - "retain_for_ms": 2592000000 - }, - "canonical_json_utf8": "{\"activated_active\":{\"activation_generation\":1,\"config_revision\":\"2222222222222222222222222222222222222222222222222222222222222222\",\"data_hash\":\"1111111111111111111111111111111111111111111111111111111111111111\",\"immutable_blob_id\":\"app_config/1\",\"logical_root\":\"app_config\",\"minimum_binary_generation\":1,\"model_epoch\":\"pre_epic_v1\",\"ordinal\":1,\"policy_digest\":\"3333333333333333333333333333333333333333333333333333333333333333\",\"row_schema_floor\":1,\"source_version\":1},\"attempt_id\":\"00000000000000000000000000000000\",\"candidate_incarnation\":\"11111111111111111111111111111111\",\"controller_id\":\"deploy-controller\",\"displaced_active\":{\"activation_generation\":0,\"config_revision\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"data_hash\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"immutable_blob_id\":\"builtin\",\"logical_root\":\"builtin\",\"minimum_binary_generation\":1,\"model_epoch\":\"pre_epic_v1\",\"ordinal\":0,\"policy_digest\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"row_schema_floor\":1,\"source_version\":0},\"drain_attempt\":1,\"expected_activation_generation\":0,\"membership_epoch\":7,\"previous_journal_id\":null,\"promotion_not_before_unix_ms\":1700000001000,\"pruned_through_journal_id\":null,\"quiesced_members\":[\"edge-a\",\"edge-b\"],\"ready_members\":[\"edge-a\",\"edge-b\"],\"retain_for_ms\":2592000000,\"schema_version\":1,\"serve_admission_lease_bound_ms\":1000,\"transition_kind\":\"config\"}", - "sha256_hex": "7af3934b4e5500903ef77ad8a1367db83b03fc62a0bb83bd0849289c685d2e88" - } - ], - "rejection_vectors": [ - { - "name": "unsafe-top-level-u64", - "base_vector": "genesis-config-promotion", - "replace_json_pointer": "/expected_activation_generation", - "raw_json_number": "9007199254740992", - "error": "integer exceeds portable JCS profile" - }, - { - "name": "unsafe-embedded-active-u64", - "base_vector": "genesis-config-promotion", - "replace_json_pointer": "/activated_active/source_version", - "raw_json_number": "9007199254740992", - "error": "integer exceeds portable JCS profile" - }, - { - "name": "fractional-journal-number", - "base_vector": "genesis-config-promotion", - "replace_json_pointer": "/retain_for_ms", - "raw_json_number": "2592000000.5", - "error": "journal number is not an integer" - }, - { - "name": "unsafe-admission-lease-bound", - "base_vector": "genesis-config-promotion", - "replace_json_pointer": "/serve_admission_lease_bound_ms", - "raw_json_number": "9007199254740992", - "error": "integer exceeds portable JCS profile" - }, - { - "name": "zero-promotion-admission-lease-bound", - "base_vector": "genesis-config-promotion", - "replace_json_pointer": "/serve_admission_lease_bound_ms", - "raw_json_number": "0", - "error": "config/model admission lease bound is not positive" - } - ] -} -``` - - - ## 6. Blob config pipeline `trusted-server.toml` remains the human-authored source format. The deployed @@ -619,74 +322,24 @@ Behavior: 1. Runs the same Trusted Server typed app-config validation as `ts config validate`. -2. Allocates `push_sequence` through the selected adapter's Trusted Server - deployment-metadata capability (dry-run reads and reports the next value - but does not reserve it). -3. Builds a `BlobEnvelope` from the validated app-config JSON and allocated - sequence. -4. Writes the envelope under the new immutable `(logical root, push_sequence)` - identity, reads it back, and verifies both hashes. It never - overwrites an object for an already allocated sequence. -5. CAS-installs the exact immutable object as the sole activation candidate; - the candidate has a new never-reused CSPRNG incarnation, binds current - active's complete tuple and logical activation generation, includes logical root, source version, data hash, - effective-config revision, policy digest, proposed policy ordinal, and the - unchanged active model fields. A competing or existing candidate makes the - CAS fail. The newly written unreferenced object is safe to collect later; it - never becomes live by being the most recently written blob. -6. Fleet readiness and controller promotion follow the permission spec §5.5. - Only promotion changes the active configuration. A config-only push still - takes this path but retains the policy ordinal. - -The underlying immutable-object read/diff/consent/dry-run/write behavior -delegates to EdgeZero's typed config push primitive using: - -- adapter from `--adapter`; -- manifest from `--manifest`; -- logical config store from `--store`; -- config entry key from `--key` or default; -- local mode from `--local`; -- dry-run mode from `--dry-run`; -- adapter runtime config from `--runtime-config`, when supplied. +2. Builds a `BlobEnvelope` from the validated app-config JSON. +3. Delegates read/diff/consent/dry-run/write behavior to EdgeZero's typed config + push primitive using: + - adapter from `--adapter`; + - manifest from `--manifest`; + - logical config store from `--store`; + - config entry key from `--key` or default; + - local mode from `--local`; + - dry-run mode from `--dry-run`; + - adapter runtime config from `--runtime-config`, when supplied. `--store` selects the logical config store for the Trusted Server config blob. `--key` selects the entry key within that config store. -`--dry-run` must not allocate a sequence, write an immutable object, or mutate -the activation register. It validates config, computes a provisional envelope -using the reported next sequence, resolves the EdgeZero push target, and -reports the immutable identity and candidate tuple that would be written. -Because another push may win, that sequence is explicitly advisory. Full -config values should not be printed by default. - -### 7.5 `ts config gc` - -```bash -ts config gc \ - --adapter \ - [--manifest ] \ - [--store ] \ - [--key ] \ - [--dry-run] \ - [--yes] \ - [--runtime-config ] -``` - -The command applies only §5.1's qualified immutable-object inventory and -deletion protocol; it never guesses physical keys or prunes the journal head. -It resolves the app-config root from `--key` (default `app_config`) and the -fixed `ts_activation_journal` root in the same selected config store, completes -one snapshot-consistent paginated inventory, verifies all hashes, lifecycle -metadata, active/candidate/history references, and the journal chain, then -computes unreachable objects whose store-enforced not-before has passed. -`--dry-run` prints only object IDs, roots, reasons, and lifecycle timestamps and -does not delete. Without `--dry-run`, deletion requires `--yes` or interactive -confirmation and uses the object-ID CAS from §5.1. Any uncertainty aborts the -entire run before the first delete; a partial platform deletion error stops the -run, reports exact completed IDs, and is safe to retry because reachability and -object-ID CAS are re-evaluated. Publishing a checkpoint is a separate -authenticated controller operation and is never an implicit side effect of -this command. +`--dry-run` must not mutate platform or local adapter state. It should still +validate config, compute the local envelope, resolve the EdgeZero push target, +and report what would be written. Full config values should not be printed by +default. ## 8. EdgeZero integration boundary @@ -697,9 +350,8 @@ There are two integration modes: 1. Pure lifecycle delegation for `ts auth`, `ts provision`, `ts serve`, `ts build`, and `ts deploy`. -2. Trusted Server config initialization/validation plus EdgeZero typed blob - push for `ts config validate` and `ts config push`, and the qualified - immutable-object inventory/CAS-delete path for `ts config gc`. +2. Trusted Server config initialization/validation plus EdgeZero typed blob push + for `ts config validate` and `ts config push`. Pure lifecycle delegate commands should call EdgeZero command/library APIs with the parsed CLI arguments and selected adapter. They should not perform Trusted @@ -707,9 +359,7 @@ Server config transformation, direct platform API calls, or adapter-specific command construction. `ts config push` is intentionally different: it validates Trusted Server app -config first, then delegates blob config-store writes to EdgeZero. `ts config -gc` delegates listing, lifecycle metadata, and object-ID CAS deletion but owns -the Trusted Server reachability/journal validation in §5.1. +config first, then delegates blob config-store writes to EdgeZero. Allowed implementation approach: @@ -851,24 +501,9 @@ contact real platforms in unit tests. `--dry-run`, `--no-env`, `--no-diff`, `--yes`, and `--runtime-config` to EdgeZero; - `--dry-run` performs no mutation; -- stages only a settings candidate bound to the complete active tuple and - activation generation; config push cannot alter model fields; - does not write secret-store entries; - does not print full config values by default. -### 13.6 `config gc` - -- complete snapshot pagination is required before the first delete; -- active, candidate, history, journal-chain, and protected-blob reachability - each prevent deletion; -- broken hash/link, cycle, expired pagination token, uncertain store clock, or - missing lifecycle metadata aborts with zero deletion; -- not-before boundary, dry-run, object-ID CAS conflict, partial-error retry, and - interactive/`--yes` confirmation follow §7.5; -- GC cannot publish a checkpoint or alter active/model state; -- the activation-journal known-answer vector verifies identically in runtime, - controller, and GC tests. - ## 14. Implementation sequencing 1. Update this spec and docs to the blob app-config contract. @@ -876,9 +511,7 @@ contact real platforms in unit tests. validation. 3. Collapse `crates/trusted-server-cli` to the thin downstream-CLI shape: direct EdgeZero args/run functions plus TS-owned `config init`. -4. Route `config validate` and `config push` through EdgeZero typed blob APIs; - add the qualified listing/lifecycle/object-CAS surface required by `config -gc` without platform-specific logic in Trusted Server. +4. Route `config validate` and `config push` through EdgeZero typed blob APIs. 5. Keep `edgezero_enabled` in `trusted_server_config` and restore any accidental coupling to `app_config`. 6. Keep runtime blob loading verified and avoid Trusted Server-owned platform diff --git a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md index aa523d8df..5003163bc 100644 --- a/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md +++ b/docs/superpowers/specs/2026-07-30-integration-response-header-hook-design.md @@ -3,7 +3,8 @@ **Status:** Draft **Author:** Engineering **Issue references:** #782 -**Related specs:** `2026-07-30-pluggable-providers-design.md` +**Related specs:** `2026-07-30-pluggable-providers-design.md`; baseline +DataDome design `2026-06-11-datadome-server-side-protection-design.md` **Last updated:** 2026-07-31 > **Context.** Issue #782 already specifies this feature well; its done-when @@ -18,6 +19,14 @@ ## 1. Overview +The pre-existing DataDome design remains unchanged. For implementations +governed by this spec, §4a is the normative PR-specific delta and supersedes +that baseline wherever its generic request/effect API, header ordering, +session-by-header behavior, endpoint/request surface, cookie lifecycle, +challenge transport, or response-pointer behavior conflicts. The baseline +continues to provide historical context; it is not a second normative source +for those surfaces. + Integrations can today rewrite request-path behavior (proxies, attribute rewriters, head injectors) but cannot mutate **response** headers. The hook adds that: an integration registers a response-header mutator via its @@ -587,7 +596,7 @@ referrers are never in the security view. Every degree of freedom is closed: and `X-DD-B`, and the higher-priority header session would never see a cookie update; the older DataDome spec's "always send X-DataDome-X-Set-Cookie when the header ID is used" is superseded for - v1 by its banner). Supporting header mode later means the full vendor + v1 by this section). Supporting header mode later means the full vendor protocol — typed owner-scoped `X-Set-Cookie`/`X-DD-B` forwarding, CORS exposure, and a JavaScript/local-storage identifier observer — as an explicit opt-in under **sign-off 23**. Every `ts-*` @@ -680,7 +689,7 @@ referrers are never in the security view. Every degree of freedom is closed: operator explicitly sets `[integrations.datadome] expose_client_id_to_origin = true` (default `false`); every other `X-DataDome-*` - field is rejected until a reviewed commit adds it to that file + field is rejected until a reviewed commit adds it to §4a.2 ("documented enrichment set, listed one by one" without an actual list was a wildcard whose contents could change outside the spec) — resolving what was a contradiction. When the opt-in is false, the @@ -712,7 +721,7 @@ referrers are never in the security view. Every degree of freedom is closed: allowlist does not admit those fields, and ambiguity here decides whether a challenge enforces or silently fails open (batch rejection → Continue). If the vendor ever requires more, it arrives as a reviewed - allowlist-file addition. A _Continue_ decision may not touch + §4a.2 field-contract addition. A _Continue_ decision may not touch representation metadata of publisher bytes. - **Respond transport is bounded, with exact measurement points.** The challenge body has a maximum size (64 KiB) and a **complete-response @@ -765,9 +774,10 @@ referrers are never in the security view. Every degree of freedom is closed: mutations comes from its position, not a "wins" rule; nothing outranks the invariant pass, or a challenge could combine `Set-Cookie` with public caching. The older DataDome spec's "applies last, after - finalization" wording is **superseded by this order** — updating that - document is a done-when item, since as written it would place DataDome - after the invariant pass and reopen the public-cache-plus-cookie bug. + finalization" wording is **superseded by this order**. That baseline remains + unchanged; this PR-specific section prevents an implementation from placing + DataDome after the invariant pass and reopening the + public-cache-plus-cookie bug. - The channel adopts the shared layers: structured attributed batches (§3, atomic per batch — a 302 must never lose `Location` to a budget while keeping its cookie), reserved header names, budgets, and the @@ -1063,13 +1073,62 @@ merely inside the vendor batch. (`Set-Cookie X-DD-B`) asserts Continue proceeds with the cookie applied and `X-DD-B` forwarded exactly once — neither fixture may fail open. +### 4a.3 DataDome configuration delta (normative) + +The canonical v1 effective configuration is the pre-existing DataDome schema +plus this exact PR-specific delta. The configuration parser rejects unknown +security/session fields rather than ignoring them as compatibility toggles, +and the materialized values below participate in permission §5.5's complete +effective-config digest. + +```toml +[integrations.datadome] + +# Existing timeout_ms remains the 1500 ms first-byte bound. +complete_response_timeout_ms = 3000 +challenge_body_max_bytes = 65536 + +# Required when enable_protection = true; there is no silent lifetime default. +security_cookie_max_age = 2592000 # example; allowed 604800..=31536000 +security_cookie_domain = "host-only" # or one exact normalized ASCII domain +security_cookie_same_site = "Lax" # Lax | Strict | None + +expose_client_id_to_origin = false +expose_host_fingerprints_to_vendor = false +``` + +The Protection API authority is not configurable: it is the core-owned +`https://api-fastly.datadome.co/validate-request` endpoint defined in §4a.2.1, +with redirects disabled. The baseline `protection_api_origin` field is a +startup error under this delta, as are `sessionByHeader`, `session_by_header`, +any equivalent session-mode spelling, and every other unknown +security/session field. Supporting another region or a publisher proxy is a +reviewed endpoint-registry and product/security decision, never a free-form +URL setting. + +`complete_response_timeout_ms` defaults to and may not exceed 3000; +`challenge_body_max_bytes` defaults to and may not exceed 65,536. +`security_cookie_max_age` is mandatory when protection is enabled and must be +in `604800..=31536000` seconds. `security_cookie_domain` defaults to +`host-only`; an explicit value must pass §4a's exact-domain, domain-match, PSL, +and active-scope-change rules. `security_cookie_same_site` accepts exactly +`Lax`, `Strict`, or `None`; `None` is valid only with the unconditionally +emitted `Secure` attribute, and the cookie never carries `HttpOnly`. + +Both exposure booleans default to `false`. ClientID-to-origin additionally +requires the owner-scoped overlay capability. Host fingerprints additionally +require qualified JA4 availability and sign-offs 23/28. A selected adapter +that cannot preserve admitted request-header field-line order or enforce the +request/body limits fails startup for protection rather than synthesizing +different evidence. + ## 4. Done-when (from #782, sharpened) 1. Trait + builder + registry application, each public item documented. 2. **The old generic `RequestFilterEffects.response_headers` channel is removed.** DataDome uses the separate sealed, core-registered `DataDomeSecurityRequestFilter` and typed `DataDomeSecurityEffects` defined - by its design; §4a defines that closed boundary. Generic request filters + by §4a's PR-specific delta. Generic request filters receive only `RedactedRequestView` and ordinary attributed effects and cannot express the security view, owner overlay, cookie operation, or reserved security header. The dedicated channel is necessary because @@ -1109,13 +1168,13 @@ and `X-DD-B` forwarded exactly once — neither fixture may fail open. ## 5. Size and sequencing -This is a modest feature plus tests with zero coupling to the provider -architecture or, in its v1 headers-only form (§3), to the permission -model. It lands whenever its first real consumer is identified (§4, item 3); -cookie operations arrive only with their own follow-up spec (§3) and its -permission-model coupling. If no consumer -materializes, it does not land; being unblocked is not a reason to ship -scaffolding. +The ordinary mutator API in §§2–3 is a modest headers-only feature with no +provider or permission-model coupling. It lands only with the real consumer +required by §4 item 3; without one, scaffolding does not ship. The separately +typed security channel in §4a is already that consumer's PR-specific contract: +it owns DataDome cookie operations and intentionally couples configuration +activation to permission §5.5. Those capabilities never become part of the +ordinary mutator API. ## 6. Divergences from issue #782 diff --git a/docs/superpowers/specs/2026-07-30-permission-model-design.md b/docs/superpowers/specs/2026-07-30-permission-model-design.md index 5a03d3e59..41c21874d 100644 --- a/docs/superpowers/specs/2026-07-30-permission-model-design.md +++ b/docs/superpowers/specs/2026-07-30-permission-model-design.md @@ -1184,7 +1184,7 @@ the admission-lease bound and promotion-not-before time, retention horizon, and controller identity. The journal object's qualified immutable-store metadata supplies store-issued `created_at`; its canonical schema, object ID, known-answer vector, lifecycle, listing, genesis, and pruning rules are -normative in CLI §5.1. The controller writes and read-verifies that object +normative in §5.5.3. The controller writes and read-verifies that object before promotion; the one register CAS both promotes the candidate and changes `activation_journal_head` to its object ID. A losing CAS leaves an unreferenced journal object, never an active tuple without a journal entry. Journal records @@ -1601,6 +1601,223 @@ revision fixture. +#### 5.5.3 Activation journal object and GC protocol + +The journal uses the same qualified immutable config-object service under the +reserved logical root `ts_activation_journal`, never the mutable app-config +root or the identity graph. Its logical object ID is lowercase +`SHA-256("tsactj1|" || RFC8785-JCS-UTF8(journal))`; adapters map +`("ts_activation_journal", object_id)` to a write-once physical object. The +object materializes exactly these fields and rejects unknown/missing fields: + +Every JSON number in the journal, including every number nested in an active +tuple, is an integer in `0..=9,007,199,254,740,991` (2^53 − 1). Booleans, +floats, negative values, and larger otherwise-valid `u64` values are rejected +before JCS; implementations may use wider internal integers but cannot emit +them here. Store-supplied lifecycle timestamps use the same portable range, +and addition that would exceed it fails closed. This profile makes the JCS +object ID identical in JavaScript, Rust, and every adapter rather than relying +on a language's larger integer type. + +- `schema_version = 1`; `attempt_id` as 32 lowercase hex characters from 16 + CSPRNG bytes, allowing a timed-out attempt to publish a new object; +- `candidate_incarnation` as the exact candidate's never-reused 32 lowercase + hex CSPRNG identity for `config`/`model`, or null for `checkpoint`; +- `previous_journal_id` and `pruned_through_journal_id`, each 64 lowercase hex + or null under the link/pruning rules below; +- `expected_activation_generation: u64` and `transition_kind` exactly + `config`, `model`, or `checkpoint`; +- `drain_attempt: u64`, which is the exact nonzero candidate drain attempt for + `config`/`model` and zero for `checkpoint`; +- `serve_admission_lease_bound_ms: u64`, the exact positive + deployment-qualified bound snapshotted by the candidate, and + `promotion_not_before_unix_ms: u64`, the exact store-clock gate written by + that drain attempt; both are zero only for a `checkpoint`; +- complete `displaced_active` and `activated_active` tuples from §5.5, + including settings bindings, policy identity, model epoch, minimum + binary generation, row schema floor, and logical activation generation; +- `membership_epoch: u64`, sorted unique `ready_members` and + `quiesced_members` using the stable member grammar, authenticated + `controller_id`, and `retain_for_ms: u64` constrained + to at least 2,592,000,000 and the longest applicable artifact, cookie-scope, + rollback, and audit horizon. + +The cross-language known-answer and rejection vectors are inline in §5.5.3.1; +every controller, runtime verifier, and GC must reproduce both JCS bytes and +object ID and reject every numeric boundary vector. For the first promotion, +`previous_journal_id` is null only when the register head is +null and expected generation is zero. Every later config/model promotion must +name the exact current head and has null `pruned_through_journal_id`; the active +register CAS rejects any link/generation mismatch. For config/model entries, +`expected_activation_generation` must equal current active's logical +activation generation, and activated active must set it to that value + 1; +both member lists must equal the candidate snapshot's complete sorted member +list, `membership_epoch` must equal that snapshot's epoch, `drain_attempt` must +equal the candidate's current attempt and every quiescence acknowledgment, and +`candidate_incarnation` must equal every readiness/quiescence binding, +`serve_admission_lease_bound_ms` and `promotion_not_before_unix_ms` must equal +the candidate's exact drain fields, the admission-lease bound must be positive, +the immutable-store `created_at` for the journal must be at or after the +promotion-not-before time and no more than 60 seconds before the promotion CAS, +and the promotion CAS must independently enforce that its register store clock +has reached that time. These comparisons are defined only because the +activation register and immutable object service expose the same qualified, +authenticated Unix-millisecond time domain; adapters with incomparable clocks +fail activation qualification rather than comparing local timestamps. +Independently, `displaced_active` must equal current active and +`activated_active` must equal the candidate's computed post-CAS tuple. Overflow +is a hard error. A checkpoint +uses the current membership epoch, empty `ready_members` and +`quiesced_members` lists, and identical displaced and activated tuples +(including unchanged activation generation), with null +`candidate_incarnation`, `drain_attempt = 0`, +`serve_admission_lease_bound_ms = 0`, and +`promotion_not_before_unix_ms = 0`; it cannot stand in for fleet readiness or +quiescence. + +The immutable store returns authenticated `created_at_unix_ms` object metadata +from the shared qualified activation time domain and maintains a separate, +extend-only `delete_not_before_unix_ms` lifecycle value. On every config or journal object +write, the adapter atomically initializes deletion protection to at least store +creation time + 30 days. For a promotion journal it extends protection for the +journal and both named blobs to at least `created_at + 60 seconds + +retain_for_ms` before the active CAS may bind the journal. These lifecycle +values can only increase. Therefore failed publication, aborted candidates, +losing journal attempts, and other unreferenced objects still have a store-clock +not-before value even though no successful promotion names them. + +The object service's qualification supplies snapshot-consistent complete +listing for both logical roots: a listing returns one snapshot generation and +opaque pagination token; every page is from that generation, and mutation or +expiry of the token forces GC to restart without deleting. GC first completes +the listing, traverses and verifies the journal from the active head, and builds +the active/candidate/history/journal reachable set. Missing objects, broken +hashes/links, unknown schema, cycles, incomplete pages, or uncertain lifecycle +metadata abort the run. Deletion then uses object-ID CAS and is allowed only +when the object is unreachable and its store-enforced not-before has passed. + +Journal pruning is an authenticated controller operation, never implicit GC. +Only when every record reachable from the current head is older than its full +retention horizon may the controller publish a `checkpoint` whose displaced +and activated tuples both equal current active, whose previous ID is null, and +whose `pruned_through_journal_id` is the old head. One register CAS verifies the +unchanged active tuple/generation and replaces only the journal head. The +checkpoint names and protects the current active blob; old journal objects +remain until their individual not-before values pass. Frequent activation can +therefore retain a longer chain but can never cut a still-required segment. + +##### 5.5.3.1 Activation-journal vectors + +The JSON object between the stable markers is the sole normative +machine-readable activation-journal fixture. Extractors exclude the markers +and code fences, parse the enclosed UTF-8 JSON, and must reject duplicate +object keys. + + + +```json +{ + "schema_version": 1, + "canonicalization": "RFC 8785 (JCS)", + "digest": "SHA-256", + "domain_prefix_utf8": "tsactj1|", + "numeric_profile": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "vectors": [ + { + "name": "genesis-config-promotion", + "journal": { + "schema_version": 1, + "attempt_id": "00000000000000000000000000000000", + "candidate_incarnation": "11111111111111111111111111111111", + "previous_journal_id": null, + "pruned_through_journal_id": null, + "expected_activation_generation": 0, + "drain_attempt": 1, + "serve_admission_lease_bound_ms": 1000, + "promotion_not_before_unix_ms": 1700000001000, + "transition_kind": "config", + "displaced_active": { + "logical_root": "builtin", + "immutable_blob_id": "builtin", + "source_version": 0, + "data_hash": "0000000000000000000000000000000000000000000000000000000000000000", + "config_revision": "0000000000000000000000000000000000000000000000000000000000000000", + "policy_digest": "0000000000000000000000000000000000000000000000000000000000000000", + "ordinal": 0, + "model_epoch": "pre_epic_v1", + "minimum_binary_generation": 1, + "row_schema_floor": 1, + "activation_generation": 0 + }, + "activated_active": { + "logical_root": "app_config", + "immutable_blob_id": "app_config/1", + "source_version": 1, + "data_hash": "1111111111111111111111111111111111111111111111111111111111111111", + "config_revision": "2222222222222222222222222222222222222222222222222222222222222222", + "policy_digest": "3333333333333333333333333333333333333333333333333333333333333333", + "ordinal": 1, + "model_epoch": "pre_epic_v1", + "minimum_binary_generation": 1, + "row_schema_floor": 1, + "activation_generation": 1 + }, + "membership_epoch": 7, + "ready_members": ["edge-a", "edge-b"], + "quiesced_members": ["edge-a", "edge-b"], + "controller_id": "deploy-controller", + "retain_for_ms": 2592000000 + }, + "canonical_json_utf8": "{\"activated_active\":{\"activation_generation\":1,\"config_revision\":\"2222222222222222222222222222222222222222222222222222222222222222\",\"data_hash\":\"1111111111111111111111111111111111111111111111111111111111111111\",\"immutable_blob_id\":\"app_config/1\",\"logical_root\":\"app_config\",\"minimum_binary_generation\":1,\"model_epoch\":\"pre_epic_v1\",\"ordinal\":1,\"policy_digest\":\"3333333333333333333333333333333333333333333333333333333333333333\",\"row_schema_floor\":1,\"source_version\":1},\"attempt_id\":\"00000000000000000000000000000000\",\"candidate_incarnation\":\"11111111111111111111111111111111\",\"controller_id\":\"deploy-controller\",\"displaced_active\":{\"activation_generation\":0,\"config_revision\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"data_hash\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"immutable_blob_id\":\"builtin\",\"logical_root\":\"builtin\",\"minimum_binary_generation\":1,\"model_epoch\":\"pre_epic_v1\",\"ordinal\":0,\"policy_digest\":\"0000000000000000000000000000000000000000000000000000000000000000\",\"row_schema_floor\":1,\"source_version\":0},\"drain_attempt\":1,\"expected_activation_generation\":0,\"membership_epoch\":7,\"previous_journal_id\":null,\"promotion_not_before_unix_ms\":1700000001000,\"pruned_through_journal_id\":null,\"quiesced_members\":[\"edge-a\",\"edge-b\"],\"ready_members\":[\"edge-a\",\"edge-b\"],\"retain_for_ms\":2592000000,\"schema_version\":1,\"serve_admission_lease_bound_ms\":1000,\"transition_kind\":\"config\"}", + "sha256_hex": "7af3934b4e5500903ef77ad8a1367db83b03fc62a0bb83bd0849289c685d2e88" + } + ], + "rejection_vectors": [ + { + "name": "unsafe-top-level-u64", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/expected_activation_generation", + "raw_json_number": "9007199254740992", + "error": "integer exceeds portable JCS profile" + }, + { + "name": "unsafe-embedded-active-u64", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/activated_active/source_version", + "raw_json_number": "9007199254740992", + "error": "integer exceeds portable JCS profile" + }, + { + "name": "fractional-journal-number", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/retain_for_ms", + "raw_json_number": "2592000000.5", + "error": "journal number is not an integer" + }, + { + "name": "unsafe-admission-lease-bound", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/serve_admission_lease_bound_ms", + "raw_json_number": "9007199254740992", + "error": "integer exceeds portable JCS profile" + }, + { + "name": "zero-promotion-admission-lease-bound", + "base_vector": "genesis-config-promotion", + "replace_json_pointer": "/serve_admission_lease_bound_ms", + "raw_json_number": "0", + "error": "config/model admission lease bound is not positive" + } + ] +} +``` + + + ## 6. Failure-mode matrix — normative | Condition | Resolution behavior | diff --git a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md index b8084d6de..f51e168ff 100644 --- a/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md +++ b/docs/superpowers/specs/2026-07-30-provider-migration-rollout-design.md @@ -579,6 +579,86 @@ global honoring of opt-out signals is unconditional. reader after a provider switch (providers spec §6.1), which is the deliberate end of the identities only that reader can resolve. +### 6.1 Operator CLI delta for this epic + +The pre-existing CLI design remains unchanged. For implementations governed by +this spec, this subsection is the normative delta to its `ts config push` +contract; where the baseline describes a direct mutable blob-store write, this +subsection and permission §5.5 supersede that behavior. The delta does not +authorize a generic raw metadata command or a model-transition command. + +`ts config push` keeps the baseline arguments and validation behavior, then: + +1. Obtains the next never-reused `push_sequence` from the selected adapter's + qualified linearizable deployment-metadata allocator. Concurrent losers + retry with a new value; gaps are allowed and reuse or overflow is forbidden. +2. Builds the envelope with both the canonical data hash and the permission + §5.5.2 sequence-binding hash, writes it under the immutable + `(logical_root, push_sequence)` identity, reads it back, and verifies both + hashes. Writing the mutable logical root does not stage or activate it. If + an adapter must split an envelope for platform limits, its pointer/manifest + belongs to that immutable identity and names immutable chunks; the adapter + reconstructs and verifies the complete envelope before readiness or + settings exposure. The manifest and chunks are one blob for reachability + and lifecycle protection. +3. CAS-installs the exact object and complete content-binding tuple as the sole + settings candidate, with a never-reused candidate incarnation and unchanged + active model fields. A competing candidate fails; config push cannot create + or promote a model candidate and no force option bypasses that boundary. +4. Uses permission §5.5's authoritative membership, readiness, bounded + admission-lease drain, quiescence, journal, and promotion protocol. Only the + final register CAS activates settings. A config-only push retains the policy + ordinal but still follows the complete protocol and scheduled unavailable + interval. + +`--dry-run` performs validation and reports the provisional immutable identity +and candidate tuple using the allocator's observed next value, but does not +reserve a sequence, write an object, or mutate the activation register. The +reported value is advisory because another publisher can win. + +The baseline reserved-future-key description is also superseded narrowly: +`ts-config-signature`, if implemented later, is a signature or DSSE envelope +over `sequence_binding_hash`, not merely the data/blob hash. Signing remains +out of scope for this epic. + +This epic also adds the following operator command as a delta owned here, not +as a modification to the baseline CLI design: + +```bash +ts config gc \ + --adapter \ + [--manifest ] \ + [--store ] \ + [--key ] \ + [--dry-run] \ + [--yes] \ + [--runtime-config ] +``` + +The command uses only permission §5.5.3's qualified immutable-object listing, +lifecycle, reachability, and object-ID-CAS deletion protocol. `--store` selects +the same logical config store for both roots; `--key` selects the app-config +root and defaults to `app_config`, while `ts_activation_journal` is fixed and +cannot be overridden. EdgeZero's selected adapter supplies snapshot listing, +object/lifecycle metadata, and object-ID-CAS deletion. Trusted Server owns +journal/hash validation and reachability decisions and never constructs a +platform-specific physical key. The command completes one snapshot-consistent +paginated inventory before deleting, verifies all hashes and journal links, +and protects every active, candidate, operational-history, retained-journal, +referenced-manifest/chunk, and referenced-blob object. Any incomplete listing, +expired pagination token, broken hash/link, cycle, unknown schema, uncertain +store clock, or missing lifecycle value aborts before the first deletion. + +`--dry-run` prints only roots, object IDs, reasons, and lifecycle timestamps. +Without it, deletion requires `--yes` or interactive confirmation. A partial +platform deletion error stops the run and reports exact completed IDs; retry is +safe because reachability and object-ID CAS are re-evaluated. The command never +publishes a checkpoint, changes the journal head, alters active/model state, or +guesses physical keys. Conformance tests cover dry-run non-mutation, complete +pagination, every reachability root, lifecycle boundaries, malformed graphs, +CAS conflicts, partial-error retry, confirmation, and reproduction of the +activation-journal vector in permission §5.5.3.1. + ## 7. Documentation deliverables - Migration guide page (§5), linked from `CHANGELOG.md` and the release @@ -625,7 +705,7 @@ implemented. | 16 | Persist use-opt-out suppression until ordered explicit authorization for that use or identity deletion. TCF `LastUpdated` or an authenticated monotonic authorization revision proves order; bare timestamp-less GPP/USP does not. A currently presented identical timestamp-less opt-out starts a new restrictive episode after a clear without refreshing its original age. TTL and saturation never shorten it. This rejects the prior TTL-sticky alternative under which suppression became inert at consent-TTL expiry, as well as administrative clear without newer authorization and saturation-based shortening. | permission §4.3 | — | open | | 17 | N/A, absent, reserved, unknown, and unsupported values never grant processing. | permission §4.5 | — | open | | 18 | A selected geo provider's lookup failure uses the compiled-in protective profile; `default_country` is only for acknowledged static-jurisdiction mode. | permission §5.2 | — | open | -| 19 | Use immutable version-addressed whole-config publication plus prepare/commit activation of the complete blob/data/config/policy tuple with authenticated authoritative fleet membership/readiness, a deployment-qualified bounded admission lease, a shared authenticated register/journal time domain, store-enforced promotion-not-before, an all-request quiescence barrier, and a time-retained immutable activation journal. Use a second unanimous, lease-drained and quiescent model transition on the same register to advance model epoch, minimum binary generation, and row schema floor atomically. Every ordinary settings promotion intentionally causes a scheduled fleet-wide deployment-unavailable interval; this is not a zero-downtime protocol, and controller failure may extend the outage until authenticated cancellation or promotion. A mutable “latest” blob never activates settings or the writer; membership changes restage the candidate; no request admitted under a displaced logical `activation_generation` remains able to produce effects after either promotion. The activation fence is universal, including stateless-identity and identity-free traffic; an adapter that cannot qualify it cannot serve under this spec. The lease amortizes only whole-settings admission: positive authority, revocation, outbox, `w`, and breaker decisions retain fresh strong reads. | permission §5.5; CLI §5 | — | open | +| 19 | Use immutable version-addressed whole-config publication plus prepare/commit activation of the complete blob/data/config/policy tuple with authenticated authoritative fleet membership/readiness, a deployment-qualified bounded admission lease, a shared authenticated register/journal time domain, store-enforced promotion-not-before, an all-request quiescence barrier, and a time-retained immutable activation journal. Use a second unanimous, lease-drained and quiescent model transition on the same register to advance model epoch, minimum binary generation, and row schema floor atomically. Every ordinary settings promotion intentionally causes a scheduled fleet-wide deployment-unavailable interval; this is not a zero-downtime protocol, and controller failure may extend the outage until authenticated cancellation or promotion. A mutable “latest” blob never activates settings or the writer; membership changes restage the candidate; no request admitted under a displaced logical `activation_generation` remains able to produce effects after either promotion. The activation fence is universal, including stateless-identity and identity-free traffic; an adapter that cannot qualify it cannot serve under this spec. The lease amortizes only whole-settings admission: positive authority, revocation, outbox, `w`, and breaker decisions retain fresh strong reads. | permission §5.5; rollout §6.1 | — | open | | 20 | N+1 keeps v1 minting and pre-epic live gating. It reads/enforces N+2 negative state for rollback safety and can persist an explicit pre-epic withdrawal/deletion, but it does not originate durable P4 use suppression. New-shape settings alone do not activate the new writer/model: N+2 emulates N+1 until the fleet-wide `permissions_v2` model CAS, after which the register's minimum binary generation bars N+1 from serving. The N+1 batch boundary already fails closed as soon as new-shape settings are active. | migration §4.4 | — | open | | 21 | Expire and re-mint rowless legacy cookies without continuity; a prefix match cannot authenticate the cookie suffix. Non-destructive signals are request-local and create no negative record for the old rowless identity; if ordinary P1-gated re-mint succeeds, the new row-backed family commits the current suppression before use. | providers §5 | — | open | | 22 | Defer host JA4/H2 fingerprint processing to a separate approved design; reject `[device] provider = "fastly"` at startup and do not persist fingerprint-derived classifications. | providers §5 | — | open | diff --git a/docs/superpowers/specs/pr986-review-ledger.md b/docs/superpowers/specs/pr986-review-ledger.md index cc72ee82a..528e181f7 100644 --- a/docs/superpowers/specs/pr986-review-ledger.md +++ b/docs/superpowers/specs/pr986-review-ledger.md @@ -883,3 +883,29 @@ The independent internal recomputation produced This verifies the current KAT inside the review process but is not relabeled as the fresh external recomputation requested in pass 5. Decision rows and release gates remain unchanged. + +## Post-decision pass 7 (baseline-spec scope correction, 2026-08-07) + +The product owner set a repository-scope boundary: specifications that predate +this PR are references, not modification targets. The final PR therefore keeps +both pre-existing June designs byte-identical to `origin/main`: + +- `2026-06-11-datadome-server-side-protection-design.md`; and +- `2026-06-16-edgezero-based-ts-cli-design.md`. + +The PR-specific contracts formerly written into those baselines now live only +in new PR-owned specifications. Hook §1 declares §4a the normative DataDome +delta and sole authority where the unchanged baseline conflicts. Permission +§5.5.3 owns the activation-journal schema, lifecycle, listing/pruning protocol, +and its byte-identical known-answer/rejection fixture; rollout §6.1 owns the +`ts config push` activation delta, immutable chunk-manifest mapping, future +signature binding, and `ts config gc` command/ownership contract. Hook §4a.3 +owns the canonical DataDome configuration delta and startup rejection rules. +Sign-off 19 cites those PR-owned sections rather than the baseline CLI spec. + +Earlier ledger entries describing banners or inline edits to the old DataDome +document record intermediate revisions and are superseded by this scope +correction; they are not descriptions of the final tree. The activation object +ID remains +`7af3934b4e5500903ef77ad8a1367db83b03fc62a0bb83bd0849289c685d2e88`. +Decision rows and release gates remain unchanged.