From df41b6a7a28fe6a1c5914cc3af9a03e6936e841e Mon Sep 17 00:00:00 2001 From: James Sadler Date: Mon, 20 Jul 2026 23:16:45 +1000 Subject: [PATCH 01/10] docs: handoff for the EQL v3 type checker design Captures the EQL v3 domain model, three impact maps over eql-mapper (type system, SQL surface, declarations/macros), and the open design questions, so the design session starts fresh with full context. Records one corrected finding: literals are inference sinks (value.rs:19 unifies them with a fresh tvar, not Native), so a token type on EqlValue buys no literal checking. Two subagents disagreed on this; resolved by reading the code. Stable-Commit-Id: q-2r73hgg9ex95aba238a8fbe4kp --- .../2026-07-20-eql-v3-type-checker-handoff.md | 397 ++++++++++++++++++ 1 file changed, 397 insertions(+) create mode 100644 docs/plans/2026-07-20-eql-v3-type-checker-handoff.md diff --git a/docs/plans/2026-07-20-eql-v3-type-checker-handoff.md b/docs/plans/2026-07-20-eql-v3-type-checker-handoff.md new file mode 100644 index 00000000..31062137 --- /dev/null +++ b/docs/plans/2026-07-20-eql-v3-type-checker-handoff.md @@ -0,0 +1,397 @@ +# EQL v3 type checker — session handoff + +**Date:** 2026-07-20 +**Branch:** `chore/agent-skills-setup` (PR #415) +**Purpose:** carry context into a fresh session for `/grill-with-docs` on extending the +EQL Mapper type checker for EQL v3. + +This is a handoff, not a spec. It records what is already true, what was already decided, +and what is still open. The design itself has not started. + +--- + +## 1. How to use this + +Open a fresh session, reference this file, and run `/grill-with-docs`. The design questions +in §8 are the agenda. Everything above them is evidence gathered so the grill does not have +to re-derive it. + +Glossaries exist and should be updated as terms get resolved: +`CONTEXT-MAP.md`, `packages/eql-mapper/CONTEXT.md`, `packages/cipherstash-proxy/CONTEXT.md`. + +--- + +## 2. Where the branch is + +Five commits, unpushed beyond #415's first. In order: + +| Commit | What | +|---|---| +| `3d33c8ed` | Agent skills config (issue tracker, domain docs) | +| `74b63721` | Domain glossaries for Proxy and EQL Mapper | +| `64c658f2` | Deps: cipherstash-client `0.34.1-alpha.4` → `0.42.0`, EQL `2.3.0-pre.3` → `3.0.1` | +| `b3f293d7` | Encrypt path moved to `encrypt_eql_v3`; v2 retired | +| `a10b60cd` | v3 decrypt for scalar + SteVec | +| `387ca790` | Adopted the 0.42.0 representation; workspace compiles | + +**State:** the workspace compiles, `cargo fmt` and `cargo clippy --all-targets` are clean. +**Nothing has been run.** Compiling is not working — `eql-mapper` still emits `eql_v2_encrypted` +casts and `eql_v2.*` calls into every rewritten statement, so nothing works end to end. +229 `eql_v2` references remain repo-wide. + +The vendored `stack-auth` CIP-3159 patch was dropped (0.42.0 requires `stack-auth ^0.42.0`, +which carries the CancelGuard fix upstream). `vendor/stack-auth/` is still on disk, unreferenced. + +--- + +## 3. The EQL v3 model + +EQL 3.0 removes the `eql_v2` schema entirely. The single opaque `eql_v2_encrypted` +**composite** type is replaced by **53 typed domains over jsonb**, two-dimensional: + +**Token type** × **capability suffix** + +- Tokens: `integer`, `smallint`, `bigint`, `date`, `timestamp`, `numeric`, `text`, + `boolean`, `real`, `double`, `json` +- Suffixes: *(none)*, `_eq`, `_ord`, `_ord_ope`, `_ord_ore`, `_match`, `_search`, `_search_ore` + +Column domains live in `public` (`public.eql_v3_integer_ord`); query-operand twins live in +`eql_v3` (`eql_v3.query_integer_ord`). 39 query twins, enumerated separately. + +Capability is a property of the **type**, and the domain's CHECK enforces that the payload +actually carries the required terms — so a missing term fails on insert, not later at query time. + +### SEM terms (the storage beneath a capability) + +| Suffix | Operators | Term | +|---|---|---| +| *(none)* | — (storage only) | `c` | +| `_eq` | `=` `<>` | `hm` (HMAC-256) | +| `_ord` | `=` `<>` `<` `<=` `>` `>=`, MIN/MAX | `op` (CLLW-OPE) | +| `_ord_ope` | as `_ord` | `op` | +| `_ord_ore` | as `_ord` | `ob` (block-ORE) | +| `_match` | `@@` | `bf` (bloom filter) | +| `_search` | all | `hm` + `op` + `bf` | +| `_search_ore` | all | `hm` + `ob` + `bf` | + +Text is the exception: `text_ord*` carries `hm` **as well as** the ordering term, because +lexicographic ORE/OPE over text is not equality-lossless. + +> **Upstream doc bug.** `eql-bindings` `src/v3/mod.rs:26` claims "`ob` for `_ord`/`_ord_ore`, +> `op` for `_ord_ope`". The generated code disagrees — `IntegerOrd` requires `op`, +> `IntegerOrdOre` requires `ob`. Trust `term_json_keys_static()`, not that module doc. + +### Vocabulary + +`EqlTrait` is a **capability**; a **SEM term** (searchable encrypted metadata) is the storage +that satisfies it. Many-to-one. Neither is an "index" — that word is reserved for a PostgreSQL +index. The EQL config JSON and `cipherstash_client`'s `ColumnConfig` still spell SEM terms +`indexes`/`IndexType`; that is a shared wire format and not ours to rename. + +--- + +## 4. Upstream facts worth knowing + +**`eql-bindings` 3.0.1** (crates.io) is generated from `eql-domains::CATALOG` by `eql-codegen` — +the same catalog that generates the SQL. Light deps (serde, serde_json, schemars, ts-rs). + +The useful part for us: + +```rust +IntegerOrd::sql_domain_static() // "public.eql_v3_integer_ord" +IntegerOrd::term_json_keys_static() // Some(&["op"]) +``` + +`v3::all()` enumerates the 53 stored domains, `all_query()` the 39 query twins. Inverting +`all()` gives a **typname → (token type, required SEM terms)** map straight from the catalog, +which cannot drift from the SQL. `sql.rs` also embeds `INSTALL_SQL`/`UNINSTALL_SQL` as consts, +which could replace the `curl` in `mise.toml`'s `eql:download`. + +**Containment is gone.** `@>`/`<@` survive only as internal blockers that raise (CIP-3517). +Fuzzy match is `@@` via `eql_v3.matches`. + +**`boolean` is storage-only by design** — no `_eq`, no `_ord`, every operator blocked, because +a two-value column leaks its distribution under any index. + +### The 0.41.1 / 0.42.0 SteVec split — UNRESOLVED + +cipherstash-client changed the v3 SteVec wire format and nothing else followed: + +| | 0.41.1 (protect-ffi, eql-bindings 3.0.1) | 0.42.0 (what Proxy now pins) | +|---|---|---| +| Document | `{v,k,i,sv}` | `{v,k,i,**h**,sv}` | +| Entry `c` | self-describing mp_base85 record | raw base85 AEAD bytes | +| Key material | per entry | once, in `h` | +| Nonce / AAD | data key IV | `selector[..12]` / all 16 selector bytes | + +EQL 3.0.1's SQL CHECK accepts **both** (it never forbids extra keys), so the database will not +catch a mismatch. ProtectJS and Proxy writing encrypted jsonb to the same table would produce +mutually undecryptable documents. Either Proxy pins back to 0.41.1 or protect-ffi and +eql-bindings move forward. Not Proxy's call alone. **This is orthogonal to the type checker +work but blocks shipping.** + +### CLLW-ORE data is stranded + +`from_v2` returns `UnconvertibleOreTerm` for `oc` SteVec entries — re-encryption is the only +path. `cipherstash-config` now models this: `IndexType::SteVec { mode }` where +`SteVecMode::Compat` (CLLW-OPE, `op`) is the default and v3-compatible, and `SteVecMode::Standard` +(CLLW-ORE, `oc`) is documented upstream as "the legacy v2 protocol, still used by Proxy". +This is CIP-3233 made explicit in the config model. + +--- + +## 5. Impact map — the type system + +Three parallel agents surveyed `eql-mapper`. Findings, with the conflicts resolved. + +### Where encrypted types come from + +**Only two production sites** construct an `EqlValue` from schema data: + +- `inference/unifier/types.rs:508-517` — `Projection::new_from_schema_table` +- `inference/infer_type_impls/insert_statement.rs:40-49` — INSERT target columns + +Both produce `EqlTerm::Full`. Everything else clones. Upstream of both: +`packages/cipherstash-proxy/src/proxy/schema/manager.rs:142-148`, which matches the single +string `"eql_v2_encrypted"` and hardcodes `EqlTraits::all()`. **That one match arm is where the +53-domain mapping has to go.** + +### The shapes that must widen + +- `EqlValue(TableColumn, EqlTraits)` — `unifier/types.rs:274` +- `ColumnKind::{Native, Eql(EqlTraits)}` — `model/schema.rs:41-45` +- `Column::eql(name, features)` — `model/schema.rs:48` +- `SchemaTableColumn` — `model/schema.rs:63` + +There is **no field anywhere in eql-mapper holding a plaintext scalar type for an encrypted +column.** The `_ord` half of `eql_v3_integer_ord` is derivable from `EqlTraits`; the `integer` +half is derivable from nothing in the package. + +### Two authorities on the scalar type, no reconciliation + +The scalar type already exists in the system as `ColumnType`, sourced from the **encrypt config** +and consumed at `cipherstash-proxy/src/postgresql/data/from_sql.rs:123-292` and +`context/column.rs:68-86`. Under v3 the **Postgres type name** also encodes it. A config/schema +disagreement is currently undetectable. + +### Encrypted columns never unify with each other + +`unifier/types.rs:121` — *"An encrypted column never shares a type with another encrypted column."* +`unify_types.rs:84-121`: two EQL terms unify iff their `EqlValue`s are `==`, so `TableColumn` +identity dominates. `users.email = orders.email` is already a type error. + +**Therefore a token type is redundant for `Full`/`Full` unification.** Same column → same token +type, always. + +### Literals are inference sinks — CORRECTED FINDING + +Two agents disagreed here; resolved by reading the code. + +`infer_type_impls/value.rs:19` — a non-placeholder literal unifies with `self.fresh_tvar()`, +an **unbound variable**, not `Native`. And `(Eql, Native)` is a hard error +(`unify_types.rs:68-70`). + +So `WHERE encrypted_col = 'literal'` works because the literal has no type of its own and +**absorbs the column's**. Consequence: adding a token type to `EqlValue` buys **zero** literal +checking — there is nothing for `eql_v3_integer_ord = 'abc'` to contradict. Getting that check +would require literals to carry a syntactic type they deliberately do not have. + +This is not the same as "Native satisfies all bounds". That escape hatch +(`unifier/eql_traits.rs:279`) is about **bound satisfaction**, not cross-kind unification. + +### Bound checking is dead code today + +`Unifier::satisfy_bounds` (`unifier/mod.rs:235-248`) and `Type::must_implement` +(`unifier/types.rs:451-460`) can never fail in production, because: + +- every column gets `EqlTraits::all()` (`manager.rs:146`) +- `Native` ⇒ `ALL_TRAITS` (`eql_traits.rs:279`) +- `Type::Var` short-circuits to `Ok(())` (`mod.rs:236`) + +**Two latent bugs will become user-visible the moment these go live:** + +1. `EqlTraits::difference` (`eql_traits.rs:223-231`) is implemented as **XOR**, not set + difference. `UnsatisfiedBounds` will report the symmetric difference — listing traits the + type *has* but the bound never required. +2. `must_implement` (`types.rs:456`) passes its operands **reversed** relative to + `satisfy_bounds` (`mod.rs:245`). Harmless only because XOR is commutative. + +Fix both before making bounds reachable. + +### Associated types + +`AssociatedTypeSelector { eql_trait, type_name }` (`types.rs:236`) is keyed on +`(EqlTrait, &'static str)` only. Resolution (`eql_traits.rs:56-115`) maps +`Eq::Only`/`Ord::Only`/`Contain::Only` → `Partial`, `TokenMatch::Tokenized` → `Tokenized`, +`JsonLike::{Path,Accessor}` → `JsonPath`/`JsonAccessor`. + +**If the token type lives inside `EqlValue`, this machinery needs no change** — every arm does +`eql_col.clone()`, so it propagates free. Do **not** add a field to `AssociatedTypeSelector`: +it would break the `lhs.selector == rhs.selector` equality at `unify_types.rs:160` for a +dimension already determined by `impl_ty`. + +--- + +## 6. Impact map — the SQL surface + +### The entire v2 SQL surface is three call sites + +Six rules, composed at `type_checked_statement.rs:153-160`. Only three emit SQL. + +**`helpers.rs:6-27`** — `cast_as_encrypted`. The single source of every +`::JSONB::eql_v2_encrypted` cast. Takes **only** an `ast::Value` — no `EqlTerm`, no +`TableColumn`, no traits. The cast target is a hardcoded `Ident::new("eql_v2_encrypted")` +at line 17. **This is the one place the v3 domain name would be chosen.** + +**`cast_params_as_encrypted.rs:51`** — emits `$1::JSONB::eql_v2_encrypted`. Its gate at line 67 +matches `Some(Type::Value(Value::Eql(_)))` — **the `EqlTerm` is available and discarded by the +`_`, one line before use.** That is the hook for selecting a `query_` twin. + +**`cast_literals_as_encrypted.rs:33`** — emits `''::JSONB::eql_v2_encrypted`. Gated +on map membership, not type. Holds **no `node_types` field at all** (line 13), so it cannot see +type info even in principle. Needs plumbing, not just a pattern change. + +**`rewrite_standard_sql_fns_on_eql_types.rs:59-61`** — blindly prepends `eql_v2` to any +`pg_catalog.*` name. Emits `eql_v2.{count,min,max,jsonb_path_query,jsonb_path_query_first, +jsonb_path_exists,jsonb_array_length,jsonb_array_elements,jsonb_array_elements_text}`. +Note it emits `eql_v2.count`, which **is not in the declaration table** — a name the type +registry cannot round-trip. + +**`rewrite_containment_ops.rs:54-57,86-87`** — `@>` → `eql_v2.jsonb_contains`, +`<@` → `eql_v2.jsonb_contained_by`. Motivated by GIN index usage, not just type dispatch. + +`preserve_effective_aliases.rs` and `fail_on_placeholder_change.rs` emit nothing. + +### Rules that exist only because v2 had one opaque type + +`RewriteStandardSqlFnsOnEqlTypes` — PG cannot dispatch `min`/`max`/`jsonb_*` on one opaque +jsonb-backed type, so they are shadowed by hand-written `eql_v2.` overloads. Under v3, PG +resolves overloads by argument type natively — **but only if v3 ships operator/function +overloads bound to those domains.** Redundant *conditional on* that, and that is overload +resolution, not capability-in-the-type. Different mechanisms; verify before assuming. + +`RewriteContainmentOps` — same root cause: `@>` cannot be defined on a bare opaque type +without conflicting with jsonb's own operator. + +**Not in this category:** the two cast rules (still needed, only the target changes), +`PreserveEffectiveAliases` (projection naming, orthogonal), `FailOnPlaceholderChange` (assertion). + +### Discipline to preserve + +`rewrite_containment_ops.rs:92-99` uses `mem::replace` against a throwaway `Value::Null` rather +than cloning, specifically so operand `NodeKey` identity survives for the cast rules that run +after it. **Clone there and literal encryption inside containment expressions silently breaks.** + +--- + +## 7. Impact map — declarations and macros + +### Operator → EqlTrait, in full (`inference/sql_types/sql_decls.rs:16-31`) + +| Operator | Signature | Trait | +|---|---|---| +| `=` `<>` | `(T op T) -> Native` | `Eq` | +| `<` `<=` `>` `>=` | `(T op T) -> Native` | `Ord` | +| `->` `->>` | `(T op ::Accessor) -> T` | `JsonLike` | +| `@>` `<@` | `(T op T) -> Native` | `Contain` | +| `~~` `!~~` `~~*` `!~~*` | `(T op ::Tokenized) -> Native` | `TokenMatch` | + +No `@@` is declared. Anything unlisted falls to `Fallback`, which forces lhs, rhs and result +to `Type::native()` (`sql_binary_operator_types.rs:40-44`). + +Functions at `sql_decls.rs:58-79`: `min`/`max` require `Ord`; the `jsonb_*` family requires +`JsonLike`; `jsonb_array`/`jsonb_contains`/`jsonb_contained_by` require `Contain`; **`count` +has no bound at all** and accepts any `T` including EQL. + +### The macro grammar is capability-only by construction + +`eql-mapper-macros/src/parse_type_decl.rs:11-25` — the entire custom-keyword vocabulary is +`Accessor, Contain, EQL, Eq, Full, JsonLike, Native, Only, Ord, Partial, Path, SetOf, TokenMatch`. +Every one is a trait name, an associated-type name, or a type constructor. **Not one names a +data type.** + +`EQL(customer.age: Ord)` is expressible. "Encrypted integer with Ord" is not — not in the +keyword table, not in the grammar, not in the generated `EqlValue`. A v3 port needs a new +lexical class, new productions in `EqlTerm` (`:289-331`) and `VarDecl` (`:70-91`), and a +widened payload. Work is concentrated in that one file. + +`@@` will not parse today: `token::At` at `:480` unconditionally expects a following `Gt`. + +### Things that become ambiguous with a token type + +1. **`(T = T)` — the core ambiguity.** One tvar currently conflates *same column*, *same + token type*, and *any encrypted*. Indistinguishable today because encrypted identity **is** + the column. +2. **`EqlTerm::Partial`'s union-on-unify** (`unify_types.rs:91`) accumulates capabilities + monotonically. If capability is fixed by the domain, you cannot union your way into a + capability the column does not have. +3. **The payload-less variants** — `JsonAccessor`, `JsonPath`, `Tokenized` all return + `EqlTraits::none()` (`eql_traits.rs:320-322`). "A tokenized *what*" needs answering. +4. **`Native` ⇒ `ALL_TRAITS`** and **`Type::Var` ⇒ `Ok(())`** are two independent wildcards in + the bound checker. + +### Pre-existing gaps this work will expose + +- **`Expr::Like`/`ILike` bypass the trait system entirely.** `infer_type_impls/expr.rs:115-131` + only unifies the result with `Native` — never requires `TokenMatch`, never produces a + `Tokenized` term. The `~~` declarations fire only on the operator form. LIKE on an encrypted + column is currently unchecked. +- **`schema_delta.rs:447,479,529` uses `EqlTraits::default()`** (all false) while the loader uses + `all()` (all true). The two paths disagree. Invisible while bounds are dead. +- **Empty projections** return `EqlTraits::none()` (`eql_traits.rs:310`) while the + `satisfy_bounds` doc (`mod.rs:220`) says they satisfy all bounds. Doc and code disagree. +- `EqlTrait` parse error string (`parse_type_decl.rs:172`) is stale — omits `Contain`. + +### Removing `Contain` touches + +Keyword table (`parse_type_decl.rs:13`), enum (`eql_traits.rs:23`), assoc types (`:39`), +resolution (`:73`, `:101-105`), `ALL_TRAITS` (`:154`), struct field (`:145`), `add_mut` (`:200`), +`to_eql_trait_impls!` (`schema.rs:225-227`), two operators (`sql_decls.rs:25-26`), three +functions (`sql_decls.rs:76-78`). + +--- + +## 8. Open design questions — the grill agenda + +**The framing question.** The research says the token type's job is **naming a v3 domain at +cast time**, not type checking: it is redundant for unification (§5), and it buys no literal +checking (§5). So — does it belong in the type lattice at all, or should the transformation +rules look the domain up from the schema by `TableColumn` at emit time and leave the lattice +alone? The second is materially cheaper and nothing found so far rules it out. **Settle this +before anything else; most other answers follow from it.** + +Then: + +1. If it does go in the lattice: inside `EqlValue` (associated types come free, all six + unification arms come free) or on `EqlTerm` (six explicit merges)? +2. `_ord` vs `_ord_ope` vs `_ord_ore` — identical operators, different SEM terms. Does the type + checker distinguish them, or is that purely a cast-target concern? +3. `integer_eq` and `bigint_eq` are byte-identical on the wire but different domains. Type error + to compare? (Note: already a type error via `TableColumn` identity — is that enough?) +4. What replaces `Contain`? `@>`/`<@` are **directional**; `@@`/`eql_v3.matches` is **symmetric**. + `<@` has nowhere to go. Delete the trait, or repoint at `@@` and lose direction? +5. Do bounds go live? Real errors for `ORDER BY` on an `_eq` column — but the domain CHECK + catches it anyway, and `sql_decls.rs:49` states the existing philosophy is *"let the database + do its job."* Does v3 change that stance? +6. Params must cast to `eql_v3.query_` twins, not the stored domain. Does the mapper need + the query inventory too, or can the twin name be derived by prefixing? +7. Which authority owns an encrypted column's scalar type — `pg_type` or the encrypt config? + Should the other be cross-checked? +8. Does `eql-bindings` become a dependency of `eql-mapper` (inverting `all()` into a + typname → capability map), or does the mapper keep its own table? + +### Prerequisites, whatever the design + +- Fix `EqlTraits::difference` XOR bug and the reversed `must_implement` operands **before** + bounds become reachable. +- Reconcile `schema_delta.rs`'s `default()` with the loader's `all()`. +- Verify whether EQL 3 ships operator/function overloads bound to the domains — that alone + decides whether `RewriteStandardSqlFnsOnEqlTypes` retires. + +--- + +## 9. Related memory + +`~/.claude/projects/-Users-jamessadler-cipherstash-proxy/memory/`: +`eql-3-upgrade-in-flight.md`, `cip-3233-client-038-ore-cllw-break.md`, +`proxy-222-stackauth-15min-auth-bug.md`. From d2a5f3fda84f3756fa2eeb08a2f6099095f23454 Mon Sep 17 00:00:00 2001 From: James Sadler Date: Tue, 21 Jul 2026 23:57:08 +1000 Subject: [PATCH 02/10] docs(eql-mapper): record v3 type-checker design (glossary + ADRs) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures the shared understanding from the grill-with-docs session on extending the EQL Mapper type checker for EQL v3. CONTEXT.md: refresh the glossary to the v3 target vocabulary — EqlValue now carries domain identity; EqlTrait drops Contain and is documented as coarse; add term-extraction function, functional-index rewrite, query operand, token type, and domain identity. The old 'known model gap' note becomes 'design decisions in flight' pointing at the ADRs. ADRs (new packages/eql-mapper/docs/adr/): - 0001 functional-index rewrite (term functions, not native operators) — forced by Supabase's lack of CREATE OPERATOR; term-fn selection is the capability check. - 0002 token type as inert identity, not a checked dimension — it buys no safety (encrypted cols never unify; literals are inference sinks). Assessed against EQL/eql-bindings 3.0.2 (released 2026-07-20): additive query-twin bindings only; none of the design decisions change. Stable-Commit-Id: q-06b9k91e7pnntayq3rs11tqznz --- packages/eql-mapper/CONTEXT.md | 79 +++++++++++++++---- .../docs/adr/0001-functional-index-rewrite.md | 33 ++++++++ .../adr/0002-token-type-as-inert-identity.md | 34 ++++++++ 3 files changed, 132 insertions(+), 14 deletions(-) create mode 100644 packages/eql-mapper/docs/adr/0001-functional-index-rewrite.md create mode 100644 packages/eql-mapper/docs/adr/0002-token-type-as-inert-identity.md diff --git a/packages/eql-mapper/CONTEXT.md b/packages/eql-mapper/CONTEXT.md index 31e532b2..65b8931c 100644 --- a/packages/eql-mapper/CONTEXT.md +++ b/packages/eql-mapper/CONTEXT.md @@ -1,9 +1,15 @@ # EQL Mapper Parses SQL, infers a type for every node, and rewrites statements that touch encrypted -columns into their EQL v2 equivalents. It knows nothing about the PostgreSQL wire +columns into their EQL v3 equivalents. It knows nothing about the PostgreSQL wire protocol, ZeroKMS, or ciphertext — it reasons about types and rewrites syntax. +> **Migration in flight.** The code still emits the EQL v2 surface +> (`eql_v2_encrypted` casts, `eql_v2.*` calls); the vocabulary below is the **v3 +> target** the type-checker extension is being designed against. See +> [`docs/adr/`](./docs/adr/) for the load-bearing decisions and +> `docs/plans/2026-07-20-eql-v3-type-checker-handoff.md` for the impact maps. + ## Language ### The type system @@ -25,14 +31,30 @@ bound — that is a deliberate escape hatch for the type checker, not a claim ab database. **EqlValue**: -The identity of an encrypted column paired with the capabilities configured for it — -a `TableColumn` plus `EqlTraits`. +The identity of an encrypted column: a `TableColumn`, its **domain identity**, and its +`EqlTraits`. Two encrypted columns never share a type, so the `TableColumn` alone settles +unification — the domain identity and traits ride along for rewriting, not for checking. + +**Domain identity**: +The inert `(token type, v3 domain)` an encrypted column carries — e.g. `text` / +`eql_v3_text_ord_ore`. Populated by the schema loader from the Postgres domain name, +never a checked dimension of unification. It does two jobs at rewrite time: names the +cast target and selects the **term-extraction function** variant (`ord_term` vs +`ord_term_ore`). It is the home of every v3 specific — token type, OPE-vs-ORE — that the +coarse `EqlTrait` deliberately does not carry. + +**Token type**: +The plaintext scalar half of a v3 domain — `integer`, `text`, `timestamp`, … The +capability half (`_eq`, `_ord`, …) is the `EqlTraits`; together they name the domain. **EqlTrait**: -A class of operation an encrypted value supports: `Eq`, `Ord`, `TokenMatch`, `JsonLike`, -`Contain`. A **capability**, not a storage structure — several SEM terms can satisfy one -trait. See the shared vocabulary in `CONTEXT-MAP.md`. -_Avoid_: index, index type (those name the storage, not the capability). +A class of operation an encrypted value supports: `Eq`, `Ord`, `TokenMatch`, `JsonLike`. +A **capability**, not a storage structure — several SEM terms can satisfy one trait. It +is **coarse** by design: `Ord` says "ordering is allowed" without distinguishing OPE from +ORE, because that variant lives in the domain identity. See the shared vocabulary in +`CONTEXT-MAP.md`. +_Avoid_: index, index type (those name the storage, not the capability); `Contain` (the +v2 containment trait — removed in v3, where `@>`/`<@` on encrypted values raise). **EqlTraits**: A set of `EqlTrait`s. Read in two opposite directions depending on position: as @@ -83,7 +105,7 @@ The result of type checking — the projection, params, literals and node types, entry point that applies transformations. **TransformationRule**: -A composable mutation of the typed AST that rewrites plaintext SQL into EQL v2 +A composable mutation of the typed AST that rewrites plaintext SQL into EQL v3 operations. Tuples of rules are themselves rules. _Avoid_: using "rule" for a typing rule; those are operator/function signature declarations. @@ -92,11 +114,40 @@ declarations. A `$N` placeholder position in the statement. Its PostgreSQL type OID is Proxy's concern, not this crate's. -## Known model gap +### EQL v3 rewriting + +**Term-extraction function**: +An `eql_v3.*` function that pulls one SEM term out of an encrypted value into a natively +comparable scalar: `eq_term` → HMAC, `ord_term` → CLLW-OPE, `ord_term_ore` → block-ORE, +`match_term` → bloom filter. These are the target of rewriting, and the thing a functional +index is built on. + +**Functional-index rewrite**: +The core v3 transformation: `col x` becomes `term_fn(col) term_fn(x)`, where +`term_fn` is chosen by the capability the operator exercises. It is the only form portable +to managed Postgres (Supabase forbids `CREATE OPERATOR`), and it lets a functional index +`CREATE INDEX ON t (eql_v3.ord_term(col))` engage via the term type's native opclass. +Selecting the term function *is* the capability check — a column whose domain provides no +such term is a capability error, not a fallback. +_Avoid_: describing this as "rewriting to operators"; v3 dispatches through functions. + +**Query operand** (query twin): +The term-only payload used as the right-hand side of a rewritten predicate — +`{v, i, }`, the envelope minus the record ciphertext `c`. It inhabits the +`eql_v3.query_` domains (the `query_` twins), distinct from the `public` +column domains. A stored value is a full column domain; an operand is a query twin. + +## Design decisions in flight The trait-bounds machinery here is complete — `satisfy_bounds` and `UnsatisfiedBounds` -exist and work — but Proxy currently feeds every encrypted column `EqlTraits::all()` -(`packages/cipherstash-proxy/src/proxy/schema/manager.rs:146`) rather than the traits its -configured SEM terms actually provide. So bound violations cannot be caught at type-check -time in production today. Treat `EqlTraits` on an `EqlValue` as *intended* capability, -not observed capability, until that join exists. +exist and work — but is **dead code today** because Proxy feeds every encrypted column +`EqlTraits::all()` (`packages/cipherstash-proxy/src/proxy/schema/manager.rs:146`). + +The v3 type-checker extension closes this: the loader will source real per-column +`(token, capability)` from the Postgres domain name, making capability **observed**, not +intended, and turning bound-checking live. See the ADRs: + +- [ADR-0001](./docs/adr/0001-functional-index-rewrite.md) — rewrite through + term-extraction functions rather than native operators. +- [ADR-0002](./docs/adr/0002-token-type-as-inert-identity.md) — the token type is inert + identity data, not a checked type dimension. diff --git a/packages/eql-mapper/docs/adr/0001-functional-index-rewrite.md b/packages/eql-mapper/docs/adr/0001-functional-index-rewrite.md new file mode 100644 index 00000000..13eaf0d8 --- /dev/null +++ b/packages/eql-mapper/docs/adr/0001-functional-index-rewrite.md @@ -0,0 +1,33 @@ +--- +status: accepted +--- + +# Rewrite encrypted operators through term-extraction functions, not native operators + +EQL v3 ships native operators (`CREATE OPERATOR public.=`, btree opclasses) bound to its +53 domain types, which makes it tempting to leave comparisons untouched and let Postgres +dispatch them. We will **not** do that. The mapper rewrites every operator on an encrypted +column into the functional form `eql_v3._term(lhs) eql_v3._term(rhs)` +— `eq_term` (→ HMAC), `ord_term` / `ord_term_ore` (→ CLLW-OPE / block-ORE), `match_term` +(→ bloom filter) — with the term function chosen by the capability the operator exercises. + +**Why:** our primary deployment target is Supabase and other managed Postgres, where +`CREATE OPERATOR` DDL is unavailable to non-superusers, so the native-operator path simply +does not exist there. The functional form is the only surface portable across every +install, it engages functional indexes (`CREATE INDEX ON t (eql_v3.ord_term(col))`) via +the term type's native opclass, and we have tested that Postgres's query planner handles +it well. This also collapses two concerns into one: **selecting the term function is the +capability check** — a column whose domain provides no matching term (e.g. `ORDER BY` on an +`eql_v3_text_eq` column, or any operator on storage-only `eql_v3_boolean` / `eql_v3_json`) +has no valid rewrite target, which is exactly the type error we raise. + +## Consequences + +- The v2 transformation layer does **not** evaporate under v3 — it is retargeted from the + `eql_v2.*` opaque-type functions to the `eql_v3.*_term()` functional-index surface. This + is close to the v2 structure, not a rewrite from scratch. +- The `_ord` vs `_ord_ore` distinction (OPE `op` vs block-ORE `ob`) becomes load-bearing: + the mapper must emit `ord_term` vs `ord_term_ore` per the column's domain. That variant + is read from the domain identity (see ADR-0002), not from the coarse `Ord` trait. +- Native `@>`/`<@`/operators on **plaintext** columns are untouched; only encrypted + operands are rewritten. diff --git a/packages/eql-mapper/docs/adr/0002-token-type-as-inert-identity.md b/packages/eql-mapper/docs/adr/0002-token-type-as-inert-identity.md new file mode 100644 index 00000000..f1930120 --- /dev/null +++ b/packages/eql-mapper/docs/adr/0002-token-type-as-inert-identity.md @@ -0,0 +1,34 @@ +--- +status: accepted +--- + +# The v3 token type is inert identity data, not a checked type dimension + +EQL v3 encrypted columns are two-dimensional — a token type (`integer`, `text`, …) crossed +with a capability (`_eq`, `_ord`, …) — where v2 had one opaque `eql_v2_encrypted`. We carry +the token type (and full domain) inside `EqlValue` as **inert identity data**: populated by +the schema loader from the Postgres domain name, read only at rewrite time to name the cast +target and select the term-extraction-function variant, and **not** a dimension the unifier +checks. + +**Why not make it a checked dimension:** it would buy no safety. Two encrypted columns +never unify with each other — `TableColumn` identity already settles it (`unify_types.rs`), +so `users.email = orders.email` is already a type error regardless of token type. And a +plaintext literal is an inference sink: it starts as an unbound variable and *absorbs* the +column's type (`value.rs`), so there is nothing for `eql_v3_integer_ord = 'abc'` to +contradict at the literal. A checked token dimension would force the macro grammar, all six +unification arms, and the bound logic to learn a new axis for zero return. Capability +(`EqlTraits`) stays the only checked axis; the token type and the OPE-vs-ORE variant ride +in the identity and surface only during code generation. + +## Consequences + +- `EqlValue` becomes `(TableColumn, domain identity, EqlTraits)`; the identity threads + through the associated-type machinery and unification arms for free because it is never + inspected there. +- The source of truth for the identity is the Postgres domain name, parsed in + `cipherstash-proxy`'s `SchemaManager` via the `eql-bindings` inventory; the encrypt + config is demoted to a cross-check. `eql-mapper` stays wire-format-agnostic and takes no + dependency on `eql-bindings`. +- If a future capability genuinely needs cross-column token-type checking (none does + today), this decision is the thing to revisit. From 2cc454a08593347ad60f3ccfd0028db8cabfccb8 Mon Sep 17 00:00:00 2001 From: James Sadler Date: Wed, 22 Jul 2026 00:15:47 +1000 Subject: [PATCH 03/10] fix(eql-mapper): correct EqlTraits set difference and bound-error operands Two latent bugs in the trait-bound machinery, harmless only while bound checking is dead code (every encrypted column currently gets EqlTraits::all()). Fix them before the EQL v3 loader makes bounds reachable. - EqlTraits::difference was implemented as XOR (symmetric difference), so UnsatisfiedBounds would list traits the type *has* alongside the ones it is missing. It is now a true set difference (self AND NOT other). - Type::must_implement passed its operands reversed relative to Unifier::satisfy_bounds (implemented.difference(required) instead of required.difference(implemented)). Harmless only because XOR is commutative; wrong the moment difference is corrected. Now consistent. Adds unit tests pinning the asymmetric set-difference semantics and that must_implement reports exactly the missing bounds. Refs CIP-3595. Stable-Commit-Id: q-0e9kbc03hrqef307fg7qb5sqj1 --- .../src/inference/unifier/eql_traits.rs | 86 +++++++++++++++++-- .../eql-mapper/src/inference/unifier/types.rs | 5 +- 2 files changed, 85 insertions(+), 6 deletions(-) diff --git a/packages/eql-mapper/src/inference/unifier/eql_traits.rs b/packages/eql-mapper/src/inference/unifier/eql_traits.rs index 25c4bec3..d54013aa 100644 --- a/packages/eql-mapper/src/inference/unifier/eql_traits.rs +++ b/packages/eql-mapper/src/inference/unifier/eql_traits.rs @@ -220,13 +220,19 @@ impl EqlTraits { } } + /// The set of traits in `self` that are not in `other` (set difference, + /// `self ∧ ¬other`). + /// + /// Used to report the bounds a type is *missing*: `required.difference(&implemented)`. + /// Not symmetric — do not swap the operands (see [`Type::must_implement`] and + /// [`Unifier::satisfy_bounds`], which must call it the same way around). pub(crate) fn difference(&self, other: &Self) -> Self { EqlTraits { - eq: self.eq ^ other.eq, - ord: self.ord ^ other.ord, - token_match: self.token_match ^ other.token_match, - json_like: self.json_like ^ other.json_like, - contain: self.contain ^ other.contain, + eq: self.eq && !other.eq, + ord: self.ord && !other.ord, + token_match: self.token_match && !other.token_match, + json_like: self.json_like && !other.json_like, + contain: self.contain && !other.contain, } } } @@ -408,3 +414,73 @@ impl EqlValue { ILIKE { expr: Self, pattern: Self::Tokenized, .. } -> Native; } */ + +#[cfg(test)] +mod test { + use super::*; + use crate::unifier::TableColumn; + use sqltk::parser::ast::Ident; + + #[test] + fn difference_is_asymmetric_set_difference() { + let eq = EqlTraits { + eq: true, + ..EqlTraits::none() + }; + let eq_ord = EqlTraits { + eq: true, + ord: true, + ..EqlTraits::none() + }; + let ord_only = EqlTraits { + ord: true, + ..EqlTraits::none() + }; + + // `A.difference(B)` is `A ∧ ¬B` — elements in A but not in B. + assert_eq!(eq_ord.difference(&eq), ord_only); + + // The discriminator against the old XOR (symmetric-difference) impl, + // which returned `{ord}` here instead of the empty set. + assert_eq!(eq.difference(&eq_ord), EqlTraits::none()); + + // Difference with self is always empty. + assert_eq!(eq_ord.difference(&eq_ord), EqlTraits::none()); + } + + #[test] + fn must_implement_reports_only_the_missing_bounds() { + // A column that implements TokenMatch but not Eq. + let ty = Type::Value(Value::Eql(EqlTerm::Full(EqlValue( + TableColumn { + table: Ident::new("t"), + column: Ident::new("c"), + }, + EqlTraits { + token_match: true, + ..EqlTraits::none() + }, + )))); + + let eq = EqlTraits { + eq: true, + ..EqlTraits::none() + }; + + // Requiring a bound the type already has succeeds. + assert!(ty + .must_implement(&EqlTraits { + token_match: true, + ..EqlTraits::none() + }) + .is_ok()); + + // Requiring Eq fails, and the reported set is exactly the missing bound + // (`{eq}`). The old reversed-operand + XOR bug reported `{eq, token_match}` + // — i.e. it listed a trait the type *has* but the bound never required. + match ty.must_implement(&eq) { + Err(TypeError::UnsatisfiedBounds(_, missing)) => assert_eq!(missing, eq), + other => panic!("expected UnsatisfiedBounds, got {other:?}"), + } + } +} diff --git a/packages/eql-mapper/src/inference/unifier/types.rs b/packages/eql-mapper/src/inference/unifier/types.rs index 8bff0355..306006a6 100644 --- a/packages/eql-mapper/src/inference/unifier/types.rs +++ b/packages/eql-mapper/src/inference/unifier/types.rs @@ -454,7 +454,10 @@ impl Type { } else { Err(TypeError::UnsatisfiedBounds( Arc::new(self.clone()), - self.effective_bounds().difference(bounds), + // Report the *missing* bounds: required (`bounds`) minus implemented + // (`self.effective_bounds()`). Operand order must match + // `Unifier::satisfy_bounds`. + bounds.difference(&self.effective_bounds()), )) } } From 493cdc60c92a58b056416638ca45eadd98a289eb Mon Sep 17 00:00:00 2001 From: James Sadler Date: Wed, 22 Jul 2026 00:29:03 +1000 Subject: [PATCH 04/10] feat(eql-mapper): carry inert v3 domain identity in the type carriers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Groundwork for the EQL v3 type checker (ADR-0002). Introduces the token type and its inert per-column domain identity, threading it through the type carriers without making it a checked dimension of unification. - New `TokenType` (integer/text/timestamp/…) and `DomainIdentity` (token type + v3 domain typname, e.g. eql_v3_text_ord_ore). - `EqlValue` widens to (TableColumn, Option, EqlTraits); `ColumnKind::Eql`, `SchemaTableColumn` and the new `Column::eql_with_identity` carry the identity. `Column::eql` stays as a back-compat constructor that defaults the identity to None. - The identity is `None` everywhere for now: the branch still emits the v2 surface and no loader populates it yet (CIP-3598 does). It rides through unification and the associated-type machinery untouched, exactly because it is never inspected there. - ColumnKind loses `Copy` (DomainIdentity is not Copy); the handful of by-value uses now clone. - Also fixes the never-invoked, latently-broken no-bounds arm of the concrete_ty!/EQL() macro (wrong EqlValue arity, missing .into()). eql-mapper suite 79 passing; workspace checks clean; fmt + clippy clean. Refs CIP-3597. Stable-Commit-Id: q-5dxadzsggt5seabqhg3sdj8ht5 --- .../eql-mapper-macros/src/parse_type_decl.rs | 8 ++- .../infer_type_impls/insert_statement.rs | 6 +- .../src/inference/unifier/eql_traits.rs | 1 + .../src/inference/unifier/type_env.rs | 1 + .../eql-mapper/src/inference/unifier/types.rs | 58 +++++++++++++++++-- packages/eql-mapper/src/lib.rs | 10 ++++ packages/eql-mapper/src/model/schema.rs | 28 +++++++-- packages/eql-mapper/src/model/schema_delta.rs | 10 ++-- packages/eql-mapper/src/test_helpers.rs | 4 +- 9 files changed, 102 insertions(+), 24 deletions(-) diff --git a/packages/eql-mapper-macros/src/parse_type_decl.rs b/packages/eql-mapper-macros/src/parse_type_decl.rs index d6d8eefb..7861b116 100644 --- a/packages/eql-mapper-macros/src/parse_type_decl.rs +++ b/packages/eql-mapper-macros/src/parse_type_decl.rs @@ -310,6 +310,7 @@ impl Parse for EqlTerm { table: #table.into(), column: #column.into(), }, + None, #bounds, ), ) @@ -319,11 +320,12 @@ impl Parse for EqlTerm { crate::inference::unifier::EqlTerm::Full( crate::inference::unifier::EqlValue( crate::inference::unifier::TableColumn { - table: #table, - column: #column + table: #table.into(), + column: #column.into(), }, + None, + crate::inference::unifier::EqlTraits::none(), ), - crate::inference::unifier::EqlTraits::none(), ) })) } diff --git a/packages/eql-mapper/src/inference/infer_type_impls/insert_statement.rs b/packages/eql-mapper/src/inference/infer_type_impls/insert_statement.rs index 81562849..b1a0ff50 100644 --- a/packages/eql-mapper/src/inference/infer_type_impls/insert_statement.rs +++ b/packages/eql-mapper/src/inference/infer_type_impls/insert_statement.rs @@ -44,9 +44,9 @@ impl<'ast> InferType<'ast, Insert> for TypeInferencer<'ast> { let value_ty = match &stc.kind { ColumnKind::Native => Value::Native(NativeValue(Some(tc.clone()))), - ColumnKind::Eql(features) => { - Value::Eql(EqlTerm::Full(EqlValue(tc.clone(), *features))) - } + ColumnKind::Eql(features, identity) => Value::Eql(EqlTerm::Full( + EqlValue(tc.clone(), identity.clone(), *features), + )), }; (Arc::new(Type::Value(value_ty)), Some(tc.column.clone())) diff --git a/packages/eql-mapper/src/inference/unifier/eql_traits.rs b/packages/eql-mapper/src/inference/unifier/eql_traits.rs index d54013aa..4593b569 100644 --- a/packages/eql-mapper/src/inference/unifier/eql_traits.rs +++ b/packages/eql-mapper/src/inference/unifier/eql_traits.rs @@ -456,6 +456,7 @@ mod test { table: Ident::new("t"), column: Ident::new("c"), }, + None, EqlTraits { token_match: true, ..EqlTraits::none() diff --git a/packages/eql-mapper/src/inference/unifier/type_env.rs b/packages/eql-mapper/src/inference/unifier/type_env.rs index e5a41532..caad2219 100644 --- a/packages/eql-mapper/src/inference/unifier/type_env.rs +++ b/packages/eql-mapper/src/inference/unifier/type_env.rs @@ -271,6 +271,7 @@ mod test { table: "customer".into(), column: "name".into() }, + None, EqlTraits::from(EqlTrait::JsonLike) ),)))) ); diff --git a/packages/eql-mapper/src/inference/unifier/types.rs b/packages/eql-mapper/src/inference/unifier/types.rs index 306006a6..444e0253 100644 --- a/packages/eql-mapper/src/inference/unifier/types.rs +++ b/packages/eql-mapper/src/inference/unifier/types.rs @@ -269,9 +269,51 @@ pub struct TableColumn { pub column: Ident, } +/// The plaintext scalar half of a v3 domain — the "token type". +/// +/// Crossed with a capability suffix (`_eq`, `_ord`, …) it names a v3 domain, +/// e.g. `text` + `_ord_ore` ⇒ `eql_v3_text_ord_ore`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Display)] +pub enum TokenType { + SmallInt, + Integer, + BigInt, + Real, + Double, + Numeric, + Text, + Boolean, + Date, + Timestamp, + Json, +} + +/// The inert `(token type, v3 domain)` an encrypted column carries (ADR-0002). +/// +/// Populated by the schema loader from the Postgres domain name; **never** a +/// checked dimension of unification. It is read only at rewrite time — to name +/// the cast target and to select the term-extraction-function variant +/// (`ord_term` vs `ord_term_ore`) — so it threads through unification and the +/// associated-type machinery untouched. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Display)] +#[display("{}", domain)] +pub struct DomainIdentity { + pub token: TokenType, + /// The v3 domain typname, e.g. `eql_v3_text_ord_ore`. + pub domain: Ident, +} + +/// The identity of an encrypted column: its `TableColumn`, its inert +/// [`DomainIdentity`] (see ADR-0002), and its [`EqlTraits`] capabilities. +/// +/// The domain identity is `None` while the mapper still emits the EQL v2 +/// surface; the v3 schema loader (CIP-3598) populates it. It is deliberately +/// not part of `PartialEq`/`Ord`-driven unification — two encrypted columns +/// never share a type because their `TableColumn`s differ, so the identity +/// never decides unification even though it is compared here. #[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone, Display, Hash)] #[display("EQL({})", _0)] -pub struct EqlValue(pub TableColumn, pub EqlTraits); +pub struct EqlValue(pub TableColumn, pub Option, pub EqlTraits); #[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone, Display, Hash)] #[display("{}", _0.as_ref().map(|tc| format!("Native({tc})")).unwrap_or(String::from("Native")))] @@ -468,8 +510,14 @@ impl EqlValue { &self.0 } + /// The inert v3 domain identity, if the schema loader has populated it. + /// `None` while the mapper still emits the EQL v2 surface. + pub fn domain_identity(&self) -> Option<&DomainIdentity> { + self.1.as_ref() + } + pub fn trait_impls(&self) -> EqlTraits { - self.1 + self.2 } } @@ -515,9 +563,9 @@ impl Projection { let value_ty = match &col.kind { ColumnKind::Native => Type::Value(Value::Native(NativeValue(Some(tc)))), - ColumnKind::Eql(features) => { - Type::Value(Value::Eql(EqlTerm::Full(EqlValue(tc, *features)))) - } + ColumnKind::Eql(features, identity) => Type::Value(Value::Eql( + EqlTerm::Full(EqlValue(tc, identity.clone(), *features)), + )), }; ProjectionColumn::new(value_ty, Some(col.name.clone())) diff --git a/packages/eql-mapper/src/lib.rs b/packages/eql-mapper/src/lib.rs index 09afc9a1..e628cf56 100644 --- a/packages/eql-mapper/src/lib.rs +++ b/packages/eql-mapper/src/lib.rs @@ -110,6 +110,7 @@ mod test { table: id("users"), column: id("email"), }, + None, EqlTraits::from(EqlTrait::Eq) ),), &ast::Value::SingleQuotedString("hello@cipherstash.com".into()), @@ -143,6 +144,7 @@ mod test { table: id("users"), column: id("email") }, + None, EqlTraits::default() )), &ast::Value::SingleQuotedString("hello@cipherstash.com".into()), @@ -175,6 +177,7 @@ mod test { table: id("users"), column: id("email") }, + None, EqlTraits::default() )), &ast::Value::SingleQuotedString("hello@cipherstash.com".into()), @@ -208,6 +211,7 @@ mod test { table: id("users"), column: id("email") }, + None, EqlTraits::default() )), &ast::Value::SingleQuotedString("hello@cipherstash.com".into()), @@ -549,6 +553,7 @@ mod test { table: id("users"), column: id("email"), }, + None, EqlTraits::default(), ))); @@ -557,6 +562,7 @@ mod test { table: id("users"), column: id("first_name"), }, + None, EqlTraits::default(), ))); @@ -597,6 +603,7 @@ mod test { table: id("users"), column: id("salary"), }, + None, EqlTraits::from(EqlTrait::Ord), ))); @@ -605,6 +612,7 @@ mod test { table: id("users"), column: id("age"), }, + None, EqlTraits::from(EqlTrait::Ord), ))); @@ -1118,6 +1126,7 @@ mod test { table: id("employees"), column: id("salary") }, + None, EqlTraits::from(EqlTrait::Ord) ),), &ast::Value::Number(200000.into(), false), @@ -1168,6 +1177,7 @@ mod test { table: id("employees"), column: id("salary") }, + None, EqlTraits::default() )), &ast::Value::Number(20000.into(), false) diff --git a/packages/eql-mapper/src/model/schema.rs b/packages/eql-mapper/src/model/schema.rs index a06cd1ea..2b7ac98f 100644 --- a/packages/eql-mapper/src/model/schema.rs +++ b/packages/eql-mapper/src/model/schema.rs @@ -3,7 +3,10 @@ //! Column type information is unused currently. use super::ident_case::*; -use crate::{iterator_ext::IteratorExt, unifier::EqlTraits}; +use crate::{ + iterator_ext::IteratorExt, + unifier::{DomainIdentity, EqlTraits}, +}; use core::fmt::Debug; use derive_more::Display; use sqltk::parser::ast::{Ident, ObjectName, ObjectNamePart}; @@ -38,17 +41,30 @@ pub struct Column { pub kind: ColumnKind, } -#[derive(Debug, Clone, Copy, PartialEq, Eq, Display, Hash)] +#[derive(Debug, Clone, PartialEq, Eq, Display, Hash)] pub enum ColumnKind { + #[display("Native")] Native, - Eql(EqlTraits), + /// An encrypted column: its capabilities, plus the inert v3 domain identity + /// (see ADR-0002). The identity is `None` while the mapper still emits the + /// EQL v2 surface; the v3 schema loader (CIP-3598) populates it. + #[display("Eql({})", _0)] + Eql(EqlTraits, Option), } impl Column { pub fn eql(name: Ident, features: EqlTraits) -> Self { + Self::eql_with_identity(name, features, None) + } + + pub fn eql_with_identity( + name: Ident, + features: EqlTraits, + identity: Option, + ) -> Self { Self { name, - kind: ColumnKind::Eql(features), + kind: ColumnKind::Eql(features, identity), } } @@ -123,7 +139,7 @@ impl Schema { .map(|col| SchemaTableColumn { table: table.name.clone(), column: col.name.clone(), - kind: col.kind, + kind: col.kind.clone(), }) .collect()) } @@ -143,7 +159,7 @@ impl Schema { Ok(column) => Ok(SchemaTableColumn { table: table.name.clone(), column: column.name.clone(), - kind: column.kind, + kind: column.kind.clone(), }), Err(_) => Err(SchemaError::ColumnNotFound( table_name.to_string(), diff --git a/packages/eql-mapper/src/model/schema_delta.rs b/packages/eql-mapper/src/model/schema_delta.rs index 24f7d190..94cd3089 100644 --- a/packages/eql-mapper/src/model/schema_delta.rs +++ b/packages/eql-mapper/src/model/schema_delta.rs @@ -78,7 +78,7 @@ impl SchemaWithEdits { .map(|col| SchemaTableColumn { table: table.name.clone(), column: col.name.clone(), - kind: col.kind, + kind: col.kind.clone(), }) .collect()) } @@ -100,7 +100,7 @@ impl SchemaWithEdits { Ok(SchemaTableColumn { table: table_name.clone(), column: column_name.clone(), - kind: col.kind, + kind: col.kind.clone(), }) } None => Err(SchemaError::ColumnNotFound( @@ -444,7 +444,7 @@ mod test { Ok(SchemaTableColumn { table: id("users"), column: id("primary_email"), - kind: ColumnKind::Eql(EqlTraits::default()) + kind: ColumnKind::Eql(EqlTraits::default(), None) }) ) } @@ -476,7 +476,7 @@ mod test { Ok(SchemaTableColumn { table: id("app_users"), column: id("email"), - kind: ColumnKind::Eql(EqlTraits::default()) + kind: ColumnKind::Eql(EqlTraits::default(), None) }) ) } @@ -526,7 +526,7 @@ mod test { Ok(SchemaTableColumn { table: id("users"), column: id("email"), - kind: ColumnKind::Eql(EqlTraits::default()) + kind: ColumnKind::Eql(EqlTraits::default(), None) }) ); diff --git a/packages/eql-mapper/src/test_helpers.rs b/packages/eql-mapper/src/test_helpers.rs index 4ff254f8..2f702351 100644 --- a/packages/eql-mapper/src/test_helpers.rs +++ b/packages/eql-mapper/src/test_helpers.rs @@ -145,7 +145,7 @@ macro_rules! col { ty: Arc::new(Type::Value(Value::Eql(EqlTerm::Full(EqlValue(TableColumn { table: id(stringify!($table)), column: id(stringify!($column)), - }, $crate::to_eql_traits!($($($eql_traits)*)?)))))), + }, None, $crate::to_eql_traits!($($($eql_traits)*)?)))))), alias: None, } }; @@ -155,7 +155,7 @@ macro_rules! col { ty: Arc::new(Type::Value(Value::Eql(EqlTerm::Full(EqlValue(TableColumn { table: id(stringify!($table)), column: id(stringify!($column)), - }, $crate::to_eql_traits!($($($eql_traits)*)?)))))), + }, None, $crate::to_eql_traits!($($($eql_traits)*)?)))))), alias: Some(id(stringify!($alias))), } }; From 26dd7b7b82bfb64dd0358883c657c9bba9a84d85 Mon Sep 17 00:00:00 2001 From: James Sadler Date: Wed, 22 Jul 2026 00:43:09 +1000 Subject: [PATCH 05/10] feat(proxy): load EQL v3 per-column capability from the domain name Implements the SchemaManager side of the v3 type checker (ADR-0002): per-column capability becomes *observed* from the Postgres domain name instead of a hardcoded EqlTraits::all(). - New schema/eql_domains.rs inverts the eql-bindings v3 catalog (v3::all()) into `typname -> (TokenType, EqlTraits)`. Because the catalog is generated from the same source as the installed SQL domains, the mapping cannot drift from them. Term -> capability: hm->Eq, op/ob->Ord, bf->TokenMatch, empty->storage-only; the JSON SteVec domains (term_json_keys == None) map to JsonLike (provisional, see note). - load_schema now resolves each column's v3 domain identity and traits and builds the column via Column::eql_with_identity. The legacy eql_v2_encrypted composite-type arm is retained for reading pre-v3 schemas; unrecognised domains load as Native. - select_table_schemas.sql now also selects information_schema domain_name: v3 encrypted columns are jsonb-backed DOMAINs, so udt_name reports the base type (jsonb) and the domain typname is only available via domain_name (cf. CIP-3441). - Adds eql-bindings 3.0.1 as a workspace dependency, used only by the proxy's schema loader; eql-mapper stays wire-format-agnostic. Re-exports DomainIdentity / TokenType from eql-mapper's crate root. Unit-tested offline against the catalog (12 tests asserting the SEM-term table, incl. the text hm+ord exception and query-twin exclusion). End-to-end validation needs a database with EQL v3 installed and is deferred; the encrypt-config cross-check is a follow-up within CIP-3598. Refs CIP-3598. Stable-Commit-Id: q-6p9ykbew5b6sjaf5p7y9qza4ft --- Cargo.lock | 118 +++++++++ Cargo.toml | 4 + packages/cipherstash-proxy/Cargo.toml | 1 + .../src/proxy/schema/eql_domains.rs | 229 ++++++++++++++++++ .../src/proxy/schema/manager.rs | 49 ++-- .../cipherstash-proxy/src/proxy/schema/mod.rs | 1 + .../src/proxy/sql/select_table_schemas.sql | 6 +- packages/eql-mapper/src/lib.rs | 4 +- 8 files changed, 394 insertions(+), 18 deletions(-) create mode 100644 packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs diff --git a/Cargo.lock b/Cargo.lock index e3cc5b16..28db8845 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -832,6 +832,7 @@ dependencies = [ "clap", "config", "cts-common", + "eql-bindings", "eql-mapper", "exitcode", "hex", @@ -1482,6 +1483,12 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813" +[[package]] +name = "dyn-clone" +version = "1.0.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" + [[package]] name = "either" version = "1.15.0" @@ -1512,6 +1519,18 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "eql-bindings" +version = "3.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39a6e26dbefb089908aa10521c6efffbe3f5404f9adfa7d43e4244d41c1d6db0" +dependencies = [ + "schemars", + "serde", + "serde_json", + "ts-rs", +] + [[package]] name = "eql-mapper" version = "1.0.0" @@ -3521,6 +3540,26 @@ dependencies = [ "thiserror 1.0.69", ] +[[package]] +name = "ref-cast" +version = "1.0.26" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "216e8f773d7923bcba9ceb86a86c93cabb3903a11872fc3f138c49630e50b96d" +dependencies = [ + "ref-cast-impl", +] + +[[package]] +name = "ref-cast-impl" +version = "1.0.26" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9283685feec7d69af75fb0e858d5e7378f33fe4fc699383b2916ab9273e03c" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.2", +] + [[package]] name = "regex" version = "1.11.1" @@ -3874,6 +3913,31 @@ dependencies = [ "windows-sys 0.59.0", ] +[[package]] +name = "schemars" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2b42f36aa1cd011945615b92222f6bf73c599a102a300334cd7f8dbeec726cc" +dependencies = [ + "dyn-clone", + "ref-cast", + "schemars_derive", + "serde", + "serde_json", +] + +[[package]] +name = "schemars_derive" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d115b50f4aaeea07e79c1912f645c7513d81715d0420f8bc77a18c6260b307f" +dependencies = [ + "proc-macro2", + "quote", + "serde_derive_internals", + "syn 2.0.117", +] + [[package]] name = "scoped-tls" version = "1.0.1" @@ -3976,6 +4040,17 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "serde_derive_internals" +version = "0.29.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "serde_html_form" version = "0.2.8" @@ -4340,6 +4415,17 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "syn" +version = "3.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a207d6d6a2b7fc470b80443726053f18a2481b7e1eee970597051596567987a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + [[package]] name = "sync_wrapper" version = "1.0.2" @@ -4402,6 +4488,15 @@ dependencies = [ "parking_lot", ] +[[package]] +name = "termcolor" +version = "1.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755" +dependencies = [ + "winapi-util", +] + [[package]] name = "thiserror" version = "1.0.69" @@ -4795,6 +4890,29 @@ version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" +[[package]] +name = "ts-rs" +version = "10.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e640d9b0964e9d39df633548591090ab92f7a4567bc31d3891af23471a3365c6" +dependencies = [ + "lazy_static", + "thiserror 2.0.18", + "ts-rs-macros", +] + +[[package]] +name = "ts-rs-macros" +version = "10.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e9d8656589772eeec2cf7a8264d9cda40fb28b9bc53118ceb9e8c07f8f38730" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", + "termcolor", +] + [[package]] name = "typenum" version = "1.20.1" diff --git a/Cargo.toml b/Cargo.toml index 8f747ece..e1888ff5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -46,6 +46,10 @@ sqltk = { version = "0.10.0" } cipherstash-client = { version = "=0.42.0" } cipherstash-config = { version = "=0.42.0" } cts-common = { version = "=0.42.0" } +# EQL v3 domain catalog: maps Postgres domain names to (token type, SEM terms). +# The authoritative source for a column's domain identity (ADR-0002); only the +# SchemaManager depends on it, keeping eql-mapper wire-format-agnostic. +eql-bindings = { version = "=3.0.1" } thiserror = "2.0.9" tokio = { version = "1.44.2", features = ["full"] } diff --git a/packages/cipherstash-proxy/Cargo.toml b/packages/cipherstash-proxy/Cargo.toml index 790cea6f..73dce1b9 100644 --- a/packages/cipherstash-proxy/Cargo.toml +++ b/packages/cipherstash-proxy/Cargo.toml @@ -21,6 +21,7 @@ config = { version = "0.15", features = [ "toml", ], default-features = false } cts-common = { workspace = true } +eql-bindings = { workspace = true } eql-mapper = { path = "../eql-mapper" } exitcode = "1.1.2" hex = "0.4.3" diff --git a/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs b/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs new file mode 100644 index 00000000..45f155d3 --- /dev/null +++ b/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs @@ -0,0 +1,229 @@ +//! Resolves EQL v3 Postgres domain typnames to the inert domain identity and +//! capabilities the type checker needs (ADR-0002). +//! +//! The mapping is inverted from the `eql-bindings` v3 catalog (`v3::all()`), +//! which is generated from the same source as the installed SQL domains, so it +//! cannot drift from them. `eql-mapper` stays wire-format-agnostic — this +//! `eql-bindings` dependency lives only in the proxy's schema loader. +//! +//! Term → capability mapping (from `eql-bindings` `v3::terms`): +//! - `hm` (HMAC-256) → `Eq` +//! - `op` (CLLW-OPE) → `Ord` +//! - `ob` (block-ORE) → `Ord` +//! - `bf` (bloom) → `TokenMatch` +//! - `c` / empty → storage-only (no capabilities) +//! +//! `term_json_keys() == None` marks the JSON SteVec domains +//! (`eql_v3_json_search`, `eql_v3_json_entry`), whose searchable terms live +//! per-entry rather than on the domain. Those are mapped to `JsonLike`. +//! +//! NOTE (CIP-3598): the precise capability set for the JSON SteVec domains was +//! not pinned down by the v3 type-checker ADRs; `JsonLike` is a provisional +//! choice pending that decision. + +use std::collections::HashMap; +use std::sync::OnceLock; + +use eql_bindings::v3; +use eql_mapper::{DomainIdentity, EqlTrait, EqlTraits, TokenType}; +use sqltk::parser::ast::Ident; + +/// The `eql_v3_` typname prefix every public-schema column domain carries. +const DOMAIN_PREFIX: &str = "eql_v3_"; + +/// `typname` (e.g. `eql_v3_integer_ord`) → (token type, capabilities), inverted +/// once from the `eql-bindings` catalog. +fn catalog() -> &'static HashMap { + static MAP: OnceLock> = OnceLock::new(); + MAP.get_or_init(|| { + let mut map = HashMap::new(); + for domain in v3::all() { + // `sql_domain()` is schema-qualified, e.g. `public.eql_v3_integer_ord`. + let qualified = domain.sql_domain(); + let typname = qualified.rsplit('.').next().unwrap_or(qualified); + + if let Some(token) = token_type(typname) { + let traits = traits_from_terms(domain.term_json_keys()); + map.insert(typname.to_string(), (token, traits)); + } + } + map + }) +} + +/// Resolve a Postgres domain typname to its inert v3 domain identity and +/// capabilities, or `None` if it is not a recognised v3 EQL domain. +pub(crate) fn resolve(typname: &str) -> Option<(DomainIdentity, EqlTraits)> { + let (token, traits) = catalog().get(typname).copied()?; + let identity = DomainIdentity { + token, + domain: Ident::new(typname), + }; + Some((identity, traits)) +} + +/// The token type is the first segment after the `eql_v3_` prefix; every token +/// type is a single underscore-free word, so the capability suffix (which may +/// itself contain underscores, e.g. `ord_ore`) never interferes. +fn token_type(typname: &str) -> Option { + let rest = typname.strip_prefix(DOMAIN_PREFIX)?; + let token = rest.split('_').next()?; + Some(match token { + "smallint" => TokenType::SmallInt, + "integer" => TokenType::Integer, + "bigint" => TokenType::BigInt, + "real" => TokenType::Real, + "double" => TokenType::Double, + "numeric" => TokenType::Numeric, + "text" => TokenType::Text, + "boolean" => TokenType::Boolean, + "date" => TokenType::Date, + "timestamp" => TokenType::Timestamp, + "json" => TokenType::Json, + _ => return None, + }) +} + +fn traits_from_terms(term_keys: Option<&[&str]>) -> EqlTraits { + match term_keys { + // JSON SteVec domains carry their terms per entry, not on the domain. + None => EqlTraits::from(EqlTrait::JsonLike), + Some(terms) => terms + .iter() + .filter_map(|term| match *term { + "hm" => Some(EqlTrait::Eq), + "op" | "ob" => Some(EqlTrait::Ord), + "bf" => Some(EqlTrait::TokenMatch), + // `c` is the storage-only source-ciphertext term; anything else + // is unknown and contributes no capability. + _ => None, + }) + .collect(), + } +} + +#[cfg(test)] +mod test { + use super::*; + + fn traits(typname: &str) -> EqlTraits { + resolve(typname) + .unwrap_or_else(|| panic!("{typname} did not resolve")) + .1 + } + + fn token(typname: &str) -> TokenType { + resolve(typname).unwrap().0.token + } + + #[test] + fn storage_only_domain_has_no_capabilities() { + assert_eq!(traits("eql_v3_integer"), EqlTraits::none()); + assert_eq!(traits("eql_v3_text"), EqlTraits::none()); + // boolean is storage-only by design (a two-value column leaks its + // distribution under any index). + assert_eq!(traits("eql_v3_boolean"), EqlTraits::none()); + } + + #[test] + fn eq_domain_implements_eq_only() { + assert_eq!(traits("eql_v3_integer_eq"), EqlTraits::from(EqlTrait::Eq)); + } + + #[test] + fn ord_domains_imply_eq() { + // `op` and `ob` both back Ord; Ord implies Eq. + let ord = EqlTraits::from(EqlTrait::Ord); + assert_eq!(traits("eql_v3_integer_ord"), ord); // op + assert_eq!(traits("eql_v3_integer_ord_ope"), ord); // op + assert_eq!(traits("eql_v3_integer_ord_ore"), ord); // ob + assert!(traits("eql_v3_integer_ord").eq); // Ord ⇒ Eq + } + + #[test] + fn match_domain_implements_token_match() { + assert_eq!( + traits("eql_v3_text_match"), + EqlTraits::from(EqlTrait::TokenMatch) + ); + } + + #[test] + fn text_ord_carries_the_hm_equality_exception() { + // Lexicographic ORE/OPE over text is not equality-lossless, so text_ord* + // stores `hm` alongside its ordering term — Eq is explicit, not merely + // implied. + let eq_ord: EqlTraits = [EqlTrait::Eq, EqlTrait::Ord].into_iter().collect(); + assert_eq!(traits("eql_v3_text_ord"), eq_ord); // [hm, op] + assert_eq!(traits("eql_v3_text_ord_ore"), eq_ord); // [hm, ob] + } + + #[test] + fn search_domains_implement_eq_ord_and_match() { + let all_three: EqlTraits = [EqlTrait::Eq, EqlTrait::Ord, EqlTrait::TokenMatch] + .into_iter() + .collect(); + assert_eq!(traits("eql_v3_text_search"), all_three); // [hm, op, bf] + assert_eq!(traits("eql_v3_text_search_ore"), all_three); // [hm, ob, bf] + } + + #[test] + fn json_search_domain_is_json_like() { + assert_eq!( + traits("eql_v3_json_search"), + EqlTraits::from(EqlTrait::JsonLike) + ); + } + + #[test] + fn token_type_is_parsed_across_suffixes() { + assert_eq!(token("eql_v3_integer_ord_ore"), TokenType::Integer); + assert_eq!(token("eql_v3_bigint_eq"), TokenType::BigInt); + assert_eq!(token("eql_v3_text_search"), TokenType::Text); + assert_eq!(token("eql_v3_timestamp_ord"), TokenType::Timestamp); + assert_eq!(token("eql_v3_json_search"), TokenType::Json); + } + + #[test] + fn domain_identity_carries_the_typname() { + let (identity, _) = resolve("eql_v3_integer_ord").unwrap(); + assert_eq!(identity.domain.value, "eql_v3_integer_ord"); + assert_eq!(identity.token, TokenType::Integer); + } + + #[test] + fn non_eql_domain_names_do_not_resolve() { + assert!(resolve("jsonb").is_none()); + assert!(resolve("text").is_none()); + assert!(resolve("eql_v2_encrypted").is_none()); + assert!(resolve("").is_none()); + } + + #[test] + fn every_public_column_domain_resolves_to_a_known_token_type() { + // Guards against a new token type being added upstream that this loader + // silently drops. Only `public.eql_v3_*` column domains are user-facing + // column types; the `eql_v3.query_*` operand twins that `all()` also + // yields (e.g. `eql_v3.query_json`) are deliberately not resolved. + let mut checked = 0; + for domain in v3::all() { + let qualified = domain.sql_domain(); + if let Some(typname) = qualified.strip_prefix("public.") { + assert!( + resolve(typname).is_some(), + "public column domain {typname} did not resolve to a token type" + ); + checked += 1; + } + } + assert!(checked > 0, "no public column domains found in the catalog"); + } + + #[test] + fn query_operand_twins_are_not_resolved_as_column_domains() { + // `all()` yields at least one `eql_v3.query_*` twin; those are operands, + // never column types, so they must not resolve here. + assert!(resolve("query_json").is_none()); + assert!(resolve("query_integer_eq").is_none()); + } +} diff --git a/packages/cipherstash-proxy/src/proxy/schema/manager.rs b/packages/cipherstash-proxy/src/proxy/schema/manager.rs index 5fa2710d..fe3605b2 100644 --- a/packages/cipherstash-proxy/src/proxy/schema/manager.rs +++ b/packages/cipherstash-proxy/src/proxy/schema/manager.rs @@ -1,3 +1,4 @@ +use super::eql_domains; use crate::config::DatabaseConfig; use crate::error::Error; use crate::proxy::{AGGREGATE_QUERY, SCHEMA_QUERY}; @@ -133,24 +134,42 @@ pub async fn load_schema(config: &DatabaseConfig) -> Result { let table_name: String = table.get("table_name"); let columns: Vec = table.get("columns"); let column_type_names: Vec> = table.get("column_type_names"); + let column_domain_names: Vec> = table.get("column_domain_names"); let mut table = Table::new(Ident::new(&table_name)); - columns.iter().zip(column_type_names).for_each(|(col, column_type_name)| { - let ident = Ident::with_quote('"', col); - - let column = match column_type_name.as_deref() { - Some("eql_v2_encrypted") => { - debug!(target: SCHEMA, msg = "eql_v2_encrypted column", table = table_name, column = col); - - let eql_traits = EqlTraits::all(); - Column::eql(ident, eql_traits) - } - _ => Column::native(ident), - }; - - table.add_column(Arc::new(column)); - }); + columns + .iter() + .zip(column_type_names) + .zip(column_domain_names) + .for_each(|((col, column_type_name), column_domain_name)| { + let ident = Ident::with_quote('"', col); + + // Prefer the v3 domain: encrypted columns are jsonb-backed + // DOMAINs whose typname encodes the token type and capabilities. + // The domain identity and traits are read from the eql-bindings + // catalog (ADR-0002); a domain we do not recognise is treated as + // a plaintext column. + let v3 = column_domain_name + .as_deref() + .and_then(eql_domains::resolve); + + let column = match (v3, column_type_name.as_deref()) { + (Some((identity, eql_traits)), _) => { + debug!(target: SCHEMA, msg = "eql_v3 column", table = table_name, column = col, domain = %identity.domain.value, traits = %eql_traits); + Column::eql_with_identity(ident, eql_traits, Some(identity)) + } + // Legacy EQL v2 columns are a composite type, reported by + // `udt_name`. Retained for reading pre-v3 schemas. + (None, Some("eql_v2_encrypted")) => { + debug!(target: SCHEMA, msg = "eql_v2_encrypted column", table = table_name, column = col); + Column::eql(ident, EqlTraits::all()) + } + _ => Column::native(ident), + }; + + table.add_column(Arc::new(column)); + }); schema.add_table(table); } diff --git a/packages/cipherstash-proxy/src/proxy/schema/mod.rs b/packages/cipherstash-proxy/src/proxy/schema/mod.rs index ca86225a..c34d83ce 100644 --- a/packages/cipherstash-proxy/src/proxy/schema/mod.rs +++ b/packages/cipherstash-proxy/src/proxy/schema/mod.rs @@ -1,3 +1,4 @@ +mod eql_domains; mod manager; pub use manager::SchemaManager; diff --git a/packages/cipherstash-proxy/src/proxy/sql/select_table_schemas.sql b/packages/cipherstash-proxy/src/proxy/sql/select_table_schemas.sql index afb7cb5a..469121c9 100644 --- a/packages/cipherstash-proxy/src/proxy/sql/select_table_schemas.sql +++ b/packages/cipherstash-proxy/src/proxy/sql/select_table_schemas.sql @@ -2,7 +2,11 @@ SELECT t.table_schema, t.table_name, array_agg(c.column_name)::text[] AS columns, - array_agg(c.udt_name)::text[] AS column_type_names + array_agg(c.udt_name)::text[] AS column_type_names, + -- EQL v3 encrypted columns are jsonb-backed DOMAINs, so `udt_name` reports + -- the base type (`jsonb`); the domain typname (e.g. `eql_v3_integer_ord`) + -- is only available via `domain_name`. NULL for non-domain columns. + array_agg(c.domain_name)::text[] AS column_domain_names FROM information_schema.tables t LEFT JOIN diff --git a/packages/eql-mapper/src/lib.rs b/packages/eql-mapper/src/lib.rs index e628cf56..bce428f6 100644 --- a/packages/eql-mapper/src/lib.rs +++ b/packages/eql-mapper/src/lib.rs @@ -21,8 +21,8 @@ pub use model::*; pub use param::*; pub use type_checked_statement::*; pub use unifier::{ - Array, AssociatedType, EqlTerm, EqlTermVariant, EqlTrait, EqlTraits, EqlValue, NativeValue, - Projection, ProjectionColumn, SetOf, TableColumn, Type, Value, + Array, AssociatedType, DomainIdentity, EqlTerm, EqlTermVariant, EqlTrait, EqlTraits, EqlValue, + NativeValue, Projection, ProjectionColumn, SetOf, TableColumn, TokenType, Type, Value, }; pub(crate) use dep::*; From 29ed557f04e30df243a30098a270adb8fac877a5 Mon Sep 17 00:00:00 2001 From: James Sadler Date: Wed, 22 Jul 2026 14:16:32 +1000 Subject: [PATCH 06/10] refactor(eql): make v3 domain identity non-optional; JSON = JsonLike + Contain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies three confirmed design decisions on the EQL v3 type checker. 1. Domain identity is non-optional (strict ADR-0002). EqlValue is now (TableColumn, DomainIdentity, EqlTraits) and ColumnKind::Eql carries a mandatory DomainIdentity — no more Option. The schema loader always supplies the real identity; there is no honest v3 identity for a legacy eql_v2_encrypted column, so the loader now drops the v2 arm and logs a warning instead of fabricating one (v2 is already retired on this build). - New helpers: TokenType::{as_domain_str, from_domain_name}, DomainIdentity::{from_domain_name, canonical}, and a test-only EqlValue::with_canonical_identity. - The schema!/concrete_ty!/test-helper macros synthesise a canonical text-token identity by default; schema! also gains an EQL("") form to pin an explicit v3 domain (token + OPE/ORE variant) for tests that care. 2. JSON domains map to JsonLike + Contain, verified against the installed v3 SQL (cipherstash-encrypt.sql): -> / ->> and @> / <@ are real operators on eql_v3_json_search (not raise-stubs). Corrects the earlier JsonLike-only guess. 3. The encrypt-config cross-check is dropped: the domain name is the sole authority for a column's capability. (ADR-0002 doc update to follow.) The Contain trait is therefore retained (scoped to JSON), NOT deleted as the handoff/glossary/ticket assumed — @>/<@ raise only on scalar encrypted columns. Docs/tickets updated separately. eql-mapper 79 passing; proxy eql_domains 12 passing; workspace check, fmt, clippy all clean. Refs CIP-3597, CIP-3598. Stable-Commit-Id: q-31sxsrtyrptajjpj0n383cb690 --- .../src/proxy/schema/eql_domains.rs | 83 +++++------- .../src/proxy/schema/manager.rs | 22 ++-- .../eql-mapper-macros/src/parse_type_decl.rs | 6 +- .../src/inference/unifier/eql_traits.rs | 23 ++-- .../src/inference/unifier/type_env.rs | 17 +-- .../eql-mapper/src/inference/unifier/types.rs | 118 ++++++++++++++++-- packages/eql-mapper/src/lib.rs | 30 ++--- packages/eql-mapper/src/model/schema.rs | 45 ++++--- packages/eql-mapper/src/model/schema_delta.rs | 17 ++- packages/eql-mapper/src/test_helpers.rs | 8 +- 10 files changed, 232 insertions(+), 137 deletions(-) diff --git a/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs b/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs index 45f155d3..bc38a2e8 100644 --- a/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs +++ b/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs @@ -15,26 +15,24 @@ //! //! `term_json_keys() == None` marks the JSON SteVec domains //! (`eql_v3_json_search`, `eql_v3_json_entry`), whose searchable terms live -//! per-entry rather than on the domain. Those are mapped to `JsonLike`. -//! -//! NOTE (CIP-3598): the precise capability set for the JSON SteVec domains was -//! not pinned down by the v3 type-checker ADRs; `JsonLike` is a provisional -//! choice pending that decision. +//! per-entry rather than on the domain. Verified against the installed v3 SQL +//! (`cipherstash-encrypt.sql`), an encrypted JSON column supports `->`/`->>` +//! (JsonLike) **and** `@>`/`<@` containment (Contain) — the latter are real +//! implementations, not the raise-stubs the other jsonb operators get. So JSON +//! domains map to `JsonLike + Contain`. (Note: `@>`/`<@` are removed only on +//! *scalar* encrypted columns; on JSON they are the primary query surface.) use std::collections::HashMap; use std::sync::OnceLock; use eql_bindings::v3; use eql_mapper::{DomainIdentity, EqlTrait, EqlTraits, TokenType}; -use sqltk::parser::ast::Ident; - -/// The `eql_v3_` typname prefix every public-schema column domain carries. -const DOMAIN_PREFIX: &str = "eql_v3_"; -/// `typname` (e.g. `eql_v3_integer_ord`) → (token type, capabilities), inverted -/// once from the `eql-bindings` catalog. -fn catalog() -> &'static HashMap { - static MAP: OnceLock> = OnceLock::new(); +/// `typname` (e.g. `eql_v3_integer_ord`) → capabilities, inverted once from the +/// `eql-bindings` catalog. Keyed only by the public column domains; the token +/// type is recovered from the typname via [`DomainIdentity::from_domain_name`]. +fn catalog() -> &'static HashMap { + static MAP: OnceLock> = OnceLock::new(); MAP.get_or_init(|| { let mut map = HashMap::new(); for domain in v3::all() { @@ -42,9 +40,13 @@ fn catalog() -> &'static HashMap { let qualified = domain.sql_domain(); let typname = qualified.rsplit('.').next().unwrap_or(qualified); - if let Some(token) = token_type(typname) { - let traits = traits_from_terms(domain.term_json_keys()); - map.insert(typname.to_string(), (token, traits)); + // Only public column domains have a parseable token type; this skips + // the `eql_v3.query_*` operand twins that `all()` also yields. + if TokenType::from_domain_name(typname).is_some() { + map.insert( + typname.to_string(), + traits_from_terms(domain.term_json_keys()), + ); } } map @@ -54,40 +56,19 @@ fn catalog() -> &'static HashMap { /// Resolve a Postgres domain typname to its inert v3 domain identity and /// capabilities, or `None` if it is not a recognised v3 EQL domain. pub(crate) fn resolve(typname: &str) -> Option<(DomainIdentity, EqlTraits)> { - let (token, traits) = catalog().get(typname).copied()?; - let identity = DomainIdentity { - token, - domain: Ident::new(typname), - }; + let traits = *catalog().get(typname)?; + let identity = DomainIdentity::from_domain_name(typname)?; Some((identity, traits)) } -/// The token type is the first segment after the `eql_v3_` prefix; every token -/// type is a single underscore-free word, so the capability suffix (which may -/// itself contain underscores, e.g. `ord_ore`) never interferes. -fn token_type(typname: &str) -> Option { - let rest = typname.strip_prefix(DOMAIN_PREFIX)?; - let token = rest.split('_').next()?; - Some(match token { - "smallint" => TokenType::SmallInt, - "integer" => TokenType::Integer, - "bigint" => TokenType::BigInt, - "real" => TokenType::Real, - "double" => TokenType::Double, - "numeric" => TokenType::Numeric, - "text" => TokenType::Text, - "boolean" => TokenType::Boolean, - "date" => TokenType::Date, - "timestamp" => TokenType::Timestamp, - "json" => TokenType::Json, - _ => return None, - }) -} - fn traits_from_terms(term_keys: Option<&[&str]>) -> EqlTraits { match term_keys { - // JSON SteVec domains carry their terms per entry, not on the domain. - None => EqlTraits::from(EqlTrait::JsonLike), + // JSON SteVec domains: `->`/`->>` (JsonLike) plus `@>`/`<@` containment + // (Contain), per the installed v3 SQL. The per-entry terms (hm/op) are + // not column-level capabilities. + None => [EqlTrait::JsonLike, EqlTrait::Contain] + .into_iter() + .collect(), Some(terms) => terms .iter() .filter_map(|term| match *term { @@ -168,11 +149,13 @@ mod test { } #[test] - fn json_search_domain_is_json_like() { - assert_eq!( - traits("eql_v3_json_search"), - EqlTraits::from(EqlTrait::JsonLike) - ); + fn json_search_domain_is_json_like_and_contain() { + // Verified against cipherstash-encrypt.sql: -> / ->> (JsonLike) and + // @> / <@ (Contain) are real operators on eql_v3_json_search. + let json_caps: EqlTraits = [EqlTrait::JsonLike, EqlTrait::Contain] + .into_iter() + .collect(); + assert_eq!(traits("eql_v3_json_search"), json_caps); } #[test] diff --git a/packages/cipherstash-proxy/src/proxy/schema/manager.rs b/packages/cipherstash-proxy/src/proxy/schema/manager.rs index fe3605b2..cbd0300a 100644 --- a/packages/cipherstash-proxy/src/proxy/schema/manager.rs +++ b/packages/cipherstash-proxy/src/proxy/schema/manager.rs @@ -4,7 +4,6 @@ use crate::error::Error; use crate::proxy::{AGGREGATE_QUERY, SCHEMA_QUERY}; use crate::{connect, log::SCHEMA}; use arc_swap::ArcSwap; -use eql_mapper::{self, EqlTraits}; use eql_mapper::{Column, Schema, Table}; use sqltk::parser::ast::Ident; use std::sync::Arc; @@ -154,18 +153,21 @@ pub async fn load_schema(config: &DatabaseConfig) -> Result { .as_deref() .and_then(eql_domains::resolve); - let column = match (v3, column_type_name.as_deref()) { - (Some((identity, eql_traits)), _) => { + let column = match v3 { + Some((identity, eql_traits)) => { debug!(target: SCHEMA, msg = "eql_v3 column", table = table_name, column = col, domain = %identity.domain.value, traits = %eql_traits); - Column::eql_with_identity(ident, eql_traits, Some(identity)) + Column::eql(ident, eql_traits, identity) } - // Legacy EQL v2 columns are a composite type, reported by - // `udt_name`. Retained for reading pre-v3 schemas. - (None, Some("eql_v2_encrypted")) => { - debug!(target: SCHEMA, msg = "eql_v2_encrypted column", table = table_name, column = col); - Column::eql(ident, EqlTraits::all()) + None => { + // Legacy EQL v2 columns (the `eql_v2_encrypted` composite + // type) have no v3 domain identity and are unsupported on + // this v3-only build — warn rather than silently treating + // them as encrypted or plaintext. + if column_type_name.as_deref() == Some("eql_v2_encrypted") { + warn!(target: SCHEMA, msg = "ignoring unsupported eql_v2_encrypted column on a v3 build", table = table_name, column = col); + } + Column::native(ident) } - _ => Column::native(ident), }; table.add_column(Arc::new(column)); diff --git a/packages/eql-mapper-macros/src/parse_type_decl.rs b/packages/eql-mapper-macros/src/parse_type_decl.rs index 7861b116..29786bb0 100644 --- a/packages/eql-mapper-macros/src/parse_type_decl.rs +++ b/packages/eql-mapper-macros/src/parse_type_decl.rs @@ -305,12 +305,11 @@ impl Parse for EqlTerm { Ok(Self(quote! { crate::inference::unifier::EqlTerm::Full( - crate::inference::unifier::EqlValue( + crate::inference::unifier::EqlValue::with_canonical_identity( crate::inference::unifier::TableColumn { table: #table.into(), column: #column.into(), }, - None, #bounds, ), ) @@ -318,12 +317,11 @@ impl Parse for EqlTerm { } else { Ok(Self(quote! { crate::inference::unifier::EqlTerm::Full( - crate::inference::unifier::EqlValue( + crate::inference::unifier::EqlValue::with_canonical_identity( crate::inference::unifier::TableColumn { table: #table.into(), column: #column.into(), }, - None, crate::inference::unifier::EqlTraits::none(), ), ) diff --git a/packages/eql-mapper/src/inference/unifier/eql_traits.rs b/packages/eql-mapper/src/inference/unifier/eql_traits.rs index 4593b569..10ad71a9 100644 --- a/packages/eql-mapper/src/inference/unifier/eql_traits.rs +++ b/packages/eql-mapper/src/inference/unifier/eql_traits.rs @@ -451,17 +451,18 @@ mod test { #[test] fn must_implement_reports_only_the_missing_bounds() { // A column that implements TokenMatch but not Eq. - let ty = Type::Value(Value::Eql(EqlTerm::Full(EqlValue( - TableColumn { - table: Ident::new("t"), - column: Ident::new("c"), - }, - None, - EqlTraits { - token_match: true, - ..EqlTraits::none() - }, - )))); + let ty = Type::Value(Value::Eql(EqlTerm::Full( + EqlValue::with_canonical_identity( + TableColumn { + table: Ident::new("t"), + column: Ident::new("c"), + }, + EqlTraits { + token_match: true, + ..EqlTraits::none() + }, + ), + ))); let eq = EqlTraits { eq: true, diff --git a/packages/eql-mapper/src/inference/unifier/type_env.rs b/packages/eql-mapper/src/inference/unifier/type_env.rs index caad2219..78248781 100644 --- a/packages/eql-mapper/src/inference/unifier/type_env.rs +++ b/packages/eql-mapper/src/inference/unifier/type_env.rs @@ -266,14 +266,15 @@ mod test { if let Type::Associated(associated) = &*instance.get_type(&tvar!(A))? { assert_eq!( associated.resolve_selector_target(&mut unifier)?.as_deref(), - Some(&Type::Value(Value::Eql(EqlTerm::JsonAccessor(EqlValue( - TableColumn { - table: "customer".into(), - column: "name".into() - }, - None, - EqlTraits::from(EqlTrait::JsonLike) - ),)))) + Some(&Type::Value(Value::Eql(EqlTerm::JsonAccessor( + EqlValue::with_canonical_identity( + TableColumn { + table: "customer".into(), + column: "name".into() + }, + EqlTraits::from(EqlTrait::JsonLike) + ), + )))) ); } else { panic!("expected associated type"); diff --git a/packages/eql-mapper/src/inference/unifier/types.rs b/packages/eql-mapper/src/inference/unifier/types.rs index 444e0253..4bb9abe5 100644 --- a/packages/eql-mapper/src/inference/unifier/types.rs +++ b/packages/eql-mapper/src/inference/unifier/types.rs @@ -288,6 +288,47 @@ pub enum TokenType { Json, } +impl TokenType { + /// The token type's spelling inside a v3 domain typname + /// (`eql_v3__`). + pub fn as_domain_str(&self) -> &'static str { + match self { + TokenType::SmallInt => "smallint", + TokenType::Integer => "integer", + TokenType::BigInt => "bigint", + TokenType::Real => "real", + TokenType::Double => "double", + TokenType::Numeric => "numeric", + TokenType::Text => "text", + TokenType::Boolean => "boolean", + TokenType::Date => "date", + TokenType::Timestamp => "timestamp", + TokenType::Json => "json", + } + } + + /// Parse the token type from a v3 domain typname. The token type is the + /// first segment after the `eql_v3_` prefix; every token type is a single + /// underscore-free word, so a multi-part capability suffix never interferes. + pub fn from_domain_name(domain: &str) -> Option { + let rest = domain.strip_prefix("eql_v3_")?; + Some(match rest.split('_').next()? { + "smallint" => TokenType::SmallInt, + "integer" => TokenType::Integer, + "bigint" => TokenType::BigInt, + "real" => TokenType::Real, + "double" => TokenType::Double, + "numeric" => TokenType::Numeric, + "text" => TokenType::Text, + "boolean" => TokenType::Boolean, + "date" => TokenType::Date, + "timestamp" => TokenType::Timestamp, + "json" => TokenType::Json, + _ => return None, + }) + } +} + /// The inert `(token type, v3 domain)` an encrypted column carries (ADR-0002). /// /// Populated by the schema loader from the Postgres domain name; **never** a @@ -303,17 +344,63 @@ pub struct DomainIdentity { pub domain: Ident, } +impl DomainIdentity { + /// Build an identity from a v3 domain typname, parsing the token type from + /// the name. The domain name is the authority (the schema loader passes the + /// real typname); returns `None` for a name that is not a v3 EQL domain. + pub fn from_domain_name(domain: &str) -> Option { + Some(Self { + token: TokenType::from_domain_name(domain)?, + domain: Ident::new(domain), + }) + } + + /// A canonical identity for a `(token, capabilities)` pair. This is a + /// **test/fixture convenience** for constructing identities where no live + /// schema loader supplies the real domain name — production identities always + /// come from [`Self::from_domain_name`] via the loader. The synthesised domain + /// name is deterministic so both sides of a test assertion agree; it is not + /// guaranteed to equal the exact catalog typname. + pub fn canonical(token: TokenType, traits: EqlTraits) -> Self { + let mut parts: Vec<&str> = Vec::new(); + if traits.json_like { + parts.push("search"); + } else { + if traits.ord { + parts.push("ord"); + } else if traits.eq { + parts.push("eq"); + } + if traits.token_match { + parts.push("match"); + } + if traits.contain { + parts.push("contain"); + } + } + let suffix = if parts.is_empty() { + String::new() + } else { + format!("_{}", parts.join("_")) + }; + let domain = format!("eql_v3_{}{}", token.as_domain_str(), suffix); + Self { + token, + domain: Ident::new(domain), + } + } +} + /// The identity of an encrypted column: its `TableColumn`, its inert /// [`DomainIdentity`] (see ADR-0002), and its [`EqlTraits`] capabilities. /// -/// The domain identity is `None` while the mapper still emits the EQL v2 -/// surface; the v3 schema loader (CIP-3598) populates it. It is deliberately -/// not part of `PartialEq`/`Ord`-driven unification — two encrypted columns -/// never share a type because their `TableColumn`s differ, so the identity -/// never decides unification even though it is compared here. +/// The domain identity is deliberately not part of `PartialEq`/`Ord`-driven +/// unification — two encrypted columns never share a type because their +/// `TableColumn`s differ, so the identity never decides unification even though +/// it is compared here. #[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone, Display, Hash)] #[display("EQL({})", _0)] -pub struct EqlValue(pub TableColumn, pub Option, pub EqlTraits); +pub struct EqlValue(pub TableColumn, pub DomainIdentity, pub EqlTraits); #[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone, Display, Hash)] #[display("{}", _0.as_ref().map(|tc| format!("Native({tc})")).unwrap_or(String::from("Native")))] @@ -510,10 +597,21 @@ impl EqlValue { &self.0 } - /// The inert v3 domain identity, if the schema loader has populated it. - /// `None` while the mapper still emits the EQL v2 surface. - pub fn domain_identity(&self) -> Option<&DomainIdentity> { - self.1.as_ref() + /// The inert v3 domain identity — names the cast target and selects the + /// term-extraction-function variant at rewrite time (ADR-0002). + pub fn domain_identity(&self) -> &DomainIdentity { + &self.1 + } + + /// Test/fixture constructor: builds the value with the canonical `text`-token + /// [`DomainIdentity`] for `traits`. Production values come from the schema + /// loader with the real domain identity, never this. + pub fn with_canonical_identity(table_column: TableColumn, traits: EqlTraits) -> Self { + Self( + table_column, + DomainIdentity::canonical(TokenType::Text, traits), + traits, + ) } pub fn trait_impls(&self) -> EqlTraits { diff --git a/packages/eql-mapper/src/lib.rs b/packages/eql-mapper/src/lib.rs index bce428f6..73df2541 100644 --- a/packages/eql-mapper/src/lib.rs +++ b/packages/eql-mapper/src/lib.rs @@ -105,12 +105,11 @@ mod test { assert_eq!( typed.literals, vec![( - EqlTerm::Full(EqlValue( + EqlTerm::Full(EqlValue::with_canonical_identity( TableColumn { table: id("users"), column: id("email"), }, - None, EqlTraits::from(EqlTrait::Eq) ),), &ast::Value::SingleQuotedString("hello@cipherstash.com".into()), @@ -139,12 +138,11 @@ mod test { match type_check(schema, &statement) { Ok(typed) => { assert!(typed.literals.contains(&( - EqlTerm::Full(EqlValue( + EqlTerm::Full(EqlValue::with_canonical_identity( TableColumn { table: id("users"), column: id("email") }, - None, EqlTraits::default() )), &ast::Value::SingleQuotedString("hello@cipherstash.com".into()), @@ -172,12 +170,11 @@ mod test { match type_check(schema, &statement) { Ok(typed) => { assert!(typed.literals.contains(&( - EqlTerm::Full(EqlValue( + EqlTerm::Full(EqlValue::with_canonical_identity( TableColumn { table: id("users"), column: id("email") }, - None, EqlTraits::default() )), &ast::Value::SingleQuotedString("hello@cipherstash.com".into()), @@ -206,12 +203,11 @@ mod test { match type_check(schema, &statement) { Ok(typed) => { assert!(typed.literals.contains(&( - EqlTerm::Full(EqlValue( + EqlTerm::Full(EqlValue::with_canonical_identity( TableColumn { table: id("users"), column: id("email") }, - None, EqlTraits::default() )), &ast::Value::SingleQuotedString("hello@cipherstash.com".into()), @@ -548,21 +544,19 @@ mod test { match type_check(schema, &statement) { Ok(typed) => { - let a = Value::Eql(EqlTerm::Full(EqlValue( + let a = Value::Eql(EqlTerm::Full(EqlValue::with_canonical_identity( TableColumn { table: id("users"), column: id("email"), }, - None, EqlTraits::default(), ))); - let b = Value::Eql(EqlTerm::Full(EqlValue( + let b = Value::Eql(EqlTerm::Full(EqlValue::with_canonical_identity( TableColumn { table: id("users"), column: id("first_name"), }, - None, EqlTraits::default(), ))); @@ -598,21 +592,19 @@ mod test { match type_check(schema, &statement) { Ok(typed) => { - let a = Value::Eql(EqlTerm::Full(EqlValue( + let a = Value::Eql(EqlTerm::Full(EqlValue::with_canonical_identity( TableColumn { table: id("users"), column: id("salary"), }, - None, EqlTraits::from(EqlTrait::Ord), ))); - let b = Value::Eql(EqlTerm::Full(EqlValue( + let b = Value::Eql(EqlTerm::Full(EqlValue::with_canonical_identity( TableColumn { table: id("users"), column: id("age"), }, - None, EqlTraits::from(EqlTrait::Ord), ))); @@ -1121,12 +1113,11 @@ mod test { assert_eq!( typed.literals, vec![( - EqlTerm::Full(EqlValue( + EqlTerm::Full(EqlValue::with_canonical_identity( TableColumn { table: id("employees"), column: id("salary") }, - None, EqlTraits::from(EqlTrait::Ord) ),), &ast::Value::Number(200000.into(), false), @@ -1172,12 +1163,11 @@ mod test { assert_eq!( typed.literals, vec![( - EqlTerm::Full(EqlValue( + EqlTerm::Full(EqlValue::with_canonical_identity( TableColumn { table: id("employees"), column: id("salary") }, - None, EqlTraits::default() )), &ast::Value::Number(20000.into(), false) diff --git a/packages/eql-mapper/src/model/schema.rs b/packages/eql-mapper/src/model/schema.rs index 2b7ac98f..0f038437 100644 --- a/packages/eql-mapper/src/model/schema.rs +++ b/packages/eql-mapper/src/model/schema.rs @@ -46,22 +46,14 @@ pub enum ColumnKind { #[display("Native")] Native, /// An encrypted column: its capabilities, plus the inert v3 domain identity - /// (see ADR-0002). The identity is `None` while the mapper still emits the - /// EQL v2 surface; the v3 schema loader (CIP-3598) populates it. + /// (see ADR-0002), which the v3 schema loader populates from the Postgres + /// domain name. #[display("Eql({})", _0)] - Eql(EqlTraits, Option), + Eql(EqlTraits, DomainIdentity), } impl Column { - pub fn eql(name: Ident, features: EqlTraits) -> Self { - Self::eql_with_identity(name, features, None) - } - - pub fn eql_with_identity( - name: Ident, - features: EqlTraits, - identity: Option, - ) -> Self { + pub fn eql(name: Ident, features: EqlTraits, identity: DomainIdentity) -> Self { Self { name, kind: ColumnKind::Eql(features, identity), @@ -289,11 +281,32 @@ macro_rules! schema { (@add_columns $table:ident $( $column_name:ident $(($($options:tt)+))? , )* ) => { $( schema!(@add_column $table $column_name $(($($options)*))? ); )* }; + // Explicit v3 domain typname, e.g. `col (EQL("eql_v3_integer_ord_ore"): Ord)`. + // The token type is parsed from the domain name; use this when a test cares + // about the token type or the OPE-vs-ORE variant. + (@add_column $table:ident $column_name:ident (EQL($domain:literal) $(: $trait_:ident $(+ $trait_rest:ident)*)?) ) => { + { + let __traits = $crate::to_eql_trait_impls!($($trait_ $($trait_rest)*)?); + $table.add_column(std::sync::Arc::new($crate::model::Column::eql( + ::sqltk::parser::ast::Ident::new(stringify!($column_name)), + __traits, + $crate::unifier::DomainIdentity::from_domain_name($domain) + .expect("EQL() must be a valid v3 domain typname"), + ))); + } + }; + // Default: no explicit domain — synthesise a canonical `text`-token identity + // for the given capabilities (fine for the many tests that don't exercise the + // token type). (@add_column $table:ident $column_name:ident (EQL $(: $trait_:ident $(+ $trait_rest:ident)*)?) ) => { - $table.add_column(std::sync::Arc::new($crate::model::Column::eql( - ::sqltk::parser::ast::Ident::new(stringify!($column_name)), - $crate::to_eql_trait_impls!($($trait_ $($trait_rest)*)?) - ))); + { + let __traits = $crate::to_eql_trait_impls!($($trait_ $($trait_rest)*)?); + $table.add_column(std::sync::Arc::new($crate::model::Column::eql( + ::sqltk::parser::ast::Ident::new(stringify!($column_name)), + __traits, + $crate::unifier::DomainIdentity::canonical($crate::unifier::TokenType::Text, __traits), + ))); + } }; (@add_column $table:ident $column_name:ident (PK) ) => { $table.add_column( diff --git a/packages/eql-mapper/src/model/schema_delta.rs b/packages/eql-mapper/src/model/schema_delta.rs index 94cd3089..47e7e88e 100644 --- a/packages/eql-mapper/src/model/schema_delta.rs +++ b/packages/eql-mapper/src/model/schema_delta.rs @@ -363,7 +363,7 @@ mod test { use crate::{ schema, test_helpers::{id, object_name, parse}, - unifier::EqlTraits, + unifier::{DomainIdentity, EqlTraits, TokenType}, ColumnKind, SchemaError, SchemaTableColumn, TableResolver, }; @@ -444,7 +444,10 @@ mod test { Ok(SchemaTableColumn { table: id("users"), column: id("primary_email"), - kind: ColumnKind::Eql(EqlTraits::default(), None) + kind: ColumnKind::Eql( + EqlTraits::default(), + DomainIdentity::canonical(TokenType::Text, EqlTraits::default()) + ) }) ) } @@ -476,7 +479,10 @@ mod test { Ok(SchemaTableColumn { table: id("app_users"), column: id("email"), - kind: ColumnKind::Eql(EqlTraits::default(), None) + kind: ColumnKind::Eql( + EqlTraits::default(), + DomainIdentity::canonical(TokenType::Text, EqlTraits::default()) + ) }) ) } @@ -526,7 +532,10 @@ mod test { Ok(SchemaTableColumn { table: id("users"), column: id("email"), - kind: ColumnKind::Eql(EqlTraits::default(), None) + kind: ColumnKind::Eql( + EqlTraits::default(), + DomainIdentity::canonical(TokenType::Text, EqlTraits::default()) + ) }) ); diff --git a/packages/eql-mapper/src/test_helpers.rs b/packages/eql-mapper/src/test_helpers.rs index 2f702351..c5565f69 100644 --- a/packages/eql-mapper/src/test_helpers.rs +++ b/packages/eql-mapper/src/test_helpers.rs @@ -142,20 +142,20 @@ macro_rules! col { ((EQL($table:ident . $column:ident $(: $($eql_traits:ident)*)?))) => { ProjectionColumn { - ty: Arc::new(Type::Value(Value::Eql(EqlTerm::Full(EqlValue(TableColumn { + ty: Arc::new(Type::Value(Value::Eql(EqlTerm::Full(EqlValue::with_canonical_identity(TableColumn { table: id(stringify!($table)), column: id(stringify!($column)), - }, None, $crate::to_eql_traits!($($($eql_traits)*)?)))))), + }, $crate::to_eql_traits!($($($eql_traits)*)?)))))), alias: None, } }; ((EQL($table:ident . $column:ident $(: $($eql_traits:ident)*)?) as $alias:ident)) => { ProjectionColumn { - ty: Arc::new(Type::Value(Value::Eql(EqlTerm::Full(EqlValue(TableColumn { + ty: Arc::new(Type::Value(Value::Eql(EqlTerm::Full(EqlValue::with_canonical_identity(TableColumn { table: id(stringify!($table)), column: id(stringify!($column)), - }, None, $crate::to_eql_traits!($($($eql_traits)*)?)))))), + }, $crate::to_eql_traits!($($($eql_traits)*)?)))))), alias: Some(id(stringify!($alias))), } }; From 76c6159e89486ce4e2bc948bd18b758925fbb861 Mon Sep 17 00:00:00 2001 From: James Sadler Date: Thu, 23 Jul 2026 16:03:23 +1000 Subject: [PATCH 07/10] docs(eql-mapper): warn that DomainIdentity::canonical may collide with a real domain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of the v3 domain-identity work (#424) noted that canonical() can synthesise a name that is not merely approximate but identical to an unrelated real catalog domain — canonical(Text, {json_like}) yields eql_v3_text_search, whose terms are hm/op/bf (Eq+Ord+TokenMatch), nothing to do with JSON. Strengthen the doc comment so no caller mistakes the synthesised name for the column's real domain; only from_domain_name is authoritative. Stable-Commit-Id: q-1ewbmpx74jnyxfd7dyftv08ka9 --- packages/eql-mapper/src/inference/unifier/types.rs | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/packages/eql-mapper/src/inference/unifier/types.rs b/packages/eql-mapper/src/inference/unifier/types.rs index 4bb9abe5..b55a784a 100644 --- a/packages/eql-mapper/src/inference/unifier/types.rs +++ b/packages/eql-mapper/src/inference/unifier/types.rs @@ -359,8 +359,17 @@ impl DomainIdentity { /// **test/fixture convenience** for constructing identities where no live /// schema loader supplies the real domain name — production identities always /// come from [`Self::from_domain_name`] via the loader. The synthesised domain - /// name is deterministic so both sides of a test assertion agree; it is not - /// guaranteed to equal the exact catalog typname. + /// name is deterministic so both sides of a test assertion agree. + /// + /// The synthesised name is NOT authoritative and must never be treated as the + /// column's real catalog domain: it may not only diverge from the real typname + /// but actively **collide with an unrelated real domain that means something + /// else**. For example `canonical(Text, {json_like})` produces + /// `eql_v3_text_search` — a real catalog domain whose terms are `[hm, op, bf]` + /// (Eq + Ord + TokenMatch), nothing to do with JSON — and it can equally emit + /// genuinely non-catalog names (`Eq + TokenMatch` → `eql_v3_text_eq_match`, + /// `Contain` → `eql_v3_text_contain`). Only [`Self::from_domain_name`] yields a + /// real domain identity. pub fn canonical(token: TokenType, traits: EqlTraits) -> Self { let mut parts: Vec<&str> = Vec::new(); if traits.json_like { From 6752d0ea94d0e2575bd172db15e080ee68ab6560 Mon Sep 17 00:00:00 2001 From: James Sadler Date: Thu, 23 Jul 2026 21:40:34 +1000 Subject: [PATCH 08/10] feat(proxy): make the v2-column plaintext warning ops-visible MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of #424 (finding 5) noted the v2->native fallback should be hard to miss in ops, because a v2 column served as a native passthrough takes no encryption on the write path: a plaintext value written to it is stored in plaintext (the eql_v2_encrypted AS ASSIGNMENT cast does no validation). Spell that out in the warning — the column is served as PLAINTEXT, Proxy does no encrypt/decrypt on it, migrate before writing — instead of the softer "ignoring unsupported column". Stable-Commit-Id: q-4dmrmqgewe5mzj0fzb2qdk5dja --- packages/cipherstash-proxy/src/proxy/schema/manager.rs | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/packages/cipherstash-proxy/src/proxy/schema/manager.rs b/packages/cipherstash-proxy/src/proxy/schema/manager.rs index cbd0300a..136db1f2 100644 --- a/packages/cipherstash-proxy/src/proxy/schema/manager.rs +++ b/packages/cipherstash-proxy/src/proxy/schema/manager.rs @@ -163,8 +163,14 @@ pub async fn load_schema(config: &DatabaseConfig) -> Result { // type) have no v3 domain identity and are unsupported on // this v3-only build — warn rather than silently treating // them as encrypted or plaintext. + // + // The column is served as a native passthrough: Proxy runs + // no encrypt/decrypt on it, so new writes are stored as-is + // (in plaintext) and existing values are returned as-is. + // This is a data-at-rest exposure on the write path, so the + // warning must be impossible to miss in ops. if column_type_name.as_deref() == Some("eql_v2_encrypted") { - warn!(target: SCHEMA, msg = "ignoring unsupported eql_v2_encrypted column on a v3 build", table = table_name, column = col); + warn!(target: SCHEMA, msg = "eql_v2_encrypted column is unsupported on this EQL v3 build and is being served as a PLAINTEXT (native) column: Proxy performs no encryption on writes or decryption on reads. Migrate the column to an EQL v3 domain before writing to it.", table = table_name, column = col); } Column::native(ident) } From 84c53189cb089ddf24be04d20a7cbd1f2ac39f31 Mon Sep 17 00:00:00 2001 From: James Sadler Date: Thu, 23 Jul 2026 17:43:21 +1000 Subject: [PATCH 09/10] fix(proxy): exclude eql_v3_json_entry from the column-domain catalog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of #424 (finding 1) flagged that `eql_v3_json_entry` — the element a `->` traversal returns, whose per-entry terms are `hm` XOR `op` — resolved to JsonLike+Contain, because `ste_vec_domain_type!` leaves `term_json_keys()` at None. A column mistakenly declared with that type would have type-checked as supporting `->`/`@>` but not `=`/`<`, which is inverted. Exclude the SteVec sub-structural domains (`NON_COLUMN_DOMAINS`) from the column catalog so such a column falls through to a native (plaintext) column instead of a wrong capability set. The every_public_column_domain assertion skips the same list. Makes the misclassification impossible rather than latent. Stable-Commit-Id: q-4dg749ehycbdazr4etn9rk1fvm --- .../src/proxy/schema/eql_domains.rs | 45 ++++++++++++++++++- 1 file changed, 43 insertions(+), 2 deletions(-) diff --git a/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs b/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs index bc38a2e8..917bba7f 100644 --- a/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs +++ b/packages/cipherstash-proxy/src/proxy/schema/eql_domains.rs @@ -28,6 +28,20 @@ use std::sync::OnceLock; use eql_bindings::v3; use eql_mapper::{DomainIdentity, EqlTrait, EqlTraits, TokenType}; +/// SteVec sub-structural domains that live in the `public` schema but are NOT +/// user-declarable column types, so they must never resolve to a column +/// capability set. +/// +/// `eql_v3_json_entry` is the element a `->` traversal returns (one `sv` array +/// entry); its per-entry terms are `hm` XOR `op` (equality/ordering), not the +/// `JsonLike + Contain` that a `term_json_keys() == None` JSON *document* domain +/// (`eql_v3_json_search`) carries. Left in the catalog it would type-check a +/// column mistakenly declared with that type as supporting `->`/`@>` but not +/// `=`/`<` — inverted. Excluding it makes that misclassification impossible +/// rather than merely latent: a column so declared falls through to a native +/// (plaintext) column instead. See PR #424 review. +const NON_COLUMN_DOMAINS: &[&str] = &["eql_v3_json_entry"]; + /// `typname` (e.g. `eql_v3_integer_ord`) → capabilities, inverted once from the /// `eql-bindings` catalog. Keyed only by the public column domains; the token /// type is recovered from the typname via [`DomainIdentity::from_domain_name`]. @@ -41,8 +55,12 @@ fn catalog() -> &'static HashMap { let typname = qualified.rsplit('.').next().unwrap_or(qualified); // Only public column domains have a parseable token type; this skips - // the `eql_v3.query_*` operand twins that `all()` also yields. - if TokenType::from_domain_name(typname).is_some() { + // the `eql_v3.query_*` operand twins that `all()` also yields. The + // `public` SteVec sub-structural domains (`eql_v3_json_entry`) parse + // but are not column types, so exclude them explicitly. + if TokenType::from_domain_name(typname).is_some() + && !NON_COLUMN_DOMAINS.contains(&typname) + { map.insert( typname.to_string(), traits_from_terms(domain.term_json_keys()), @@ -192,6 +210,11 @@ mod test { for domain in v3::all() { let qualified = domain.sql_domain(); if let Some(typname) = qualified.strip_prefix("public.") { + // SteVec sub-structural domains are `public` but not column + // types (see NON_COLUMN_DOMAINS), so they are excluded by design. + if NON_COLUMN_DOMAINS.contains(&typname) { + continue; + } assert!( resolve(typname).is_some(), "public column domain {typname} did not resolve to a token type" @@ -202,6 +225,24 @@ mod test { assert!(checked > 0, "no public column domains found in the catalog"); } + #[test] + fn json_entry_operand_domain_is_not_a_column_type() { + // `eql_v3_json_entry` is the element `->` returns, not a declarable + // column type; its per-entry terms are `hm` XOR `op`, so resolving it to + // JsonLike+Contain (as `term_json_keys() == None` otherwise would) is + // inverted. It must not resolve as a column domain — a column mistakenly + // declared with it then falls through to a native column. (#424) + assert!(resolve("eql_v3_json_entry").is_none()); + // The real JSON column domains still resolve. + assert!(resolve("eql_v3_json").is_some()); // storage-only + assert_eq!( + traits("eql_v3_json_search"), + [EqlTrait::JsonLike, EqlTrait::Contain] + .into_iter() + .collect() + ); + } + #[test] fn query_operand_twins_are_not_resolved_as_column_domains() { // `all()` yields at least one `eql_v3.query_*` twin; those are operands, From 3fd4309672ca8070d2ae8e8d20b4d2a8f2a3f4fe Mon Sep 17 00:00:00 2001 From: James Sadler Date: Wed, 22 Jul 2026 14:17:33 +1000 Subject: [PATCH 10/10] docs(eql-mapper): correct Contain and cross-check in the v3 ADRs Two design corrections found during implementation: - Containment is NOT removed in v3. Verified against the installed cipherstash-encrypt.sql: @>/<@ raise on scalar encrypted columns but are real, supported operators on encrypted JSON (eql_v3_json_search). The Contain trait is retained as a JSON-only capability. Fixes the glossary ("removed in v3") and adds a consequence to ADR-0001. - ADR-0002 amended: the domain name is the sole authority for a column's capability (no encrypt-config cross-check), and the domain identity is non-optional (legacy eql_v2_encrypted columns are dropped with a warning rather than given a fabricated identity). Refs CIP-3597, CIP-3598, CIP-3599. Stable-Commit-Id: q-4rd3p778qhjfc5dessqeeg1b2e --- packages/eql-mapper/CONTEXT.md | 9 +++++++-- .../docs/adr/0001-functional-index-rewrite.md | 5 +++++ .../adr/0002-token-type-as-inert-identity.md | 20 ++++++++++++++++--- 3 files changed, 29 insertions(+), 5 deletions(-) diff --git a/packages/eql-mapper/CONTEXT.md b/packages/eql-mapper/CONTEXT.md index 65b8931c..8e2eb0ad 100644 --- a/packages/eql-mapper/CONTEXT.md +++ b/packages/eql-mapper/CONTEXT.md @@ -53,8 +53,13 @@ A **capability**, not a storage structure — several SEM terms can satisfy one is **coarse** by design: `Ord` says "ordering is allowed" without distinguishing OPE from ORE, because that variant lives in the domain identity. See the shared vocabulary in `CONTEXT-MAP.md`. -_Avoid_: index, index type (those name the storage, not the capability); `Contain` (the -v2 containment trait — removed in v3, where `@>`/`<@` on encrypted values raise). +_Avoid_: index, index type (those name the storage, not the capability). + +**`Contain`** is **retained** in v3, scoped to encrypted **JSON** columns +(`eql_v3_json_search`): `@>`/`<@` are real, supported operators there. On **scalar** +encrypted columns `@>`/`<@` raise — so `Contain` is a JSON-only capability, not a +general one. (An earlier note here claimed `Contain` was removed entirely in v3; that was +verified wrong against the installed `cipherstash-encrypt.sql`.) **EqlTraits**: A set of `EqlTrait`s. Read in two opposite directions depending on position: as diff --git a/packages/eql-mapper/docs/adr/0001-functional-index-rewrite.md b/packages/eql-mapper/docs/adr/0001-functional-index-rewrite.md index 13eaf0d8..e9fb34fc 100644 --- a/packages/eql-mapper/docs/adr/0001-functional-index-rewrite.md +++ b/packages/eql-mapper/docs/adr/0001-functional-index-rewrite.md @@ -31,3 +31,8 @@ has no valid rewrite target, which is exactly the type error we raise. is read from the domain identity (see ADR-0002), not from the coarse `Ord` trait. - Native `@>`/`<@`/operators on **plaintext** columns are untouched; only encrypted operands are rewritten. +- **Containment is not gone.** `@>`/`<@` raise on *scalar* encrypted columns, but on + encrypted **JSON** columns (`eql_v3_json_search`) they are real, supported operators + (verified against the installed `cipherstash-encrypt.sql`). The `Contain` trait is + therefore retained as a JSON-only capability, and its rewrite targets the SteVec + containment surface — it is not deleted. diff --git a/packages/eql-mapper/docs/adr/0002-token-type-as-inert-identity.md b/packages/eql-mapper/docs/adr/0002-token-type-as-inert-identity.md index f1930120..c91a385f 100644 --- a/packages/eql-mapper/docs/adr/0002-token-type-as-inert-identity.md +++ b/packages/eql-mapper/docs/adr/0002-token-type-as-inert-identity.md @@ -27,8 +27,22 @@ in the identity and surface only during code generation. through the associated-type machinery and unification arms for free because it is never inspected there. - The source of truth for the identity is the Postgres domain name, parsed in - `cipherstash-proxy`'s `SchemaManager` via the `eql-bindings` inventory; the encrypt - config is demoted to a cross-check. `eql-mapper` stays wire-format-agnostic and takes no - dependency on `eql-bindings`. + `cipherstash-proxy`'s `SchemaManager` via the `eql-bindings` inventory. The domain name + is the **sole** authority for a column's capability; the encrypt config is **not** + consulted (a mismatch cross-check was considered and dropped — see the amendment below). + `eql-mapper` stays wire-format-agnostic and takes no dependency on `eql-bindings`. - If a future capability genuinely needs cross-column token-type checking (none does today), this decision is the thing to revisit. + +## Amendment (2026-07-22) + +Two points confirmed during implementation: + +- **Domain identity is non-optional.** `EqlValue` carries a `DomainIdentity`, not an + `Option`. A migration-time `Option` was tried and rejected: the loader + always supplies the real identity, and a legacy `eql_v2_encrypted` column — which has no + v3 domain — is dropped (loaded as `Native` with a warning) rather than given a fabricated + identity, since v2 is retired on this build. +- **No config cross-check.** The domain name is the sole authority; the encrypt config is + not cross-checked against it. The drift diagnostic was considered and dropped to avoid + coupling `SchemaManager` to `EncryptConfigManager` for a non-correctness check.