diff --git a/.changeset/regex-retirement-icontains-drivers.md b/.changeset/regex-retirement-icontains-drivers.md new file mode 100644 index 0000000000..60c0a6c33f --- /dev/null +++ b/.changeset/regex-retirement-icontains-drivers.md @@ -0,0 +1,53 @@ +--- +"@objectstack/driver-sql": minor +"@objectstack/driver-sqlite-wasm": minor +"@objectstack/driver-turso": minor +"@objectstack/driver-memory": minor +"@objectstack/driver-mongodb": minor +"@objectstack/objectql": minor +--- + +feat(drivers,objectql): `$regex` / `$options` are refused everywhere, and `$icontains` is implemented on the SQL family (#5702) + +The driver half of the #4706 ruling. #5701 landed the contract (the vocabulary, +the `RETIRED_FILTER_OPERATORS` prescriptions, the shared text case-set) and +#5710 flipped the last live producer — `plugin-auth`'s ObjectQL adapter, which +emitted `$regex` on the authentication path — so the refusal can now land +without breaking sign-in. + +**BREAKING for anyone writing `$regex` or `$options` in a filter.** Both are +refused on every backend with `INVALID_FILTER` / 400 and a message that names +the replacement. `$regex` was never a declared operator: `driver-sql` compiled +it to a LIKE-escaped substring (so `a.b` matched only the literal `a.b`), +`driver-memory` ran it as a real `RegExp` (so the same filter also matched +`axb`, and an *invalid* pattern was caught and answered `false` — zero rows, in +silence), and `objectql`'s `having` did the same. Write `$icontains` for the +case-insensitive substring search this was almost always used for, `$contains` +for a case-sensitive one; a pattern that genuinely needs a regex has no +filter-level replacement. + +**`$icontains` now runs on the SQL family** — `driver-sql`, `driver-sqlite-wasm`, +and both of `driver-turso`'s transports (the remote one does not go through +knex, so it needed its own). It compiles to `LOWER(col) LIKE LOWER(?) ESCAPE ?` +through the same `applyLike` / `pushLike` that carries the `%` / `_` / `\` +escaping, as a `fold` parameter rather than a second emitter — a copied emitter +is where the escape class would have been dropped, and an unescaped `%` matches +every row. An empty or non-string comparand is refused on the validating walk +(an empty one matches every row, which widens rather than narrows). On SQLite +`lower()` folds ASCII only, which IS the contract (#4706 Q1 = A): `$icontains: +'café'` does not match `CAFÉ`. + + + +`driver-mongodb`'s unknown-operator arm was throwing a bare `Error` with no +`code` and no `status`, three lines from the helper in its own file that sets +`INVALID_FILTER` / 400 — a 500-shaped body for a 400-class client mistake. It +now speaks the same envelope as its three siblings. + +Two parts of the ruling are deliberately NOT in this change and stay tracked in +`scripts/check-driver-conformance.mjs`'s ledger: the `$contains` family's +case-sensitivity (#4706 Q2 = A) needs SQLite's `LIKE` replaced by a case-exact +construct in the driver, the RLS lowering and the analytics lowering together, +or one permission rule compiles to two row sets (#6518); and `$icontains` on the +JS evaluation faces needs the spec vocabulary to take the operator, which cannot +happen before `driver-memory` has an arm for it (#6520). diff --git a/docs/protocol-upgrade-guide.md b/docs/protocol-upgrade-guide.md index a1e531a88a..9a87253aec 100644 --- a/docs/protocol-upgrade-guide.md +++ b/docs/protocol-upgrade-guide.md @@ -375,6 +375,9 @@ Finally it narrows the aggregation vocabulary: `array_agg` and `string_agg` leav - **`driver-sql-distinct-bare-filter-typed`** — `SqlDriver.distinct() third argument — any value` → a bare FilterCondition (@objectstack/spec/data) — the same value find() carries under query.where, never a query envelope - Why not automatic: This entry records a TYPE being added, not a surface being withdrawn, and it says so up front because the distinction decides who has to do anything. `distinct` is not declared on `IDataDriver`, so #5181 / #6075 never reached it and it kept `filters?: any` while its body said something far more specific — `applyFilters(builder, filters)` is handed the ARGUMENT ITSELF, never a `.where` off it. ⚠️ RUNTIME BEHAVIOUR IS UNCHANGED by this entry's change: not one statement moved, so no upgrade breaks at run time and nothing that answered correctly stops. What the annotation removes is a compile-time hole, measured rather than assumed: a truthy NON-OBJECT third argument — `distinct('orders', 'product', 'completed')` — used to type-check and resolve the UNFILTERED set, because `applyFilters` emits no predicate at all for a truthy non-object, non-array filter. A call meaning "which products among completed orders" answered with EVERY product, silently. That spelling is now TS2345 at the call site. This is a driver CALL ARGUMENT — code, never stack metadata — so there is no source for the D2 chain to rewrite and deliberately no schema tombstone, the disposition `data-driver-find-stream-retired` (#4484), `storage-service-list-retired` (#5540), `actor-user-roles-to-positions` (#6011) and `driver-aggregate-undeclared-key-aliases-removed` (#6321) already carry. ⚠️ It differs from those four in ONE measured way a reader should not have to infer: because nothing changed at run time, an untyped JS caller is not affected BY THE UPGRADE at all. The entry is here for a different reason — such a caller is exactly the one tsc can never reach, and the silent widening above is a defect they may ALREADY be sitting on, before and after this major. The generated upgrade guide is the only channel that reaches them, which is why the fix is written down rather than left to the compiler. ⛔ The reverse mismatch is NOT closed and no type can close it: `FilterCondition` is an open map (`[key: string]: any`) because a filter key IS a field name, so a query envelope `{ object, where }` is structurally a valid filter — one constraining columns named `object` and `where` — and so is a FilterArray. Both reach `distinct` type-checked and are refused at run time, loudly, with INVALID_FILTER / 400. `driver-memory`'s opposite half — where the BARE spelling returns the unfiltered set in silence — stays open under the #5499 freeze (#6320). ADR-0087, #6320. - Done when: No caller passes a non-object to `distinct()`'s third argument. A scalar there is now a compile error (`TS2345: Argument of type 'string' is not assignable to parameter of type 'FilterCondition'`); rewrite it as the bare filter it was always meant to be — `'completed'` becomes `{ status: 'completed' }`. ⚠️ That is NOT an equivalent rewrite: the old spelling returned the UNFILTERED set, so the answer changes once fixed, and the changed answer is the one the call always meant. An untyped JS caller gets no compile error and no behaviour change — for them this entry is the only notice that the spelling never filtered anything. A query envelope or a FilterArray in that slot still compiles and is rejected at run time with INVALID_FILTER / 400. +- **`filter-regex-options-retired`** — `data.filter $regex / $options — in a STORED filter (dashboard widget filter and globalFilters, report runtimeFilter, page and component filter, solution-blueprint filter), and equally in the where clause of a query request` → $icontains for the case-insensitive substring match this was almost always used for, or $contains for a case-sensitive one — a pattern that genuinely needs a regular expression has no filter-level replacement + - Why not automatic: Like `driver-aggregate-undeclared-key-aliases-removed` and `driver-sql-distinct-bare-filter-typed`, this entry records a LENIENCY being withdrawn rather than a declared surface: `$regex` was never in `FILTER_OPERATORS` and never a key on `StringOperatorSchema`. That is measured, not assumed — `git log -S'$regex'` over `packages/spec/src` returns only doc comments describing how `$contains` LOWERS to MongoDB (`Contains substring - SQL: LIKE %?% | MongoDB: $regex`), plus #5701 itself, which added the name solely as `RETIRED_FILTER_OPERATORS` prescription data. ⚠️ But it differs from those two in the one way that decides the disposition, so a reader should not have to infer it: those were driver CALL ARGUMENTS, code and never stack metadata, whereas a filter IS stored metadata. `FilterConditionSchema` is an OPEN RECORD (`z.record(z.string(), z.unknown())`) because a filter key is a field name, so a stored `{ name: { $regex: 'acme.*' } }` parses GREEN and always will — a `retiredKey()` tombstone cannot exist on an open map, which is exactly why the ledger has to carry this. What such a stack used to get was four different answers from four backends: `driver-sql` and Turso's remote transport compiled it to a LIKE-escaped SUBSTRING (so `a.b` matched only the literal `a.b` and the regex was silently never a regex), `driver-memory` and objectql's `having` ran it as a real `RegExp` (so the same filter also matched `axb`, and an INVALID pattern was caught and answered `false` — zero rows, in silence), and `driver-mongodb` refused it with a bare `Error` carrying no `code` and no `status`. It is now refused everywhere with INVALID_FILTER / 400 naming the replacement. There is deliberately NO D2 conversion and this sits in `semantic` rather than among the mechanical transforms: rewriting `$regex` to `$icontains` is NOT lossless in either direction — a regex metacharacter becomes a literal — so an auto-applied rewrite would silently change which rows a dashboard, report or permission filter selects, a wrong number rather than a missing one. Choosing the substring the pattern MEANT is a judgment about the query, not a transform. ⚠️ This entry covers BOTH HALVES of the #4706 ruling (B), not just the driver one: the contract half (#5701 — the `$icontains` declaration, the `$contains` family pinned case-sensitive, and the `RETIRED_FILTER_OPERATORS` prescriptions) landed before the ADR-0087 disposition gate (#6148) existed and so was never asked for a ledger entry; the driver half (#5702) is where the refusal became executable. One surface, one entry, registered from the half that made it observable. ADR-0049 / ADR-0087, #4706 / #5701 / #5702. + - Done when: No stored filter and no request `where` spells `$regex` or `$options` — grep the stack for both. Each one is rewritten by asking what the pattern MEANT, not by transliterating it: a bare substring pattern becomes `$icontains` (or `$contains` when the match must stay case-sensitive), and its metacharacters are dropped rather than escaped, because they were never honoured as a regex on the SQL family in the first place. ⚠️ Expect the answer to CHANGE on any stack that ran on `driver-memory`, `driver-mongodb` or objectql `having`, where the pattern really was evaluated as a regular expression; on the SQL family the rewritten filter returns what it always returned. A pattern that genuinely needs alternation, anchoring or character classes has no filter-level replacement — move that predicate into a formula field or a server-side view, or open an issue for it. Verify by loading the stack: a surviving `$regex` or `$options` is answered INVALID_FILTER / 400 with a message naming the replacement, on every backend. - **`http-server-runtime-vocabulary-retired`** — `system.serverEvent / system.serverEventType / system.serverCapabilities / system.serverStatus (the lifecycle-event, capability-report and status vocabulary of system/http-server.zod.ts — 4 defs, 8 exported names)` → (removed — there is no replacement key, because there was never a key. Server lifecycle is the transport plugin's own start/stop seam; per-request and per-server observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus `OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a transport plugin can DO it states by implementing the kernel plugin contract — the seams it registers are the capability statement, and a self-described capability record can only disagree with them. Server-level configuration that IS authorable lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected) - Why not automatic: The second and final ADR-0049 pass over `system/http-server.zod.ts`. #4938 removed the CONFIG half (`HttpServerConfigSchema`, nine keys, zero readers, zero authoring entry); this removes the RUNTIME half — a 7-member lifecycle event union with a timestamped envelope, an eight-boolean capability report, and a five-state status record with connection and request counters. Nothing ever emitted, consumed or parsed any of them. This card was HELD for four days rather than queued, on a specific and legitimate doubt: a response/capability vocabulary can be a REFERENCE surface for host implementers, so "zero consumers in this repo" is weaker evidence for one of those than for an authorable key (the CSS-variable rebuttal). The hold was lifted by measuring the reference reader itself rather than by re-running the same grep: `plugin-hono-server`, the one in-tree host implementation, neither implements nor reports any of the three — it names no capability record, no status shape and no event union, and what it registers is routes and middleware through the kernel plugin contract. A declaration-site grep put every declaration in this one file, a quoted-name sweep across objectstack and objectui found no reader outside it, and the control passed in the SAME run: `MiddlewareConfig`, declared twelve lines away, resolves to `packages/runtime/src/middleware.ts`. So the sweep could see a reader in this file when there was one. With no carrier key there is nothing to tombstone, and with no author there is no source or `sys_metadata` row for a D2 conversion to rewrite: RETIRED_DEFS_BY_MAJOR plus this entry are the declaration — route 3, the same shape as #4938 in this very file, #4834, #4988 and #5055. If host-implementer conformance becomes a real requirement it returns through the ENFORCE route: an adapter contract with a checker behind it, vocabulary second. ADR-0049, #5295. - Done when: No source imports `ServerEvent`, `ServerEventType`, `ServerEventSchema`, `ServerCapabilities`, `ServerCapabilitiesSchema`, `ServerCapabilitiesParsed`, `ServerStatus` or `ServerStatusSchema` from `@objectstack/spec/system` — a grep over consumer code resolves none of them, and `tsc` reports TS2724/TS2305 on any that survives. The route-registration half of the same module still resolves (`RouteHandlerMetadataSchema`, `MiddlewareType`, `MiddlewareConfigSchema`, `MiddlewareConfig`), and `StackServerConfigSchema` — the one authorable server surface — is untouched: a stack declaring `server: { trustProxy, security }` parses exactly as it did in 16.x. diff --git a/packages/drivers/driver-memory/src/filter-refusal.ts b/packages/drivers/driver-memory/src/filter-refusal.ts index 59d6214c6d..ad20c6dc7c 100644 --- a/packages/drivers/driver-memory/src/filter-refusal.ts +++ b/packages/drivers/driver-memory/src/filter-refusal.ts @@ -30,7 +30,7 @@ * vocabulary; it does not get to drop what falls outside it. */ -import { FILTER_OPERATORS, LOGICAL_OPERATORS } from '@objectstack/spec/data'; +import { FILTER_OPERATORS, LOGICAL_OPERATORS, RETIRED_FILTER_OPERATORS } from '@objectstack/spec/data'; import { StandardErrorCode } from '@objectstack/spec/api'; /** @@ -148,18 +148,27 @@ export function emptyFieldConstraintError(field: string, path: string): Error { * `convertConditionToMongo`'s alias fold) — a list written out here would agree * with the spec on the day it was typed and never again. * - * Two additions the spec's list does not carry, both deliberate and both - * pre-existing behaviour rather than new capability: + * ## [#5702] The two additions are GONE — nothing is added any more * - * - **`$regex`** — not in `FILTER_OPERATORS`, but really produced: plugin-auth's - * ObjectQL adapter emits `{ field: { $regex: value } }` for a `contains` - * search. `driver-sql` compiles it (to a substring LIKE), `objectql`'s - * `having` allows it, and this driver's matcher implements it. Refusing it - * here would break a live producer. - * - **`$options`** — the regex-flags companion `memory-matcher` reads - * (`new RegExp(target, condition.$options)`) and `objectql`'s `having` skips - * for the same reason. It is a modifier of `$regex`, not a predicate of its - * own. + * This set used to be `[...FILTER_OPERATORS, '$regex', '$options']`. Both extra + * members existed for one reason, recorded here verbatim at the time: *"Refusing + * it here would break a live producer"* — plugin-auth's ObjectQL adapter emitted + * `{ field: { $regex: value } }` for better-auth's `contains` search, on the + * AUTHENTICATION path. + * + * That producer was flipped to `$contains` by #5710 (PR #5812), and a whole-repo + * scan on `origin/main` found no other: every surviving `$regex` occurrence is a + * consumer arm, a retirement prescription, or a refusal assertion. The reason the + * two members existed is therefore gone, and #4706 retired both spellings — so + * they are refused here like any other undeclared operator, with the spec's + * prescription attached (see {@link retiredFilterOperatorError}). + * + * Note what this does NOT do: it does not add `$icontains`. That name is + * declared by `StringOperatorSchema` but deliberately absent from + * `FILTER_OPERATORS` (#5701), and this set is derived, so this driver refuses it + * — fail-closed, an unimplemented capability rather than a silent widening. The + * `$icontains` implementation for the JS faces is #5499-frozen; see the + * `driver-memory` row of `scripts/check-driver-conformance.mjs`. * * Everything else is refused. That includes the mingo operators this driver used * to hand through by accident (`$elemMatch`, `$size`, `$type`, `$mod`, `$where`, @@ -168,8 +177,6 @@ export function emptyFieldConstraintError(field: string, path: string): Error { */ export const SUPPORTED_FIELD_OPERATORS: ReadonlySet = new Set([ ...FILTER_OPERATORS, - '$regex', - '$options', ]); /** The vocabulary as it appears in a refusal message, in declaration order. */ @@ -418,24 +425,53 @@ export function nonBooleanNullComparandError(field: string, value: unknown, path } /** - * [#5324] `$options` without the `$regex` it modifies. - * - * `$options` is in {@link SUPPORTED_FIELD_OPERATORS} as a MODIFIER, not a - * predicate — it carries the regex flags (`memory-matcher` reads it as - * `new RegExp(target, condition.$options)`, and objectql's `having` skips it for - * the same reason). On its own it is not a filter at all, and the two faces - * proved it: mingo raised `unknown query operator $options` — uncoded, the very - * escape #5324 is about — while the matcher ignored it and matched EVERY row. - * Allowlisting the key without requiring its partner would have left exactly one - * operator still leaking out of the envelope. + * [#5702] A RETIRED filter operator in a field constraint. + * + * Distinct from {@link unknownFieldOperatorError} on purpose, and the + * distinction is the author's: `$sounds_like` is a name that never meant + * anything, while `$regex` and `$options` are names this driver ANSWERED — with + * a real `RegExp`, the only regex evaluator in the repo — until #4706 retired + * them. Handing that author the fifteen-name vocabulary list is true and + * useless; what they need is `$icontains`. + * + * The prescription is `RETIRED_FILTER_OPERATORS[op].why`, printed VERBATIM. The + * spec table exists precisely so `driver-sql`, this driver, `driver-turso`'s + * remote transport, `driver-mongodb` and `objectql`'s `having` stop each writing + * their own sentence about one retirement (#5701). + * + * This subsumes the `$options`-with-no-`$regex` refusal #5324 added + * (`danglingRegexOptionsError`, deleted with this change): while `$options` was + * an allowlisted MODIFIER, a dangling one needed its own gate; now that both + * spellings are refused outright there is no shape left for that gate to catch, + * and the message it printed — which taught the reader to write + * `{ "$regex": "abc", "$options": "i" }` — would be prescribing the retired form. + * + * `siblings` are the other keys of the SAME field constraint, and every retired + * one among them is named too — `{ $regex: '^acme', $options: 'i' }` is ONE + * mistake with ONE fix, and a message naming only the key iteration reached + * first would send its author back for a second round-trip on the other. + * + * Returns `null` when `op` is not retired, so the caller falls through to the + * ordinary unknown-operator refusal in one expression. */ -export function danglingRegexOptionsError(field: string, path: string): Error { +export function retiredFilterOperatorError( + op: string, + field: string, + path: string, + siblings: readonly string[] = [], +): Error | null { + const guidance = RETIRED_FILTER_OPERATORS[op]; + if (!guidance) return null; + const replacement = guidance.to ? ` Write "${guidance.to}" instead.` : ''; + const alsoRetired = siblings.filter((key) => key !== op && RETIRED_FILTER_OPERATORS[key]); + const also = alsoRetired.length + ? ` The same field constraint also carries the retired ` + + `${alsoRetired.map((key) => `"${key}"`).join(', ')} — one "${guidance.to}" replaces the whole ` + + `shape, so this is ONE mistake with ONE fix, not one per key.` + : ''; return unsupportedFilterError( - `Operator "$options" on field "${field}" at ${path} has no "$regex" to modify. "$options" ` + - `carries the flags of a regex predicate (e.g. { "${field}": { "$regex": "abc", "$options": "i" } }); ` + - `it is not a predicate on its own. It is refused rather than ignored because the two ` + - `evaluation paths answered it differently — one raised an uncoded engine error, the other ` + - `matched every row (#5324).`, + `Filter operator "${op}" on field "${field}" at ${path} is RETIRED and is no longer evaluated ` + + `by this driver.${replacement} ${guidance.why}${also}`, ); } @@ -564,7 +600,13 @@ function assertFieldConstraintShape( const keys = Object.keys(spec); if (!keys.some((key) => key.startsWith('$'))) return; for (const op of keys) { - if (!SUPPORTED_FIELD_OPERATORS.has(op)) throw unknownFieldOperatorError(op, field, path); + if (!SUPPORTED_FIELD_OPERATORS.has(op)) { + // [#5702] A RETIRED spelling gets the prescription; anything else gets + // the vocabulary. Checked in this order because `$regex` satisfies both + // descriptions ("not supported" and "retired") and only the second one + // tells its author what to write. + throw retiredFilterOperatorError(op, field, path, keys) ?? unknownFieldOperatorError(op, field, path); + } // [#5345] Declared, but not by THIS face. Checked before the comparand-shape // rules below so a `$between` a face cannot compile is reported as // unsupported-here rather than as a malformed range the face would refuse @@ -583,9 +625,12 @@ function assertFieldConstraintShape( throw nonBooleanNullComparandError(field, spec[op], `${path}.$null`); } } - // `$options` is the one entry in the vocabulary that is a modifier rather than - // a predicate, so it is the one that needs a companion. - if (keys.includes('$options') && !keys.includes('$regex')) throw danglingRegexOptionsError(field, path); + // [#5702] The `$options`-without-`$regex` companion check that stood here is + // GONE. It was needed while `$options` was an allowlisted MODIFIER — a key the + // vocabulary accepted but which is not a predicate on its own. Both spellings + // are retired now, so the loop above refuses either of them on sight and there + // is no surviving shape for a companion rule to judge. See + // {@link retiredFilterOperatorError}. } /** diff --git a/packages/drivers/driver-memory/src/memory-analytics-filter-refusal.test.ts b/packages/drivers/driver-memory/src/memory-analytics-filter-refusal.test.ts index 06cfec4da4..1c7a674d40 100644 --- a/packages/drivers/driver-memory/src/memory-analytics-filter-refusal.test.ts +++ b/packages/drivers/driver-memory/src/memory-analytics-filter-refusal.test.ts @@ -141,7 +141,6 @@ describe('[#5345] MemoryAnalyticsService — filters it cannot compile are refus { op: '$startsWith', where: { name: { $startsWith: 'al' } } }, { op: '$endsWith', where: { name: { $endsWith: 'ta' } } }, { op: '$null', where: { closed_at: { $null: true } } }, - { op: '$regex', where: { name: { $regex: '^al' } } }, ]; for (const { op, where } of UNCOMPILABLE) { @@ -153,6 +152,34 @@ describe('[#5345] MemoryAnalyticsService — filters it cannot compile are refus }); } + /** + * [#5702] `$regex` was the sixth row of the table above until this change, and + * it was in the WRONG table: its assertion read "declared by the Filter + * Protocol, not compilable by this face", and `$regex` was never declared by + * the Filter Protocol at all — it was an undeclared spelling this package + * evaluated. #4706 retired it outright, so it is no longer a + * declared-but-uncompilable operator on this face; it is a refused one on + * every face, with a prescription attached. + * + * Kept as its own case rather than deleted, because the analytics face is a + * SECOND door into the same walk and "the query path refuses it" is not + * evidence that this one does — the two faces answering one filter differently + * is the divergence class this whole file exists over (#5345). + */ + for (const op of ['$regex', '$options'] as const) { + it(`refuses the retired ${op} on the analytics face too, naming $icontains`, async () => { + const err = await expectRefusal(() => count({ name: { [op]: '^al' } } as FilterCondition), op); + expect(err.message).toContain('RETIRED'); + expect(err.message).toContain('$icontains'); + // NOT the uncompilable-on-this-face sentence. Asserted against that + // message's own distinctive phrase rather than against "declared by the + // Filter Protocol", which the spec's prescription also contains — in the + // NEGATED form ("was never declared by the Filter Protocol"), so a + // substring test on it passes for both messages and pins nothing. + expect(err.message).not.toContain('Supported operators on this surface'); + }); + } + it('refuses an unmapped operator reached through the nested-relation branch', async () => { // `{profile: {verified: …}}` is re-entered as a synthesised `{'profile.verified': …}` // node the up-front gate never walked — the one path where the lowering's own diff --git a/packages/drivers/driver-memory/src/memory-driver.test.ts b/packages/drivers/driver-memory/src/memory-driver.test.ts index dc3eefe86e..02b3ca0e7b 100644 --- a/packages/drivers/driver-memory/src/memory-driver.test.ts +++ b/packages/drivers/driver-memory/src/memory-driver.test.ts @@ -509,12 +509,30 @@ describe('InMemoryDriver', () => { expect(results.map((r: any) => r.name).sort()).toEqual(['Bob', 'Diana']); }); - it('should filter with $regex operator', async () => { - const results = await driver.find(testTable, { - where: { name: { $regex: /^[AB]/ } }, - }); - expect(results).toHaveLength(2); - expect(results.map((r: any) => r.name).sort()).toEqual(['Alice', 'Bob']); + // [#5702] REPLACED WHOLESALE, not re-spelled. This case used to read + // `should filter with $regex operator` and asserted that + // `{ name: { $regex: /^[AB]/ } }` returned Alice and Bob — i.e. it pinned + // the real `RegExp` evaluation that made this driver the only backend in + // the repo answering `$regex` as a pattern while every SQL backend answered + // it as a literal substring. That divergence is what #4706 retired the + // operator over, so the limb is gone and its fixture cannot be re-spelled + // into the new world: there is no `$regex` answer left to assert. + // + // What replaces it pins the retirement instead — and pins `code`/`status`, + // not merely that something threw, because a bare `toThrow()` here would + // stay green against any error at all, including the uncoded engine errors + // #5324 spent a whole issue routing back into the ADR-0112 envelope. + it('refuses the retired $regex operator, in the ADR-0112 envelope', async () => { + const err = await driver + .find(testTable, { where: { name: { $regex: '^[AB]' } } }) + .then(() => null, (e: any) => e); + expect(err).toBeInstanceOf(Error); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('$regex'); + // The prescription, not just the verdict: an author who wrote `$regex` + // needs the name of what replaces it. + expect(err.message).toContain('$icontains'); }); it('should count with complex filter', async () => { diff --git a/packages/drivers/driver-memory/src/memory-driver.ts b/packages/drivers/driver-memory/src/memory-driver.ts index 71c97c6f67..8f8419b79b 100644 --- a/packages/drivers/driver-memory/src/memory-driver.ts +++ b/packages/drivers/driver-memory/src/memory-driver.ts @@ -1002,9 +1002,16 @@ export class InMemoryDriver implements IDataDriver { result[op] = store(val); break; // Evaluated by mingo under the same name. `$exists` is a presence - // predicate, `$regex`/`$options` a pattern and its flags — none of them - // is a comparand, so none takes the field's storage form (#4047). - case '$exists': case '$regex': case '$options': + // predicate, not a comparand, so it does not take the field's storage + // form (#4047). + // + // [#5702] `$regex` and `$options` were passed through here too, on the + // same line, for the same "not a comparand" reason. Both are RETIRED + // (#4706) and refused by the shape gate before this method runs, so the + // arm is gone rather than left as an unreachable third name — an + // evaluation arm for a refused operator is exactly what let this + // driver's two faces answer one `$regex` differently for so long. + case '$exists': result[op] = val; break; default: diff --git a/packages/drivers/driver-memory/src/memory-filter-vocabulary-refusal.test.ts b/packages/drivers/driver-memory/src/memory-filter-vocabulary-refusal.test.ts index 61a88da0b4..02b510a63a 100644 --- a/packages/drivers/driver-memory/src/memory-filter-vocabulary-refusal.test.ts +++ b/packages/drivers/driver-memory/src/memory-filter-vocabulary-refusal.test.ts @@ -153,21 +153,44 @@ describe('[#5324/#5328] a filter this driver cannot evaluate is refused, not ans }); } - it('refuses "$options" with no "$regex" to modify, on both faces', async () => { - // The one entry in the vocabulary that is a MODIFIER rather than a - // predicate, and therefore the one that could be allowlisted into a fresh - // leak: measured before this arm existed, `{ stage: { $options: 'i' } }` - // still escaped as an uncoded `unknown query operator $options` on the live - // path while the matcher ignored it and matched EVERY row — #5324's exact - // shape, surviving for a single operator. - const where = { stage: { $options: 'i' } }; - - const live = await liveRefusal(where); - expectEnvelope(live); - expect(live.message).toContain('has no "$regex" to modify'); - - expect(matcherRefusal(where).message).toBe(live.message); - }); + /** + * [#5702] REPLACED, not re-spelled. This case used to be + * `refuses "$options" with no "$regex" to modify, on both faces` and it pinned + * the companion rule #5324 added while `$regex`/`$options` were still + * ALLOWLISTED members of this driver's vocabulary (they were, because + * plugin-auth's adapter produced `$regex` on the authentication path). + * + * That producer was flipped by #5710 and both spellings are retired by #4706, + * so the companion rule has no surviving shape to judge: `$options` is now + * refused on sight, with or without a `$regex` beside it. Keeping the old + * assertion would have kept passing — the filter still throws — while pinning + * a message that PRESCRIBES the retired form (`{ "$regex": "abc", + * "$options": "i" }`), which is the failure mode where a green test documents + * the wrong contract. + */ + for (const [label, where, mustMention] of [ + ['a bare $regex', { stage: { $regex: 'W' } }, ['$regex', '$icontains']], + ['a bare $options', { stage: { $options: 'i' } }, ['$options', '$icontains']], + [ + '$regex WITH $options — one mistake, one fix', + { stage: { $regex: 'W', $options: 'i' } }, + ['$regex', '$options', '$icontains'], + ], + ] as const) { + it(`refuses ${label}, naming the replacement, on both faces`, async () => { + const live = await liveRefusal(where); + expectEnvelope(live); + expect(live.message).toContain('RETIRED'); + for (const mention of mustMention) expect(live.message).toContain(mention); + + // Both faces, one sentence — the #5324 invariant, which is exactly what a + // retirement must not be allowed to fork: this driver's matcher is the one + // surface in the repo that really evaluated `$regex`. + const reference = matcherRefusal(where); + expectEnvelope(reference); + expect(reference.message).toBe(live.message); + }); + } it('names the position, so a refusal deep in a scope tree is actionable', async () => { const err = await liveRefusal({ $or: [{ owner: 'u1' }, { $and: [{ stage: { $sounds_like: 'won' } }] }] }); @@ -271,7 +294,11 @@ describe('[#5324/#5328] a filter this driver cannot evaluate is refused, not ans [{ stage: { $null: false } }, ['1', '2', '3']], [{ stage: { $null: true } }, []], [{ stage: { $exists: true } }, ['1', '2', '3']], - [{ stage: { $regex: 'W', $options: 'i' } }, ['1']], + // [#5702] `{ stage: { $regex: 'W', $options: 'i' } } → ['1']` used to sit + // here, as a LEGAL shape. It is not one any more — #4706 retired both + // spellings and this driver refuses them — so it moved to the refusal + // table above rather than being deleted: the shape still has to have a + // pinned answer, only the answer changed from a row list to a refusal. [{ score: { $between: [10, 20] } }, ['1', '2']], [{ $or: [{ stage: 'won' }, { owner: 'u2' }] }, ['1', '2']], [{ $and: [{ owner: 'u1' }, { score: { $gt: 15 } }] }, ['3']], diff --git a/packages/drivers/driver-memory/src/memory-matcher.ts b/packages/drivers/driver-memory/src/memory-matcher.ts index afc09bb79b..835ec57b79 100644 --- a/packages/drivers/driver-memory/src/memory-matcher.ts +++ b/packages/drivers/driver-memory/src/memory-matcher.ts @@ -236,13 +236,15 @@ function checkCondition(value: any, condition: any): boolean { if (target === true && value != null) return false; if (target === false && value == null) return false; break; - case '$regex': - try { - const re = new RegExp(target, condition.$options || ''); - if (!re.test(String(value))) return false; - } catch (e) { return false; } - break; - + // [#5702] The `$regex` arm that stood here is GONE. It was the only + // real regex evaluator in the repo, and the reason #4706 retired the + // operator rather than standardising it: `new RegExp(target)` read + // `a.b` as a pattern (so it also matched `axb`) where every SQL + // backend read it as a literal, and its `catch { return false }` + // answered an ILLEGAL pattern with "no rows" — a silent wrong + // answer, not an error. Both spellings are refused by + // `filter-refusal.ts`'s vocabulary gate before this evaluator runs; + // `$icontains` is the replacement the refusal prescribes. default: // [#5324] Unreachable through `match`: the shape gate refuses an // operator this driver does not evaluate, with the same diff --git a/packages/drivers/driver-mongodb/src/mongodb-filter.test.ts b/packages/drivers/driver-mongodb/src/mongodb-filter.test.ts index 1aef79a181..6fb1b6d085 100644 --- a/packages/drivers/driver-mongodb/src/mongodb-filter.test.ts +++ b/packages/drivers/driver-mongodb/src/mongodb-filter.test.ts @@ -239,17 +239,63 @@ describe('MongoDB Filter Translator', () => { }); describe('operator allowlist (P0-4)', () => { - it('rejects $where (server-side JS execution)', () => { - expect(() => translateFilter({ name: { $where: 'this.x == 1' } })).toThrow(/unsupported filter operator '\$where'/); - }); - - it('rejects $function', () => { - expect(() => translateFilter({ name: { $function: { body: 'fn', args: [], lang: 'js' } } })).toThrow(/unsupported filter operator/); - }); + /** + * [#5702] These three used to match `/unsupported filter operator '\$x'/` — + * the bare `new Error` the `default:` arm threw, with no `code` and no + * `status`. The verdict has not changed (they were always refused); the + * ENVELOPE has, so the assertion moved with it and now pins the two fields + * that decide whether a client sees a 400-class mistake or a 500-shaped + * body. Matching on the message alone is what let the bare error live three + * lines from this file's own `INVALID_FILTER` helper for two releases. + */ + const refusalOf = (filter: Parameters[0]) => { + try { + translateFilter(filter); + } catch (e) { + return e as Error & { code?: string; status?: number }; + } + throw new Error('expected the translator to refuse this filter, but it returned'); + }; + + for (const [label, filter, op] of [ + ['$where (server-side JS execution)', { name: { $where: 'this.x == 1' } }, '$where'], + ['$function', { name: { $function: { body: 'fn', args: [], lang: 'js' } } }, '$function'], + ['an unknown $-operator rather than passing it through', { name: { $expr: 1 } }, '$expr'], + ] as const) { + it(`rejects ${label}, in the ADR-0112 envelope`, () => { + const err = refusalOf(filter); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain(`Unsupported filter operator "${op}"`); + // The field is named — a refusal that says only "some operator" makes + // the author search a filter tree by hand. + expect(err.message).toContain('on field "name"'); + }); + } - it('rejects unknown $-operators rather than passing them through', () => { - expect(() => translateFilter({ name: { $expr: 1 } })).toThrow(/unsupported filter operator '\$expr'/); - }); + /** + * [#5702] The retired spellings take the OTHER branch of that arm: they were + * refused here before this change too (mongo was the one backend already + * satisfying #4706's requirement 3), but with the generic sentence. An + * author who wrote `$regex` needs `$icontains`, not a list. + */ + for (const [label, filter, mustMention] of [ + ['$regex', { name: { $regex: 'ac.*' } }, ['$regex', '$icontains']], + ['$options on its own', { name: { $options: 'i' } }, ['$options', '$icontains']], + [ + '$regex with $options — one mistake, one fix', + { name: { $regex: '^acme', $options: 'i' } }, + ['$regex', '$options', '$icontains'], + ], + ] as const) { + it(`refuses the retired ${label}, naming the replacement`, () => { + const err = refusalOf(filter); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('RETIRED'); + for (const mention of mustMention) expect(err.message).toContain(mention); + }); + } it('still accepts every allowlisted field operator', () => { expect(() => translateFilter({ a: { $eq: 1 }, b: { $in: [1, 2] }, c: { $contains: 'x' }, d: { $exists: true } })).not.toThrow(); diff --git a/packages/drivers/driver-mongodb/src/mongodb-filter.ts b/packages/drivers/driver-mongodb/src/mongodb-filter.ts index 0a0d6e0d77..ad2d5e9473 100644 --- a/packages/drivers/driver-mongodb/src/mongodb-filter.ts +++ b/packages/drivers/driver-mongodb/src/mongodb-filter.ts @@ -36,6 +36,10 @@ import { type FilterVerdict as SharedFilterVerdict, type FilterVerdictHooks, } from '@objectstack/spec/data'; +// [#5702] The retired filter operators and the prescription each refusal +// prints, read from the spec so this driver's sentence about `$regex` cannot +// drift from the four other refusal sites' (#5701). +import { RETIRED_FILTER_OPERATORS } from '@objectstack/spec/data'; import { coerceTemporalValue, type TemporalFieldKind, @@ -266,10 +270,13 @@ function filterArrayReachedDriverError(filters: unknown[]): Error { * swaps one driver for another must see one `400 INVALID_FILTER`, not a coded * refusal on three backends and a bare `{ error }` on the fourth. * - * Note what this does NOT do: the `default:` arm of {@link translateFieldOperators} - * still throws a bare `Error` with a `[mongodb]` prefix, outside this envelope. - * That is #5346's, filed and measured separately — converting it here would be - * an unrelated behaviour change riding on #5347. + * [#5702] The carve-out that used to close this comment is GONE. It read: *"the + * `default:` arm of {@link translateFieldOperators} still throws a bare `Error` + * with a `[mongodb]` prefix, outside this envelope. That is #5346's, filed and + * measured separately."* That arm now routes through this constructor, so this + * package no longer answers an unknown or retired operator with a 500-shaped + * body while its three siblings answer `400 INVALID_FILTER` — which was the last + * place the sentence two paragraphs up was not yet true. */ function unsupportedFilterError(message: string): Error { const err = new Error(message) as Error & { code?: string; status?: number }; @@ -580,12 +587,47 @@ function translateFieldOperators( } break; - default: + default: { // Reject unknown operators instead of passing them through (P0). Keys // like `$where` / `$function` / `$expr` / `$accumulator` would reach // MongoDB and execute server-side JavaScript or bypass query intent. // Every legitimate ObjectQL field operator is allowlisted above. - throw new Error(`[mongodb] unsupported filter operator '${op}'`); + // + // [#5702] Two changes, both about the SHAPE of the refusal rather than + // its verdict — this arm already refused, including `$regex`, which is + // why mongo was the one backend already satisfying #4706's requirement 3: + // + // 1. It threw a bare `new Error`, with no `code` and no `status`, three + // lines from a helper in this same file that sets `INVALID_FILTER` / + // 400 and whose own comment says "a test suite that swaps one driver + // for another must see one `400 INVALID_FILTER`". A 500-shaped body + // for a 400-class client mistake is the half of #5324 the refusal + // itself does not fix. + // 2. A RETIRED spelling now gets the spec's prescription instead of the + // generic sentence: `$regex`'s author needs `$icontains`, and + // "unsupported filter operator" does not say so. + const retired = RETIRED_FILTER_OPERATORS[op]; + if (retired) { + const replacement = retired.to ? ` Write "${retired.to}" instead.` : ''; + const alsoRetired = Object.keys(ops).filter( + (key) => key !== op && RETIRED_FILTER_OPERATORS[key], + ); + const also = alsoRetired.length + ? ` The same field constraint also carries the retired ` + + `${alsoRetired.map((key) => `"${key}"`).join(', ')} — one "${retired.to}" replaces ` + + `the whole shape, so this is ONE mistake with ONE fix, not one per key.` + : ''; + throw unsupportedFilterError( + `Filter operator "${op}" on field "${field}" at ${path} is RETIRED and is no longer ` + + `translated by this driver.${replacement} ${retired.why}${also}`, + ); + } + throw unsupportedFilterError( + `Unsupported filter operator "${op}" on field "${field}" at ${path}. It is refused ` + + `rather than passed through to MongoDB, where keys like $where / $function / $expr ` + + `execute server-side JavaScript or bypass the query's intent (P0).`, + ); + } } } diff --git a/packages/drivers/driver-sql/src/sql-driver-icontains-and-retired-operators.test.ts b/packages/drivers/driver-sql/src/sql-driver-icontains-and-retired-operators.test.ts new file mode 100644 index 0000000000..b473fd7603 --- /dev/null +++ b/packages/drivers/driver-sql/src/sql-driver-icontains-and-retired-operators.test.ts @@ -0,0 +1,231 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#5702] `$icontains` on the SQL family, and the retirement of `$regex` / + * `$options` — the DRIVER half of the #4706 ruling (its contract half is #5701). + * + * ## What this pins, and why it is not the shared text case-set + * + * `@objectstack/spec/data` carries a canonical text case-set whose rows this + * file reuses (`FILTER_TEXT_ROWS` — the same nine, so a verdict here is + * comparable to one anywhere else). It deliberately does NOT import that + * case-set's CASES export, because `scripts/check-driver-conformance.mjs` + * judges a cell covered by that import and this driver does not yet answer the + * whole table: five of its cases require the `$contains` family to be + * case-SENSITIVE (#4706 Q2 = A), which SQLite's `LIKE` is not, and which cannot + * be fixed in this driver alone — `read-scope-sql` and `service-analytics` + * compile the same predicate for RLS and for the analytics face, so a + * driver-only change would give ONE permission rule two row sets (#3948). That + * work is filed separately and the driver's DEBT row stays open for it. + * + * Importing the case-set here would flip the cell to "covered" while five of + * its cases were unanswered — a gate reporting success over a standard nobody + * runs, which is exactly the failure the gate exists to prevent. + * + * ## The reverse verification, direction decided BEFORE it was run + * + * - **Refusal face** — predicted RED, measured RED. Restoring the deleted + * `case '$regex':` fallthrough makes every assertion below that reads `code` + * / `status` fail, because the filter compiles again and nothing throws. + * - **`$icontains` face on SQLite** — predicted red, and the prediction needed + * a correction that is recorded here rather than smoothed over: deleting the + * `case '$icontains':` arm turns these cases red LOUDLY (the operator falls to + * `default:` and is refused), but deleting only the `LOWER()` fold does NOT, + * for any comparand. SQLite's `LIKE` folds ASCII by itself, so on this + * dialect the fold is unobservable in rows. The compiled-SQL case at the end + * is what pins it, and it is the only thing here that can. + */ + +import type { Knex } from 'knex'; +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import type { DriverOptions, FilterCondition } from '@objectstack/spec/data'; +import { FILTER_TEXT_ROWS } from '@objectstack/spec/data'; +import { SqlDriver } from './sql-driver.js'; + +/** The error a refused filter produced — never a bare `toThrow()` (see below). */ +interface WireBearingError extends Error { + code?: string; + status?: number; +} + +/** + * The compiled statement for one filter, without reaching into a private field. + * + * `applyFilters` is `protected` and `getKnex()` is public, so a subclass is the + * TYPED way in — no `as any` on the driver, which would also switch off the + * checking on everything else the call touches. + */ +class CompilerProbeDriver extends SqlDriver { + compileWhere(where: FilterCondition): string { + const builder: Knex.QueryBuilder = this.getKnex()('txt'); + this.applyFilters(builder, where); + return builder.toString(); + } +} + +/** Diagnostics-only; it never changes which rows a read touches. */ +const BYPASS: DriverOptions = { bypassTenantAudit: true }; + +describe('[#5702] SqlDriver — $icontains, and the retired $regex/$options', () => { + let driver: CompilerProbeDriver; + + beforeAll(async () => { + driver = new CompilerProbeDriver({ + client: 'better-sqlite3', + connection: { filename: ':memory:' }, + useNullAsDefault: true, + }); + await driver.initObjects([{ name: 'txt', fields: { name: { type: 'string' } } }]); + for (const row of FILTER_TEXT_ROWS) { + await driver.create('txt', { ...row }, BYPASS); + } + }); + + afterAll(async () => { + await driver.disconnect(); + }); + + const ids = async (where: FilterCondition): Promise => { + const rows = await driver.find('txt', { where }, BYPASS); + return rows.map((r) => String(r.id)).sort((a, b) => a.localeCompare(b)); + }; + + const refusalOf = async (where: FilterCondition): Promise => { + const err = await driver + .find('txt', { where }, BYPASS) + .then(() => null, (e: unknown) => e as WireBearingError); + if (!err) throw new Error(`expected the driver to refuse ${JSON.stringify(where)}, but it compiled`); + return err; + }; + + it('seeded the fixture (the premise)', async () => { + expect(await ids({})).toEqual(['1', '2', '3', '4', '5', '6', '7', '8', '9']); + }); + + // ── The fold ─────────────────────────────────────────────────────────────── + + it('folds ASCII case in BOTH directions', async () => { + // The fold has to run on both operands. Folding only the comparand compares + // a lower-cased needle against a raw column and answers ['2'] to the first + // line and [] to the second — half right, which reads as "working". + expect(await ids({ name: { $icontains: 'acme' } })).toEqual(['1', '2']); + expect(await ids({ name: { $icontains: 'ACME' } })).toEqual(['1', '2']); + }); + + it('folds ASCII ONLY — the #4706 Q1 = A boundary', async () => { + // These two ARE the contract, not an edge case. A backend folding the whole + // Unicode range (a JS `toLowerCase()`, mongo's `$options: "i"`) answers + // ['3','4'] to both and is wrong on both — not because Unicode folding is + // worse, but because SQLite cannot do it, so promising it would promise what + // three of five backends cannot deliver. + expect(await ids({ name: { $icontains: 'café' } })).toEqual(['4']); + expect(await ids({ name: { $icontains: 'CAFÉ' } })).toEqual(['3']); + }); + + // ── The comparand is LITERAL ─────────────────────────────────────────────── + + it('treats "%" as a literal character, not a LIKE wildcard', async () => { + // Unescaped this compiles to LIKE '%100%%', which also matches row 6. + expect(await ids({ name: { $icontains: '100%' } })).toEqual(['5']); + }); + + it('treats "_" as a literal character, not a single-character wildcard', async () => { + // Unescaped, `%a_b%` also returns rows 8 (axb) and 9 (a.b). + expect(await ids({ name: { $icontains: 'a_b' } })).toEqual(['7']); + }); + + it('treats "\\" as a literal character, not the escape character', async () => { + // No fixture row holds a backslash, so the observable claim is that the + // pattern stays well-formed and selects nothing rather than erroring or + // degenerating — the `\` limb of the class is pinned per-dialect and per + // operator in `sql-driver-like-escape.test.ts`. + expect(await ids({ name: { $icontains: '\\' } })).toEqual([]); + expect(await ids({ name: { $icontains: 'a\\b' } })).toEqual([]); + }); + + it('treats "." as a literal character, not a regex metacharacter', async () => { + // The `$regex` defect restated as a requirement: on the regex-evaluating + // backend "a.b" also matched rows 7 and 8. + expect(await ids({ name: { $icontains: 'a.b' } })).toEqual(['9']); + }); + + // ── The comparand gate ───────────────────────────────────────────────────── + + for (const [label, comparand] of [ + ['an empty string', ''], + ['a number', 42], + ['null', null], + ['a boolean', true], + ] as const) { + it(`REFUSES ${label} comparand, in the ADR-0112 envelope`, async () => { + const err = await refusalOf({ name: { $icontains: comparand } }); + // `code` AND `status`, never `toThrow()` alone: a bare throw assertion is + // satisfied by any error, including the uncoded engine errors this + // driver's whole refusal family exists to keep out of a 500-shaped body. + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('$icontains'); + }); + } + + it('refuses the empty comparand even when a sibling identity would settle the node', async () => { + // The gate is on the validating walk, not in the emitter, so it cannot be + // skipped by a `$or` branch that reduces to TRUE first. An emitter-only gate + // refuses or silently widens depending on the filter's SIBLINGS. + const err = await refusalOf({ $or: [{}, { name: { $icontains: '' } }] }); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + }); + + // ── The retirement ───────────────────────────────────────────────────────── + + for (const [label, where, mustMention] of [ + ['a bare $regex', { name: { $regex: 'ac.*' } }, ['$regex', '$icontains']], + ['a dangling $options', { name: { $options: 'i' } }, ['$options', '$icontains']], + [ + '$regex with $options — one mistake, one fix', + { name: { $regex: '^acme', $options: 'i' } }, + ['$regex', '$options', '$icontains'], + ], + ] as const) { + it(`REFUSES ${label}, naming the replacement`, async () => { + const err = await refusalOf(where); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('RETIRED'); + for (const mention of mustMention) expect(err.message).toContain(mention); + }); + } + + it('the $regex refusal is not `expected: []` — refusing and matching nothing are different', async () => { + // Answering zero rows is what driver-memory already did for an INVALID + // pattern: the silent wrong answer #4706 retired the operator over. A + // conformance suite must be able to tell "refused to run" from "ran and + // matched nothing", which is why this asserts a throw rather than a count. + expect(await ids({ name: { $contains: 'zzz-matches-nothing' } })).toEqual([]); + await expect(driver.find('txt', { where: { name: { $regex: 'zzz' } } }, BYPASS)).rejects.toThrow(); + }); + + // ── The fold, where it is actually observable on SQLite ──────────────────── + + it('compiles LOWER() on both operands, and $contains on neither', async () => { + // On SQLite this is the ONLY witness to the fold: `LIKE` already folds ASCII + // here, so `$contains` and `$icontains` select identical rows for every + // comparand and a dropped `LOWER()` changes no answer. It changes the SQL, + // and it changes the answer on Postgres (whose LIKE is case-exact), so the + // statement is what has to be pinned. + // Identifier quoting is the dialect's (knex renders backticks on the sqlite + // clients), so the assertion is on the SHAPE, not on one dialect's quotes. + const unquote = (sql: string) => sql.replace(/[`"\[\]]/g, ''); + + const icontainsSql = unquote(driver.compileWhere({ name: { $icontains: 'acme' } })); + expect(icontainsSql).toContain('LOWER(name) LIKE LOWER('); + expect(icontainsSql).toContain('ESCAPE'); + // The escaped pattern still travels as the comparand, wildcards and all. + expect(icontainsSql).toContain('%acme%'); + + const containsSql = unquote(driver.compileWhere({ name: { $contains: 'acme' } })); + expect(containsSql).toContain('name LIKE'); + expect(containsSql).not.toContain('LOWER'); + }); +}); diff --git a/packages/drivers/driver-sql/src/sql-driver-like-escape.test.ts b/packages/drivers/driver-sql/src/sql-driver-like-escape.test.ts index 04d80f1c44..b5f0ef1805 100644 --- a/packages/drivers/driver-sql/src/sql-driver-like-escape.test.ts +++ b/packages/drivers/driver-sql/src/sql-driver-like-escape.test.ts @@ -111,20 +111,33 @@ describe('SqlDriver — contains escapes LIKE metacharacters (P0-3)', () => { await knex.destroy(); }); - it('a "%" value matches only rows containing a literal %, not every row', async () => { - const r = await driver.find('docs', { where: { title: { $contains: '%' } } }); - expect(r.map((x: any) => x.id)).toEqual(['1']); - }); + /** + * [#5702] Swept over BOTH operators `applyLike` now serves. + * + * `$icontains` reaches the identical escaping through a new `fold` parameter + * of that method, which wraps both operands in SQL `LOWER()`. Adding the fold + * as a parameter rather than as a second emitter is what makes the character + * class shared code — but "shared" is a claim, and this axis is what checks + * it: a fold written as its own emitter would have re-derived the pattern + * without the `%`/`_`/`\` class, and an unescaped `%` matches EVERY row (the + * P0-3 bypass this describe block is named after) on the newer operator only. + */ + for (const op of ['$contains', '$icontains'] as const) { + it(`${op}: a "%" value matches only rows containing a literal %, not every row`, async () => { + const r = await driver.find('docs', { where: { title: { [op]: '%' } } }); + expect(r.map((x: any) => x.id)).toEqual(['1']); + }); - it('a "_" value matches only rows containing a literal _, not any single char', async () => { - const r = await driver.find('docs', { where: { title: { $contains: '_' } } }); - expect(r.map((x: any) => x.id)).toEqual(['3']); - }); + it(`${op}: a "_" value matches only rows containing a literal _, not any single char`, async () => { + const r = await driver.find('docs', { where: { title: { [op]: '_' } } }); + expect(r.map((x: any) => x.id)).toEqual(['3']); + }); - it('an ordinary substring still matches normally', async () => { - const r = await driver.find('docs', { where: { title: { $contains: 'sale' } } }); - expect(r.map((x: any) => x.id)).toEqual(['1']); - }); + it(`${op}: an ordinary substring still matches normally`, async () => { + const r = await driver.find('docs', { where: { title: { [op]: 'sale' } } }); + expect(r.map((x: any) => x.id)).toEqual(['1']); + }); + } }); // ── The driver axis (#5589, ADR-0053 D-A3) ────────────────────────────────── @@ -237,16 +250,27 @@ function declareLikeEscapeSweep(cell: DialectCell): void { expect(String(rows[0]?.title)).toBe('C:\\logs'); }); - for (const c of LIKE_ESCAPE_CASES) { - it(c.name, async () => { - const rows = await driver.find(LIKE_TABLE, { - where: { title: { $contains: c.value } }, + // [#5702] The OPERATOR axis, crossed with the dialect axis this sweep + // already had. `$icontains` compiles `LOWER(??) LIKE LOWER(?) ESCAPE ?`, so + // every claim this file makes about the escape's survival across a wire + // protocol — mysql2's client-side interpolation of the bound `ESCAPE`, + // Postgres's `standard_conforming_strings`, the backslash surviving the + // server lexer — has to be re-decided with the operands wrapped in a + // function call. The expectations are identical because the escaping is: + // the comparands here are symbols and lower-case ASCII, so the fold changes + // which SQL is emitted, never which rows are right. + for (const op of ['$contains', '$icontains'] as const) { + for (const c of LIKE_ESCAPE_CASES) { + it(`${op}: ${c.name}`, async () => { + const rows = await driver.find(LIKE_TABLE, { + where: { title: { [op]: c.value } }, + }); + const got = rows + .map((r: any) => String(r.id)) + .sort((x: string, y: string) => x.localeCompare(y)); + expect(got, c.note).toEqual([...c.expected]); }); - const got = rows - .map((r: any) => String(r.id)) - .sort((x: string, y: string) => x.localeCompare(y)); - expect(got, c.note).toEqual([...c.expected]); - }); + } } }); } diff --git a/packages/drivers/driver-sql/src/sql-driver-null-operators.test.ts b/packages/drivers/driver-sql/src/sql-driver-null-operators.test.ts index 71d0c44a5a..ab2eee60a8 100644 --- a/packages/drivers/driver-sql/src/sql-driver-null-operators.test.ts +++ b/packages/drivers/driver-sql/src/sql-driver-null-operators.test.ts @@ -117,11 +117,33 @@ describe('SqlDriver — null / empty operators (#2704)', () => { expect(ids(rows)).toEqual(['1']); }); - it('$regex (better-auth contains) → substring LIKE, not exact match', async () => { - const rows = await driver.find('tasks', { where: { assignee: { $regex: 'aro' } } }); + // [#5702] REPLACED. This case was `$regex (better-auth contains) → substring + // LIKE, not exact match` and expected `['3']` — it pinned the `case '$regex':` + // fallthrough that existed for exactly one producer, plugin-auth's ObjectQL + // adapter. #5710 flipped that producer to `$contains`, #4706 retired the + // spelling, and the fallthrough is deleted; there is no substring-LIKE + // answer left to assert. + // + // Its replacement is the same query on the operator that DOES mean this, + // plus the refusal — asserted on `code` and `status`, not on `toThrow()` + // alone, which would stay green against any error including the uncoded ones + // the ADR-0112 envelope exists to eliminate. + it('$contains → substring LIKE, the operator $regex is retired in favour of', async () => { + const rows = await driver.find('tasks', { where: { assignee: { $contains: 'aro' } } }); expect(ids(rows)).toEqual(['3']); }); + it('$regex is REFUSED, in the ADR-0112 envelope, naming $icontains', async () => { + const err = await driver + .find('tasks', { where: { assignee: { $regex: 'aro' } } }) + .then(() => null, (e: any) => e); + expect(err).toBeInstanceOf(Error); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('$regex'); + expect(err.message).toContain('$icontains'); + }); + it('$not (CEL `!expr` scope filter) → negated sub-condition, not a bogus "$not" column', async () => { // `!(assignee == 'alice')` → { $not: { assignee: { $eq: 'alice' } } }. // diff --git a/packages/drivers/driver-sql/src/sql-driver-silent-empty-predicate.test.ts b/packages/drivers/driver-sql/src/sql-driver-silent-empty-predicate.test.ts index a5a73f61df..86011780f2 100644 --- a/packages/drivers/driver-sql/src/sql-driver-silent-empty-predicate.test.ts +++ b/packages/drivers/driver-sql/src/sql-driver-silent-empty-predicate.test.ts @@ -233,7 +233,19 @@ describe('[#5234] SqlDriver refuses the two comparand shapes that compiled to a expect(err.message).toContain('StringOperatorSchema'); }); - for (const op of ['$contains', '$notContains', '$startsWith', '$endsWith', '$regex'] as const) { + // [#5702] This list IS `TEXT_PATTERN_OPERATORS` — the operators whose + // comparand becomes the TEXT of a LIKE pattern — so it follows that set's + // membership. `$regex` left it (retired, #4706) and `$icontains` joined it. + // + // Re-spelled rather than merely dropped, because the row was pinning a real + // condition and the new member needs it more, not less: `$icontains` is the + // one text operator whose comparand is ALSO gated on the validating walk, + // so without a row here nothing would notice if the two gates ever + // disagreed about an object. (Which one fires is deliberately not asserted — + // both answer `INVALID_FILTER` / 400 naming the operator, which is the + // contract; the walk simply runs first.) The retired spelling's own refusal + // is pinned in `sql-driver-icontains-and-retired-operators.test.ts`. + for (const op of ['$contains', '$notContains', '$startsWith', '$endsWith', '$icontains'] as const) { it(`\`${op}\` refuses an object comparand`, async () => { const err = await refusalOf(() => find({ name: { [op]: { foo: 1 } } })); expect(err.code, op).toBe('INVALID_FILTER'); diff --git a/packages/drivers/driver-sql/src/sql-driver.ts b/packages/drivers/driver-sql/src/sql-driver.ts index 90014afa4f..b5d6663912 100644 --- a/packages/drivers/driver-sql/src/sql-driver.ts +++ b/packages/drivers/driver-sql/src/sql-driver.ts @@ -30,6 +30,12 @@ import { // so the engine and this driver can never disagree about what may become a // physical column DEFAULT. import { isNowDefaultToken, isRuntimeDefaultToken } from '@objectstack/spec/data'; +// [#5702] The retired filter operators and the prescription each refusal +// prints. Read from the spec rather than restated here for the reason the +// #5701 table itself gives: five refusal sites that each write their own +// sentence about `$regex` are five sentences that drift apart. This driver +// prints `why` VERBATIM. +import { RETIRED_FILTER_OPERATORS } from '@objectstack/spec/data'; import type { DriverQuery, IDataDriver } from '@objectstack/spec/contracts'; import { StandardErrorCode } from '@objectstack/spec/api'; import { StorageNameMapping } from '@objectstack/spec/system'; @@ -715,6 +721,67 @@ function filterArrayReachedDriverError(filters: unknown[]): Error { ); } +/** + * [#5702] A RETIRED filter operator reached the compiler. + * + * Separate from {@link unknownFieldOperatorMessage}'s "this name is not in the + * vocabulary" on purpose: `$regex` and `$options` were not typos, they were + * spellings this driver ANSWERED until #4706 retired them, and the author who + * wrote one needs the replacement rather than a list to search. The + * prescription is `RETIRED_FILTER_OPERATORS[op].why`, printed verbatim — the + * spec table exists so that the five refusal sites stop each composing their + * own sentence about the same retirement. + * + * `siblings` are the other keys of the SAME field constraint, and every retired + * one among them is named too. `{ $regex: '^acme', $options: 'i' }` is ONE + * mistake with ONE fix (write `$icontains`), so a message naming only the key + * the loop happened to reach first would send its author back for a second + * round-trip on the other one. + * + * Returns `null` when `op` is not retired, so the caller can fall through to + * the ordinary unknown-operator refusal with one expression. + */ +function retiredFilterOperatorError(op: string, field: string, siblings: readonly string[] = []): Error | null { + const guidance = RETIRED_FILTER_OPERATORS[op]; + if (!guidance) return null; + const replacement = guidance.to ? ` Write "${guidance.to}" instead.` : ''; + const alsoRetired = siblings.filter((key) => key !== op && RETIRED_FILTER_OPERATORS[key]); + const also = alsoRetired.length + ? ` The same field constraint also carries the retired ` + + `${alsoRetired.map((key) => `"${key}"`).join(', ')} — one "${guidance.to}" replaces the whole ` + + `shape, so this is ONE mistake with ONE fix, not one per key.` + : ''; + return unsupportedFilterError( + `Filter operator "${op}" on field "${field}" is RETIRED and is no longer evaluated by this ` + + `driver.${replacement} ${guidance.why}${also}`, + ); +} + +/** + * [#5702] `$icontains` received a comparand that is not a non-empty string. + * + * Two rejections, one constructor, because they are one mistake at the + * comparand position and the repair is the same sentence: + * + * - **non-string** — `StringOperatorSchema` declares `$icontains: z.string()`. + * Coercing `42` to `"42"` answers a query nobody wrote (the reading + * `applyLike`'s `String(value)` would otherwise give it). + * - **empty string** — every row contains the empty substring, so the predicate + * constrains nothing. A dropped predicate WIDENS a result set, and on an RLS + * read scope that is a permission bypass rather than a degraded filter + * (#3948) — the same reason #5240 refused `{ field: {} }` one level up. + */ +function icontainsComparandError(field: string, value: unknown, path: string): Error { + const shown = typeof value === 'string' ? `""` : JSON.stringify(value) ?? String(value); + return unsupportedFilterError( + `Operator "$icontains" on field "${field}" at ${path} requires a NON-EMPTY string comparand, ` + + `received ${shown}. "$icontains" is a case-insensitive LITERAL substring search, so its ` + + `comparand is the text to look for — an empty one matches every row (a predicate that ` + + `constrains nothing), and a non-string one would have to be coerced into text this query ` + + `never asked for.`, + ); +} + /** * [#5041] The referenced field name when `value` is a Filter Protocol FIELD * REFERENCE (`{ $field: 'other_column' }` — spec `FieldReferenceSchema` in @@ -812,7 +879,7 @@ function isBindableComparand(value: unknown): boolean { * object. */ const TEXT_PATTERN_OPERATORS: ReadonlySet = new Set([ - '$contains', '$notContains', '$startsWith', '$endsWith', '$regex', + '$contains', '$notContains', '$startsWith', '$endsWith', '$icontains', ]); /** @@ -1506,6 +1573,20 @@ function classifyFilterKey(key: string, value: unknown, here: string): FilterVer throw nonBooleanExistsComparandError(key, value.$exists, `${here}.$exists`); } + // [#5702] `$icontains`'s comparand is a NON-EMPTY string by declaration, + // refused on this walk for the same evaluation-order reason as the two gates + // above: an empty comparand makes the predicate match every row, and a gate + // the emitter carries alone is skipped wholesale whenever a boolean identity + // settles the enclosing node — so `{ $or: [ {}, { name: { $icontains: '' } } ] }` + // would be refused or silently widened depending on its SIBLINGS. + if ( + isFilterNode(value) && + Object.prototype.hasOwnProperty.call(value, '$icontains') && + (typeof value.$icontains !== 'string' || value.$icontains === '') + ) { + throw icontainsComparandError(key, value.$icontains, `${here}.$icontains`); + } + // A field key always contributes a predicate. return 'clause'; } @@ -7128,12 +7209,29 @@ export class SqlDriver implements IDataDriver { value: unknown, shape: 'contains' | 'starts' | 'ends', negate = false, + fold = false, ): void { const escaped = String(value).replace(/[\\%_]/g, '\\$&'); const pattern = shape === 'starts' ? `${escaped}%` : shape === 'ends' ? `%${escaped}` : `%${escaped}%`; const keyword = negate ? 'NOT LIKE' : 'LIKE'; const rawMethod = method.startsWith('or') ? 'orWhereRaw' : 'whereRaw'; - builder[rawMethod](`?? ${keyword} ? ESCAPE ?`, [field, pattern, '\\']); + // [#5702] `fold` wraps BOTH operands in SQL `LOWER()` — the `$icontains` + // lowering. It is a parameter of this method rather than a second emitter + // so that the escaping above (the `%`/`_`/`\` class and the bound `ESCAPE`) + // is literally the same code, not a copy held in sync by a comment: an + // unescaped `%` is a filter bypass (P0), and the second `$icontains` face + // is exactly where a copy would have skipped it. + // + // `LOWER()` and not a JS-side fold: the column side has to fold too, and it + // can only fold in SQL. SQLite's `lower()` folds ASCII ONLY, which IS the + // contract (#4706 Q1 = A) — `É` stays `É`, so `$icontains: 'café'` does not + // match `CAFÉ`. Postgres and MySQL fold the wider Unicode range in + // `LOWER()`, so on those dialects this over-matches on non-ASCII letters; + // that divergence is measured and recorded rather than papered over, and it + // is the same dialect axis `$contains`'s case sensitivity sits on. + const col = fold ? 'LOWER(??)' : '??'; + const bound = fold ? 'LOWER(?)' : '?'; + builder[rawMethod](`${col} ${keyword} ${bound} ESCAPE ?`, [field, pattern, '\\']); } /** @@ -7361,14 +7459,17 @@ export class SqlDriver implements IDataDriver { break; } case '$contains': - // `$regex` reaches SQL only via the better-auth adapter, which emits - // it for a `contains` search (a plain substring, not a real regex). - // SQL has no portable regex, so compile the intended substring LIKE - // — correct for that producer and safe (the value is LIKE-escaped), - // where the old equality default silently made it an exact match. - case '$regex': this.applyContainsLike(builder, method, field, opValue); break; + // [#5702] The case-INSENSITIVE twin of `$contains`, and the + // replacement `RETIRED_FILTER_OPERATORS` prescribes for `$regex`. + // Same `applyLike` — same escaped character class, same bound + // `ESCAPE` — with the fold applied to BOTH sides, because folding + // only the comparand compares a folded needle against a raw column + // and matches just the rows that were already lower-case. + case '$icontains': + this.applyLike(builder, method, field, opValue, 'contains', false, true); + break; case '$notContains': // [#5298] NULL-safe: `NOT LIKE` is UNKNOWN for a NULL column, and // "does not contain" is true of a value that is not there. @@ -7413,12 +7514,18 @@ export class SqlDriver implements IDataDriver { ? (logicalOp === 'or' ? 'orWhereNull' : 'whereNull') : (logicalOp === 'or' ? 'orWhereNotNull' : 'whereNotNull')](field); break; - default: + default: { + // [#5702] A RETIRED spelling gets the prescription, not the + // vocabulary list: the author who wrote `$regex` needs + // `$icontains`, and a list of fifteen names does not say so. + const retired = retiredFilterOperatorError(op, field, Object.keys(value as object)); + if (retired) throw retired; throw unsupportedFilterError( `Unsupported filter operator "${op}" on field "${field}". Supported operators: ` + `$eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $between, $contains, $notContains, ` + - `$startsWith, $endsWith, $regex, $null, $exists.`, + `$startsWith, $endsWith, $icontains, $null, $exists.`, ); + } } } } else { diff --git a/packages/drivers/driver-sqlite-wasm/src/sqlite-wasm-icontains-and-retired-operators.test.ts b/packages/drivers/driver-sqlite-wasm/src/sqlite-wasm-icontains-and-retired-operators.test.ts new file mode 100644 index 0000000000..9656005a63 --- /dev/null +++ b/packages/drivers/driver-sqlite-wasm/src/sqlite-wasm-icontains-and-retired-operators.test.ts @@ -0,0 +1,105 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#5702] `$icontains` and the `$regex` retirement, EXECUTED by sql.js. + * + * `SqliteWasmDriver extends SqlDriver`, so the compiler is inherited and nothing + * here re-implements it. What this pins is the other half — the half its + * temporal, pagination and filter-logic suites exist for: the compiled predicate + * has to survive a different ENGINE. This driver swaps knex's transport for a + * custom sql.js dialect (`Client_WasmSqlite`) that compiles the statement, binds + * its parameters and marshals the rows back through its own path. + * + * `$icontains` is the first operator this package has ever run whose predicate + * is a FUNCTION CALL on the column (`LOWER(col) LIKE LOWER(?) ESCAPE ?`) rather + * than a bare column reference. Every text predicate before it compiled to + * `col LIKE ?`. A dialect that mis-binds the parameters of the three-argument + * form, or that renders `??` inside a function call differently, produces + * precisely the failure a shared standard exists to rule out — a filter that + * looks applied and selects the wrong rows — and it would fail in no other suite + * in the repo. "It inherits the compiler, therefore it is fine" is the + * assumption these suites exist to disprove. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import type { DriverOptions, FilterCondition } from '@objectstack/spec/data'; +import { FILTER_TEXT_ROWS } from '@objectstack/spec/data'; +import { SqliteWasmDriver } from './index.js'; + +interface WireBearingError extends Error { + code?: string; + status?: number; +} + +/** Diagnostics-only; it never changes which rows a read touches. */ +const BYPASS: DriverOptions = { bypassTenantAudit: true }; + +describe('[#5702] driver-sqlite-wasm — $icontains and the retired $regex, on sql.js', () => { + let driver: SqliteWasmDriver; + + beforeAll(async () => { + driver = new SqliteWasmDriver({ filename: ':memory:' }); + await driver.initObjects([{ name: 'txt', fields: { name: { type: 'string' } } }]); + for (const row of FILTER_TEXT_ROWS) { + await driver.create('txt', { ...row }, BYPASS); + } + }); + + afterAll(async () => { + await driver.disconnect(); + }); + + const ids = async (where: FilterCondition): Promise => { + const rows = await driver.find('txt', { where }, BYPASS); + return rows.map((r) => String(r.id)).sort((a, b) => a.localeCompare(b)); + }; + + it('the fixture really is all nine rows', async () => { + expect(await ids({})).toEqual(['1', '2', '3', '4', '5', '6', '7', '8', '9']); + }); + + it('$icontains folds ASCII case in both directions, through the wasm engine', async () => { + expect(await ids({ name: { $icontains: 'acme' } })).toEqual(['1', '2']); + expect(await ids({ name: { $icontains: 'ACME' } })).toEqual(['1', '2']); + }); + + it('$icontains folds ASCII ONLY — sql.js `lower()` must not reach É', async () => { + // The dialect boundary #4706 Q1 = A pinned, asserted against THIS engine's + // `lower()` rather than against better-sqlite3's. Both are SQLite, but they + // are separately compiled builds and an ICU-enabled one would fold É and + // answer ['3','4'] to both lines. + expect(await ids({ name: { $icontains: 'café' } })).toEqual(['4']); + expect(await ids({ name: { $icontains: 'CAFÉ' } })).toEqual(['3']); + }); + + it('$icontains keeps the LIKE metacharacters literal across the wasm bind path', async () => { + // The bound `ESCAPE ?` is a THIRD parameter on a predicate whose first + // operand is now a function call. This is the assertion that a wasm dialect + // binding those three positionally in the wrong order would fail. + expect(await ids({ name: { $icontains: '100%' } })).toEqual(['5']); + expect(await ids({ name: { $icontains: 'a_b' } })).toEqual(['7']); + expect(await ids({ name: { $icontains: 'a.b' } })).toEqual(['9']); + }); + + it('REFUSES the retired $regex, in the ADR-0112 envelope, naming $icontains', async () => { + const err = await driver + .find('txt', { where: { name: { $regex: 'ac.*' } } }, BYPASS) + .then(() => null, (e: unknown) => e as WireBearingError); + expect(err).toBeInstanceOf(Error); + expect(err!.code).toBe('INVALID_FILTER'); + expect(err!.status).toBe(400); + expect(err!.message).toContain('$regex'); + expect(err!.message).toContain('$icontains'); + }); + + it('REFUSES an $icontains comparand that constrains nothing', async () => { + for (const comparand of ['', 42] as const) { + const err = await driver + .find('txt', { where: { name: { $icontains: comparand } } }, BYPASS) + .then(() => null, (e: unknown) => e as WireBearingError); + expect(err, `expected ${JSON.stringify(comparand)} to be refused`).toBeInstanceOf(Error); + expect(err!.code).toBe('INVALID_FILTER'); + expect(err!.status).toBe(400); + } + }); +}); diff --git a/packages/drivers/driver-turso/src/remote-transport-node-operator-refusal.test.ts b/packages/drivers/driver-turso/src/remote-transport-node-operator-refusal.test.ts index 3711c5db27..a4c4e163cf 100644 --- a/packages/drivers/driver-turso/src/remote-transport-node-operator-refusal.test.ts +++ b/packages/drivers/driver-turso/src/remote-transport-node-operator-refusal.test.ts @@ -122,6 +122,14 @@ const UNDECLARED: Array<[label: string, where: unknown, key: string, path: strin ['$nor at the top level', { $nor: [{ stage: 'won' }] }, '$nor', 'where.$nor'], ['$expr at the top level', { $expr: { $eq: ['$stage', 'won'] } }, '$expr', 'where.$expr'], ['$elemMatch at the top level', { $elemMatch: { stage: 'won' } }, '$elemMatch', 'where.$elemMatch'], + // [#5702] `$regex` MOVED here from the misplaced-field-operator table below. + // It sat there because it was a field operator this transport compiled ("and + // `$regex` because better-auth's adapter really emits it"); #4706 retired it, + // so at the node position it is no longer one level too high — it names + // nothing this protocol declares at any level, which is the tail this table + // asserts. Re-spelling the row rather than deleting it keeps the shape's + // answer pinned; only which of the two tails it takes has changed. + ['$regex at the top level', { $regex: 'wo' }, '$regex', 'where.$regex'], ['$where inside $or', { $or: [{ $where: 'x' }] }, '$where', 'where.$or[0].$where'], ['$nor inside $and', { $and: [{ $nor: [{ stage: 'won' }] }] }, '$nor', 'where.$and[0].$nor'], ['$expr inside $not', { $not: { $expr: 1 } }, '$expr', 'where.$not.$expr'], @@ -140,8 +148,13 @@ const UNDECLARED: Array<[label: string, where: unknown, key: string, path: strin * hand-written or AI-authored filter produces when the field name is dropped, * and every one of them compiled to a predicate on a column named after the * operator. `$between` is included even though this transport never compiles it - * (TursoDriver lowers it first) — misplaced is misplaced — and `$regex` because - * better-auth's adapter really emits it. + * (TursoDriver lowers it first) — misplaced is misplaced — and `$icontains` + * because this transport compiles it (#5702) even though `FILTER_OPERATORS` + * does not list it yet. + * + * [#5702] `$regex` LEFT this table for the undeclared one above: it is retired, + * so it is not a field operator at any level and its author must not be told to + * write `{ field: { $regex: … } }`. */ const MISPLACED: Array<[label: string, where: unknown, key: string]> = [ ['$eq', { $eq: 'won' }, '$eq'], @@ -157,7 +170,7 @@ const MISPLACED: Array<[label: string, where: unknown, key: string]> = [ ['$notContains', { $notContains: 'wo' }, '$notContains'], ['$startsWith', { $startsWith: 'w' }, '$startsWith'], ['$endsWith', { $endsWith: 'n' }, '$endsWith'], - ['$regex', { $regex: 'wo' }, '$regex'], + ['$icontains', { $icontains: 'wo' }, '$icontains'], ['$null', { $null: true }, '$null'], ['$exists', { $exists: true }, '$exists'], ]; diff --git a/packages/drivers/driver-turso/src/remote-transport-text-predicates.test.ts b/packages/drivers/driver-turso/src/remote-transport-text-predicates.test.ts index e99070ce71..91b37bb6b3 100644 --- a/packages/drivers/driver-turso/src/remote-transport-text-predicates.test.ts +++ b/packages/drivers/driver-turso/src/remote-transport-text-predicates.test.ts @@ -53,12 +53,26 @@ const META_ROWS = [ async function makeRemoteDriver(schema: Record, rows: Record[]) { const stub = makeLibsqlSqliteStub(); - const driver = new TursoDriver({ url: 'libsql://text.turso.io', client: stub as never }); + // [#5702] The stub is wrapped so the STATEMENTS are readable, not only the + // rows. Rows are the right witness for almost everything this file pins — but + // `$icontains`'s fold is invisible in rows on SQLite (`LIKE` already folds + // ASCII there), so the emitted SQL is the only place a dropped `LOWER()` can + // be seen. Recording here rather than standing up a third hand-rolled mock + // client keeps every case on the same engine. + const executed: string[] = []; + const recording: LibsqlSqliteStub = { + ...stub, + async execute(stmt: unknown) { + executed.push(typeof stmt === 'string' ? stmt : String((stmt as { sql?: unknown }).sql ?? '')); + return stub.execute(stmt); + }, + }; + const driver = new TursoDriver({ url: 'libsql://text.turso.io', client: recording as never }); await driver.connect(); expect(driver.transportMode).toBe('remote'); await driver.syncSchema(schema.name as string, schema); for (const row of rows) await driver.create(schema.name as string, row); - return { driver, stub }; + return { driver, stub, executed }; } const ids = async (driver: TursoDriver, object: string, where: DriverQuery['where']) => @@ -67,9 +81,10 @@ const ids = async (driver: TursoDriver, object: string, where: DriverQuery['wher describe('TursoDriver remote — declared text predicates return rows', () => { let driver: TursoDriver; let stub: LibsqlSqliteStub; + let executed: string[]; beforeAll(async () => { - ({ driver, stub } = await makeRemoteDriver(TEXT_OBJECT, ROWS)); + ({ driver, stub, executed } = await makeRemoteDriver(TEXT_OBJECT, ROWS)); }); afterAll(async () => { @@ -99,11 +114,96 @@ describe('TursoDriver remote — declared text predicates return rows', () => { expect(await ids(driver, 'widget', { name: { $contains: 'lp' } })).toEqual(['w1', 'w2']); }); - // Not spec-declared: better-auth's adapter emits `$regex` for a plain - // substring search, and SqlDriver compiles it as one. Remote must agree or a - // Turso-backed auth store answers differently from a local one. - it('$regex compiles as the substring search its only producer means', async () => { - expect(await ids(driver, 'widget', { name: { $regex: 'lph' } })).toEqual(['w1']); + /** + * [#5702] REPLACED. This slot held `$regex compiles as the substring search + * its only producer means` — `{ name: { $regex: 'lph' } } → ['w1']` — kept in + * lockstep with `SqlDriver`'s identical fallthrough so a Turso-backed auth + * store answered like a local one. #5710 flipped that producer to `$contains` + * and #4706 retired the spelling on all five backends, so the row it pinned + * no longer exists. + * + * What takes its place is the operator that replaces it. `$icontains` is + * REMOTE-side work in its own right: this transport does not go through knex, + * it hand-assembles its own `LIKE`, so "local inherits SqlDriver" buys it + * nothing — the fold has to be written here too, and the only witness that it + * was is rows. + */ + it('$icontains folds ASCII case, in both directions', async () => { + expect(await ids(driver, 'widget', { name: { $icontains: 'alp' } })).toEqual(['w1', 'w2']); + expect(await ids(driver, 'widget', { name: { $icontains: 'ALP' } })).toEqual(['w1', 'w2']); + // The fold must run on BOTH operands: folding only the comparand leaves the + // column raw and quietly matches nothing here, since no row is lower-case. + expect(await ids(driver, 'widget', { name: { $icontains: 'BETA' } })).toEqual(['w3']); + }); + + /** + * [#5702] MEASURED, and the measurement contradicts the assertion this case + * was first written with — recorded as it came out rather than as it was + * predicted. + * + * The intended pin was `$contains: 'ALP'` → `[]` beside `$icontains: 'ALP'` → + * `['w1','w2']`, i.e. the two operators told apart by their answers. It fails: + * `$contains` returns `['w1','w2']` too. SQLite's `LIKE` folds ASCII case by + * itself, so on this dialect `col LIKE '%ALP%'` and `LOWER(col) LIKE + * LOWER('%ALP%')` select the SAME rows for EVERY comparand — the second fold + * is a no-op on top of the first. + * + * That is not a defect in `$icontains`; it is the `$contains` half of the + * #4706 Q2 = A ruling ("the `$contains` family is case-SENSITIVE"), which is + * NOT delivered by this PR. Making SQLite's LIKE case-exact needs a different + * construct (GLOB / `instr()` / a binary collation) applied in three places + * that must move together — this transport, `SqlDriver.applyLike`, and the + * RLS/analytics twins (`read-scope-sql`, `service-analytics`'s + * `like-pattern.ts`) — or one permission rule compiles to two row sets. It is + * filed separately; the `FILTER_TEXT` DEBT rows in + * `scripts/check-driver-conformance.mjs` stay open for it. + * + * So the honest pin here is the CURRENT pair — equal on this dialect — plus + * the compiled SQL, which is the only place the fold is observable on SQLite + * and therefore the only thing that can catch a `LOWER()` silently dropped. + */ + it('$contains and $icontains agree on SQLite today — LIKE already folds ASCII', async () => { + expect(await ids(driver, 'widget', { name: { $contains: 'ALP' } })).toEqual(['w1', 'w2']); + expect(await ids(driver, 'widget', { name: { $icontains: 'ALP' } })).toEqual(['w1', 'w2']); + }); + + it('$icontains compiles LOWER() on BOTH operands, where $contains compiles neither', async () => { + executed.length = 0; + await driver.find('widget', { where: { name: { $icontains: 'ALP' } } }); + const icontainsSql = executed.join('\n'); + expect(icontainsSql).toContain('LOWER("name") LIKE LOWER(?) ESCAPE'); + + executed.length = 0; + await driver.find('widget', { where: { name: { $contains: 'ALP' } } }); + const containsSql = executed.join('\n'); + expect(containsSql).toContain('"name" LIKE ? ESCAPE'); + expect(containsSql).not.toContain('LOWER'); + }); + + it('REFUSES the retired $regex, in the ADR-0112 envelope, naming $icontains', async () => { + const err = await driver + .find('widget', { where: { name: { $regex: 'lph' } } }) + .then(() => null, (e: any) => e); + expect(err).toBeInstanceOf(Error); + // `code` and `status`, not a bare rejection: this transport's whole family + // of filter refusals exists to be a 400-class client error rather than an + // opaque 500, and `rejects.toThrow()` alone cannot tell the two apart. + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('$regex'); + expect(err.message).toContain('$icontains'); + }); + + it('REFUSES an $icontains comparand that constrains nothing', async () => { + for (const comparand of ['', 42]) { + const err = await driver + .find('widget', { where: { name: { $icontains: comparand } } }) + .then(() => null, (e: any) => e); + expect(err, `expected ${JSON.stringify(comparand)} to be refused`).toBeInstanceOf(Error); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('$icontains'); + } }); it('$null: true / false select the null and non-null rows', async () => { @@ -171,6 +271,31 @@ describe('TursoDriver remote — LIKE metacharacters match literally', () => { it('an ordinary substring is unaffected by the escaping', async () => { expect(await ids(driver, 'meta', { name: { $contains: 'sale' } })).toEqual(['m1']); }); + + /** + * [#5702] The same three metacharacters, through `$icontains`. + * + * Written out per character rather than once, because the escape is the P0 + * here and `$icontains` reaches it through a NEW parameter of `pushLike` + * (`fold`) that wraps both operands in `LOWER()`. A fold implemented as a + * second emitter — the obvious shape, and the one this parameter exists to + * avoid — would have re-derived the pattern without the `%`/`_`/`\` class, + * and an unescaped `%` matches every row. + */ + it('$icontains escapes "%" — a "%" comparand must not match every row', async () => { + expect(await ids(driver, 'meta', { name: { $icontains: '%' } })).toEqual(['m1']); + expect(await ids(driver, 'meta', { name: { $icontains: '% OFF' } })).toEqual(['m1']); + }); + + it('$icontains escapes "_" — a literal underscore, not any single character', async () => { + expect(await ids(driver, 'meta', { name: { $icontains: '_' } })).toEqual(['m3']); + expect(await ids(driver, 'meta', { name: { $icontains: 'A_B' } })).toEqual(['m3']); + }); + + it('$icontains escapes "\\" — a literal backslash, not the escape character', async () => { + expect(await ids(driver, 'meta', { name: { $icontains: '\\' } })).toEqual(['m4']); + expect(await ids(driver, 'meta', { name: { $icontains: 'BACK\\SLASH' } })).toEqual(['m4']); + }); }); /** diff --git a/packages/drivers/driver-turso/src/remote-transport.ts b/packages/drivers/driver-turso/src/remote-transport.ts index e68d857e3a..d28f701040 100644 --- a/packages/drivers/driver-turso/src/remote-transport.ts +++ b/packages/drivers/driver-turso/src/remote-transport.ts @@ -14,7 +14,7 @@ import type { Client, InStatement, ResultSet } from '@libsql/client'; import { StandardErrorCode } from '@objectstack/spec/api'; -import { FILTER_OPERATORS, LOGICAL_OPERATORS } from '@objectstack/spec/data'; +import { FILTER_OPERATORS, LOGICAL_OPERATORS, RETIRED_FILTER_OPERATORS } from '@objectstack/spec/data'; // The DECLARED aggregate vocabulary (#5907) — read from the spec so this // transport's "the protocol has no such function" refusal cannot drift from what // `AggregationNodeSchema.function` admits, nor from the local driver's twin. @@ -45,8 +45,14 @@ const SAFE_IDENTIFIER = /^[a-zA-Z_][a-zA-Z0-9_]*$/; * * It is the spec's `FieldOperatorsSchema` list minus `$between`, which the * driver lowers before the filter gets here (see {@link unsupportedOperator}), - * plus `$regex`, which is not spec-declared but is what better-auth's adapter - * emits for a substring search and what `SqlDriver` therefore compiles. + * plus `$icontains`, which `StringOperatorSchema` declares (#5701) and this + * transport compiles (#5702). + * + * `$regex` was here until #5702 and is now RETIRED: it is not a member, so it + * reaches {@link unsupportedOperator} and is refused with the spec's + * prescription. Its last live producer — plugin-auth's ObjectQL adapter — was + * flipped to `$contains` by #5710 first, which is why the refusal can land at + * all. */ const SUPPORTED_FILTER_OPERATORS = [ '$eq', @@ -61,7 +67,7 @@ const SUPPORTED_FILTER_OPERATORS = [ '$notContains', '$startsWith', '$endsWith', - '$regex', + '$icontains', '$null', '$exists', ] as const; @@ -87,16 +93,23 @@ const NODE_COMBINATORS: ReadonlySet = new Set(LOGICAL_OPERATORS) * it cannot — both branches of {@link RemoteTransport.undeclaredCombinator} * throw `INVALID_FILTER`, they differ only in the repair they suggest. * - * `$regex` is added for the same reason {@link SUPPORTED_FILTER_OPERATORS} - * carries it: it is not spec-declared but is what better-auth's adapter emits, - * so a caller really can misplace it. `$between` comes in with the spec list - * even though this transport never compiles it (TursoDriver lowers it first) — - * a misplaced `$between` is still a misplaced FIELD operator, and telling its - * author "unknown combinator" would send them looking for the wrong mistake. + * `$icontains` is added for the same reason {@link SUPPORTED_FILTER_OPERATORS} + * carries it: `FILTER_OPERATORS` deliberately does not list it yet (#5701 + * staged the declaration ahead of the implementations), but this transport + * compiles it, so a caller really can misplace it. `$between` comes in with the + * spec list even though this transport never compiles it (TursoDriver lowers it + * first) — a misplaced `$between` is still a misplaced FIELD operator, and + * telling its author "unknown combinator" would send them looking for the wrong + * mistake. + * + * The RETIRED spellings are deliberately NOT here (#5702). A node-position + * `$regex` is not a misplaced field operator — it is not a field operator at + * any level any more — so it takes the "names nothing this protocol declares" + * tail, which is the true one. */ const MISPLACED_FIELD_OPERATORS: ReadonlySet = new Set([ ...FILTER_OPERATORS, - '$regex', + '$icontains', ]); /** @@ -247,7 +260,7 @@ type NullGuard = 'none' | 'requireValue' | 'allowNull'; * `match`, `formula` `matchesFilterCondition`) give a value that is not there? * * The default is the large positive-comparison family (`$gt`, `$in`, - * `$contains`, `$startsWith`, `$endsWith`, `$regex`, and any operator this + * `$contains`, `$icontains`, `$startsWith`, `$endsWith`, and any operator this * transport refuses outright), every member of which answers `false` for a value * that is not there. */ @@ -1852,14 +1865,27 @@ export class RemoteTransport { // and refusing it in one family while tolerating it in the other // would leave the failure mode alive at a different spelling. case '$contains': - // `$regex` is not spec-declared: it reaches SQL only via the - // better-auth adapter, which emits it for a `contains` search (a - // plain substring, not a real regex). SqlDriver compiles it as - // that substring LIKE, so remote mode must too — otherwise a - // Turso-backed auth store answers differently from a local one. - case '$regex': this.pushLike(clauses, args, column, this.serializeComparand(object, key, op, opValue), 'contains'); break; + // [#5702] `$icontains` — the case-insensitive twin, and what + // `RETIRED_FILTER_OPERATORS` prescribes in place of `$regex`. Same + // `pushLike`, so the escape rule cannot fork between the two; the + // fold is applied to BOTH operands (see {@link pushLike}). + case '$icontains': + if (typeof opValue !== 'string' || opValue === '') { + throw this.icontainsComparand(object, key, opValue); + } + this.pushLike( + clauses, + args, + column, + this.serializeComparand(object, key, op, opValue), + 'contains', + false, + false, + true, + ); + break; case '$notContains': // [#5298] NULL-safe: `NOT LIKE` is UNKNOWN for a NULL column, and // "does not contain" is true of a value that is not there. The @@ -1950,7 +1976,7 @@ export class RemoteTransport { // reads exactly like "no rows matched" (#1004). An operator this // transport cannot compile is a programming error and must say // so. - throw this.unsupportedOperator(object, key, op); + throw this.unsupportedOperator(object, key, op, Object.keys(value as object)); } } if (clauses.length === clausesBefore) { @@ -2025,10 +2051,23 @@ export class RemoteTransport { shape: LikeShape, negate = false, nullSafe = false, + fold = false, ): void { const escaped = String(value).replace(/[\\%_]/g, '\\$&'); const pattern = shape === 'starts' ? `${escaped}%` : shape === 'ends' ? `%${escaped}` : `%${escaped}%`; - const predicate = `${column} ${negate ? 'NOT LIKE' : 'LIKE'} ? ESCAPE '\\'`; + // [#5702] `fold` wraps BOTH operands in `LOWER()` — the `$icontains` + // lowering, and a parameter of THIS method rather than a sixth text arm for + // the reason the paragraph above gives: one escape rule, one place. Folding + // only the comparand would compare a folded needle against a raw column and + // silently match only the rows that were already lower-case. + // + // libSQL is SQLite, whose `lower()` folds ASCII ONLY — which is the + // contract (#4706 Q1 = A), not a limitation to work around: `É` does not + // fold, so `$icontains: 'café'` must not match `CAFÉ`. The local transport + // reaches the same answer through `SqlDriver.applyLike`'s `LOWER()`. + const lhs = fold ? `LOWER(${column})` : column; + const rhs = fold ? 'LOWER(?)' : '?'; + const predicate = `${lhs} ${negate ? 'NOT LIKE' : 'LIKE'} ${rhs} ESCAPE '\\'`; clauses.push(nullSafe ? this.nullSafeNegative(column, predicate) : predicate); args.push(pattern); } @@ -2127,6 +2166,38 @@ export class RemoteTransport { ); } + /** + * [#5702] The error for an `$icontains` whose comparand is not a NON-EMPTY + * string. + * + * The remote twin of `driver-sql`'s `icontainsComparandError`, and the two + * rejections it covers are one mistake at one position: + * + * - **empty string** — `LOWER(col) LIKE '%%'` is TRUE of every row with a + * value, i.e. a predicate that constrains nothing. A dropped constraint + * WIDENS a result set, and on an RLS read scope that is a permission bypass + * rather than a degraded filter (#3948) — the widening class this file has + * already paid for from three other directions (#1004, #1058, #1073). + * - **non-string** — `StringOperatorSchema` declares `$icontains: z.string()`. + * `pushLike` reaches its comparand through `String(value)`, so a number + * would compile to a text search nobody wrote. + * + * Guarded in the ARM rather than inside `pushLike`, because `pushLike` serves + * the four operators whose comparand rules #1058 already settled; adding a + * fifth rule inside it would make one method answer two different questions. + */ + private icontainsComparand(object: string, field: string, value: unknown): Error { + const shown = typeof value === 'string' ? 'the empty string' : describeValue(value); + return invalidFilterError( + `[RemoteTransport] Operator "$icontains" on '${object}.${field}' requires a NON-EMPTY string ` + + `comparand. Received ${shown} (${preview(value)}). "$icontains" is a case-insensitive ` + + `LITERAL substring search, so its comparand is the text to look for: an empty one matches ` + + `every row that has a value (a predicate that constrains nothing), and a non-string one ` + + `would be coerced into text this query never asked for. @objectstack/spec ` + + `StringOperatorSchema declares $icontains as a string.`, + ); + } + /** * The error for an `$exists` whose comparand is not a boolean (#5369, landed * on this face by #5903). @@ -2177,8 +2248,34 @@ export class RemoteTransport { * the rule — so reaching this point means the lowering step was bypassed, and * saying which step it was is the whole value of the message. */ - private unsupportedOperator(object: string, field: string, op: string): Error { + private unsupportedOperator( + object: string, + field: string, + op: string, + siblings: readonly string[] = [], + ): Error { const target = `'${object}.${field}'`; + // [#5702] A RETIRED spelling is not an unknown name — it is one this + // transport ANSWERED until #4706 retired it, so its author needs the + // replacement rather than the vocabulary list. The prescription is the + // spec's `RETIRED_FILTER_OPERATORS[op].why`, printed verbatim so that this + // transport, `SqlDriver`, `driver-memory` and `driver-mongodb` say ONE + // thing about the same retirement. Every retired SIBLING is named too: + // `{ $regex, $options }` is one mistake with one fix. + const retired = RETIRED_FILTER_OPERATORS[op]; + if (retired) { + const replacement = retired.to ? ` Write "${retired.to}" instead.` : ''; + const alsoRetired = siblings.filter((key) => key !== op && RETIRED_FILTER_OPERATORS[key]); + const also = alsoRetired.length + ? ` The same field constraint also carries the retired ` + + `${alsoRetired.map((key) => `"${key}"`).join(', ')} — one "${retired.to}" replaces the ` + + `whole shape, so this is ONE mistake with ONE fix, not one per key.` + : ''; + return invalidFilterError( + `[RemoteTransport] Filter operator "${op}" on ${target} is RETIRED and is no longer ` + + `compiled in remote mode.${replacement} ${retired.why}${also}`, + ); + } if (op === '$between') { return invalidFilterError( `[RemoteTransport] $between on ${target} must be lowered to $gte/$lte before it reaches the ` + diff --git a/packages/objectql/src/having-filter.test.ts b/packages/objectql/src/having-filter.test.ts index 1abad272dd..03d5386fd4 100644 --- a/packages/objectql/src/having-filter.test.ts +++ b/packages/objectql/src/having-filter.test.ts @@ -76,9 +76,41 @@ describe('matchesHaving — the unknown-operator refusal', () => { .toThrow(/Unsupported operator '\$nand'/); }); - it('accepts $regex with $options as its sibling, not an operator', () => { - expect(matchesHaving({ k: 'Alpha' }, { k: { $regex: '^alp', $options: 'i' } })).toBe(true); - }); + /** + * [#5702] REPLACED. This case read `accepts $regex with $options as its + * sibling, not an operator` and asserted `matchesHaving({ k: 'Alpha' }, + * { k: { $regex: '^alp', $options: 'i' } }) === true` — HAVING running a real + * `RegExp`, with `$options` skipped as its flags. + * + * #4706 retired both spellings and #5710 flipped their last producer, so + * there is no true/false answer left to assert: HAVING is the fifth of the + * five refusal sites `RETIRED_FILTER_OPERATORS` names, and it now refuses. + * The old arm also answered an ILLEGAL pattern with `return false` — "this + * row does not match" for a filter that could not run at all — which is the + * silent wrong answer the retirement is about, on the one evaluation face no + * conformance table drives. + */ + for (const [label, condition, mustMention] of [ + ['a bare $regex', { k: { $regex: '^alp' } }, ["'$regex'", '$icontains']], + ['a bare $options', { k: { $options: 'i' } }, ["'$options'", '$icontains']], + [ + '$regex with $options — one mistake, one fix', + { k: { $regex: '^alp', $options: 'i' } }, + ["'$regex'", "'$options'", '$icontains'], + ], + ] as const) { + it(`REFUSES the retired ${label}, naming the replacement`, () => { + let err: Error | undefined; + try { + matchesHaving({ k: 'Alpha' }, condition); + } catch (e) { + err = e as Error; + } + expect(err, 'expected `having` to refuse a retired operator').toBeInstanceOf(Error); + expect(err!.message).toContain('RETIRED'); + for (const mention of mustMention) expect(err!.message).toContain(mention); + }); + } }); /** diff --git a/packages/objectql/src/having-filter.ts b/packages/objectql/src/having-filter.ts index 9b48372a9d..2e88aa386b 100644 --- a/packages/objectql/src/having-filter.ts +++ b/packages/objectql/src/having-filter.ts @@ -42,16 +42,50 @@ // the ruling. import type { FilterCondition } from '@objectstack/spec/data'; +// [#5702] The retired operators and the prescription a refusal prints. HAVING is +// the fifth of the five refusal sites `RETIRED_FILTER_OPERATORS`' own doc names, +// and reads the table for the same reason the four driver sites do: one +// retirement, one sentence. +import { RETIRED_FILTER_OPERATORS } from '@objectstack/spec/data'; const LOGICAL_OPERATORS = ['$and', '$or', '$not'] as const; +// [#5702] `$regex` is GONE from this vocabulary. Its arm below ran a real +// `RegExp` over the aggregated value and answered an ILLEGAL pattern with +// `return false` — "no rows", silently — which is the pair of defects #4706 +// retired the operator over, on the one evaluation face no conformance table +// covers. `$icontains` is not added in its place: this face would need its own +// ASCII-only fold, which is the same "semantic completion" investment #5499 +// freezes for the other JS faces, so HAVING refuses it fail-closed for now. const CONDITION_OPERATORS = [ '$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$between', '$in', '$nin', '$exists', '$null', - '$contains', '$notContains', '$startsWith', '$endsWith', '$regex', + '$contains', '$notContains', '$startsWith', '$endsWith', ] as const; -function unknownOperator(op: string, where: 'logical' | 'condition'): Error { +function unknownOperator( + op: string, + where: 'logical' | 'condition', + siblings: readonly string[] = [], +): Error { + // [#5702] A RETIRED spelling gets the spec's prescription rather than the + // vocabulary list — its author wrote a name this face ANSWERED until #4706, + // and needs to know what replaces it, not what else exists. Retired siblings + // are named too: `{ $regex, $options }` is one mistake with one fix. + const retired = RETIRED_FILTER_OPERATORS[op]; + if (retired && where === 'condition') { + const replacement = retired.to ? ` Write "${retired.to}" instead.` : ''; + const alsoRetired = siblings.filter((key) => key !== op && RETIRED_FILTER_OPERATORS[key]); + const also = alsoRetired.length + ? ` The same condition also carries the retired ` + + `${alsoRetired.map((key) => `'${key}'`).join(', ')} — one '${retired.to}' replaces the ` + + `whole shape, so this is ONE mistake with ONE fix, not one per key.` + : ''; + return new Error( + `Filter operator '${op}' in \`having\` is RETIRED and is no longer evaluated.${replacement} ` + + `${retired.why}${also}`, + ); + } const supported = where === 'logical' ? `${LOGICAL_OPERATORS.join(', ')} (or a column condition)` : CONDITION_OPERATORS.join(', '); @@ -142,7 +176,12 @@ function checkCondition(value: any, condition: any): boolean { } for (const op of keys) { - if (op === '$options') continue; // consumed by $regex below + // [#5702] The `if (op === '$options') continue` that stood here is GONE. It + // skipped the key because `$regex` consumed it as its flags; with `$regex` + // retired, `$options` is a key nothing consumes, and skipping it would mean + // silently ignoring a constraint the author wrote — the exact widening this + // file refuses unknown operators to avoid. It now falls to `default:` and is + // refused with the spec's prescription. const target = (condition as Record)[op]; if (value === undefined && !NO_VALUE_ANSWERED_BY_OPERATOR.has(op)) return false; switch (op) { @@ -181,18 +220,12 @@ function checkCondition(value: any, condition: any): boolean { case '$notContains': if (typeof value === 'string' && value.includes(target)) return false; break; case '$startsWith': if (typeof value !== 'string' || !value.startsWith(target)) return false; break; case '$endsWith': if (typeof value !== 'string' || !value.endsWith(target)) return false; break; - case '$regex': { - let re: RegExp; - try { - re = new RegExp(target, (condition as Record).$options || ''); - } catch { - return false; - } - if (!re.test(String(value))) return false; - break; - } + // [#5702] The `$regex` arm is GONE. It built `new RegExp(target, $options)` + // and, on an illegal pattern, `return false` — an unrunnable filter + // answered as "this row does not match", i.e. a silent empty result rather + // than an error. Retired by #4706; refused by `default:` below. default: - throw unknownOperator(op, 'condition'); + throw unknownOperator(op, 'condition', keys); } } return true; diff --git a/packages/spec/spec-changes.json b/packages/spec/spec-changes.json index 8af6543845..e856a8b87c 100644 --- a/packages/spec/spec-changes.json +++ b/packages/spec/spec-changes.json @@ -665,6 +665,13 @@ "toMajor": 17, "rationale": "This entry records a TYPE being added, not a surface being withdrawn, and it says so up front because the distinction decides who has to do anything. `distinct` is not declared on `IDataDriver`, so #5181 / #6075 never reached it and it kept `filters?: any` while its body said something far more specific — `applyFilters(builder, filters)` is handed the ARGUMENT ITSELF, never a `.where` off it. ⚠️ RUNTIME BEHAVIOUR IS UNCHANGED by this entry's change: not one statement moved, so no upgrade breaks at run time and nothing that answered correctly stops. What the annotation removes is a compile-time hole, measured rather than assumed: a truthy NON-OBJECT third argument — `distinct('orders', 'product', 'completed')` — used to type-check and resolve the UNFILTERED set, because `applyFilters` emits no predicate at all for a truthy non-object, non-array filter. A call meaning \"which products among completed orders\" answered with EVERY product, silently. That spelling is now TS2345 at the call site. This is a driver CALL ARGUMENT — code, never stack metadata — so there is no source for the D2 chain to rewrite and deliberately no schema tombstone, the disposition `data-driver-find-stream-retired` (#4484), `storage-service-list-retired` (#5540), `actor-user-roles-to-positions` (#6011) and `driver-aggregate-undeclared-key-aliases-removed` (#6321) already carry. ⚠️ It differs from those four in ONE measured way a reader should not have to infer: because nothing changed at run time, an untyped JS caller is not affected BY THE UPGRADE at all. The entry is here for a different reason — such a caller is exactly the one tsc can never reach, and the silent widening above is a defect they may ALREADY be sitting on, before and after this major. The generated upgrade guide is the only channel that reaches them, which is why the fix is written down rather than left to the compiler. ⛔ The reverse mismatch is NOT closed and no type can close it: `FilterCondition` is an open map (`[key: string]: any`) because a filter key IS a field name, so a query envelope `{ object, where }` is structurally a valid filter — one constraining columns named `object` and `where` — and so is a FilterArray. Both reach `distinct` type-checked and are refused at run time, loudly, with INVALID_FILTER / 400. `driver-memory`'s opposite half — where the BARE spelling returns the unfiltered set in silence — stays open under the #5499 freeze (#6320). ADR-0087, #6320." }, + { + "surface": "data.filter $regex / $options — in a STORED filter (dashboard widget filter and globalFilters, report runtimeFilter, page and component filter, solution-blueprint filter), and equally in the where clause of a query request", + "replacement": "$icontains for the case-insensitive substring match this was almost always used for, or $contains for a case-sensitive one — a pattern that genuinely needs a regular expression has no filter-level replacement", + "migrationId": "filter-regex-options-retired", + "toMajor": 17, + "rationale": "Like `driver-aggregate-undeclared-key-aliases-removed` and `driver-sql-distinct-bare-filter-typed`, this entry records a LENIENCY being withdrawn rather than a declared surface: `$regex` was never in `FILTER_OPERATORS` and never a key on `StringOperatorSchema`. That is measured, not assumed — `git log -S'$regex'` over `packages/spec/src` returns only doc comments describing how `$contains` LOWERS to MongoDB (`Contains substring - SQL: LIKE %?% | MongoDB: $regex`), plus #5701 itself, which added the name solely as `RETIRED_FILTER_OPERATORS` prescription data. ⚠️ But it differs from those two in the one way that decides the disposition, so a reader should not have to infer it: those were driver CALL ARGUMENTS, code and never stack metadata, whereas a filter IS stored metadata. `FilterConditionSchema` is an OPEN RECORD (`z.record(z.string(), z.unknown())`) because a filter key is a field name, so a stored `{ name: { $regex: 'acme.*' } }` parses GREEN and always will — a `retiredKey()` tombstone cannot exist on an open map, which is exactly why the ledger has to carry this. What such a stack used to get was four different answers from four backends: `driver-sql` and Turso's remote transport compiled it to a LIKE-escaped SUBSTRING (so `a.b` matched only the literal `a.b` and the regex was silently never a regex), `driver-memory` and objectql's `having` ran it as a real `RegExp` (so the same filter also matched `axb`, and an INVALID pattern was caught and answered `false` — zero rows, in silence), and `driver-mongodb` refused it with a bare `Error` carrying no `code` and no `status`. It is now refused everywhere with INVALID_FILTER / 400 naming the replacement. There is deliberately NO D2 conversion and this sits in `semantic` rather than among the mechanical transforms: rewriting `$regex` to `$icontains` is NOT lossless in either direction — a regex metacharacter becomes a literal — so an auto-applied rewrite would silently change which rows a dashboard, report or permission filter selects, a wrong number rather than a missing one. Choosing the substring the pattern MEANT is a judgment about the query, not a transform. ⚠️ This entry covers BOTH HALVES of the #4706 ruling (B), not just the driver one: the contract half (#5701 — the `$icontains` declaration, the `$contains` family pinned case-sensitive, and the `RETIRED_FILTER_OPERATORS` prescriptions) landed before the ADR-0087 disposition gate (#6148) existed and so was never asked for a ledger entry; the driver half (#5702) is where the refusal became executable. One surface, one entry, registered from the half that made it observable. ADR-0049 / ADR-0087, #4706 / #5701 / #5702." + }, { "surface": "system.serverEvent / system.serverEventType / system.serverCapabilities / system.serverStatus (the lifecycle-event, capability-report and status vocabulary of system/http-server.zod.ts — 4 defs, 8 exported names)", "replacement": "(removed — there is no replacement key, because there was never a key. Server lifecycle is the transport plugin's own start/stop seam; per-request and per-server observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus `OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a transport plugin can DO it states by implementing the kernel plugin contract — the seams it registers are the capability statement, and a self-described capability record can only disagree with them. Server-level configuration that IS authorable lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected)", @@ -1409,6 +1416,13 @@ "toMajor": 17, "rationale": "This entry records a TYPE being added, not a surface being withdrawn, and it says so up front because the distinction decides who has to do anything. `distinct` is not declared on `IDataDriver`, so #5181 / #6075 never reached it and it kept `filters?: any` while its body said something far more specific — `applyFilters(builder, filters)` is handed the ARGUMENT ITSELF, never a `.where` off it. ⚠️ RUNTIME BEHAVIOUR IS UNCHANGED by this entry's change: not one statement moved, so no upgrade breaks at run time and nothing that answered correctly stops. What the annotation removes is a compile-time hole, measured rather than assumed: a truthy NON-OBJECT third argument — `distinct('orders', 'product', 'completed')` — used to type-check and resolve the UNFILTERED set, because `applyFilters` emits no predicate at all for a truthy non-object, non-array filter. A call meaning \"which products among completed orders\" answered with EVERY product, silently. That spelling is now TS2345 at the call site. This is a driver CALL ARGUMENT — code, never stack metadata — so there is no source for the D2 chain to rewrite and deliberately no schema tombstone, the disposition `data-driver-find-stream-retired` (#4484), `storage-service-list-retired` (#5540), `actor-user-roles-to-positions` (#6011) and `driver-aggregate-undeclared-key-aliases-removed` (#6321) already carry. ⚠️ It differs from those four in ONE measured way a reader should not have to infer: because nothing changed at run time, an untyped JS caller is not affected BY THE UPGRADE at all. The entry is here for a different reason — such a caller is exactly the one tsc can never reach, and the silent widening above is a defect they may ALREADY be sitting on, before and after this major. The generated upgrade guide is the only channel that reaches them, which is why the fix is written down rather than left to the compiler. ⛔ The reverse mismatch is NOT closed and no type can close it: `FilterCondition` is an open map (`[key: string]: any`) because a filter key IS a field name, so a query envelope `{ object, where }` is structurally a valid filter — one constraining columns named `object` and `where` — and so is a FilterArray. Both reach `distinct` type-checked and are refused at run time, loudly, with INVALID_FILTER / 400. `driver-memory`'s opposite half — where the BARE spelling returns the unfiltered set in silence — stays open under the #5499 freeze (#6320). ADR-0087, #6320." }, + { + "surface": "data.filter $regex / $options — in a STORED filter (dashboard widget filter and globalFilters, report runtimeFilter, page and component filter, solution-blueprint filter), and equally in the where clause of a query request", + "replacement": "$icontains for the case-insensitive substring match this was almost always used for, or $contains for a case-sensitive one — a pattern that genuinely needs a regular expression has no filter-level replacement", + "migrationId": "filter-regex-options-retired", + "toMajor": 17, + "rationale": "Like `driver-aggregate-undeclared-key-aliases-removed` and `driver-sql-distinct-bare-filter-typed`, this entry records a LENIENCY being withdrawn rather than a declared surface: `$regex` was never in `FILTER_OPERATORS` and never a key on `StringOperatorSchema`. That is measured, not assumed — `git log -S'$regex'` over `packages/spec/src` returns only doc comments describing how `$contains` LOWERS to MongoDB (`Contains substring - SQL: LIKE %?% | MongoDB: $regex`), plus #5701 itself, which added the name solely as `RETIRED_FILTER_OPERATORS` prescription data. ⚠️ But it differs from those two in the one way that decides the disposition, so a reader should not have to infer it: those were driver CALL ARGUMENTS, code and never stack metadata, whereas a filter IS stored metadata. `FilterConditionSchema` is an OPEN RECORD (`z.record(z.string(), z.unknown())`) because a filter key is a field name, so a stored `{ name: { $regex: 'acme.*' } }` parses GREEN and always will — a `retiredKey()` tombstone cannot exist on an open map, which is exactly why the ledger has to carry this. What such a stack used to get was four different answers from four backends: `driver-sql` and Turso's remote transport compiled it to a LIKE-escaped SUBSTRING (so `a.b` matched only the literal `a.b` and the regex was silently never a regex), `driver-memory` and objectql's `having` ran it as a real `RegExp` (so the same filter also matched `axb`, and an INVALID pattern was caught and answered `false` — zero rows, in silence), and `driver-mongodb` refused it with a bare `Error` carrying no `code` and no `status`. It is now refused everywhere with INVALID_FILTER / 400 naming the replacement. There is deliberately NO D2 conversion and this sits in `semantic` rather than among the mechanical transforms: rewriting `$regex` to `$icontains` is NOT lossless in either direction — a regex metacharacter becomes a literal — so an auto-applied rewrite would silently change which rows a dashboard, report or permission filter selects, a wrong number rather than a missing one. Choosing the substring the pattern MEANT is a judgment about the query, not a transform. ⚠️ This entry covers BOTH HALVES of the #4706 ruling (B), not just the driver one: the contract half (#5701 — the `$icontains` declaration, the `$contains` family pinned case-sensitive, and the `RETIRED_FILTER_OPERATORS` prescriptions) landed before the ADR-0087 disposition gate (#6148) existed and so was never asked for a ledger entry; the driver half (#5702) is where the refusal became executable. One surface, one entry, registered from the half that made it observable. ADR-0049 / ADR-0087, #4706 / #5701 / #5702." + }, { "surface": "system.serverEvent / system.serverEventType / system.serverCapabilities / system.serverStatus (the lifecycle-event, capability-report and status vocabulary of system/http-server.zod.ts — 4 defs, 8 exported names)", "replacement": "(removed — there is no replacement key, because there was never a key. Server lifecycle is the transport plugin's own start/stop seam; per-request and per-server observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus `OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a transport plugin can DO it states by implementing the kernel plugin contract — the seams it registers are the capability statement, and a self-described capability record can only disagree with them. Server-level configuration that IS authorable lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected)", diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 3eb69d5972..882687ca4a 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -2453,6 +2453,67 @@ const step17: MigrationStep = { + 'envelope or a FilterArray in that slot still compiles and is rejected at run time ' + 'with INVALID_FILTER / 400.', }, + { + id: 'filter-regex-options-retired', + // No backticks in `surface` — see the note two entries above. + surface: + 'data.filter $regex / $options — in a STORED filter (dashboard widget filter and ' + + 'globalFilters, report runtimeFilter, page and component filter, solution-blueprint ' + + 'filter), and equally in the where clause of a query request', + replacement: + '$icontains for the case-insensitive substring match this was almost always used ' + + 'for, or $contains for a case-sensitive one — a pattern that genuinely needs a ' + + 'regular expression has no filter-level replacement', + reason: + 'Like `driver-aggregate-undeclared-key-aliases-removed` and ' + + '`driver-sql-distinct-bare-filter-typed`, this entry records a LENIENCY being ' + + 'withdrawn rather than a declared surface: `$regex` was never in `FILTER_OPERATORS` ' + + 'and never a key on `StringOperatorSchema`. That is measured, not assumed — `git ' + + 'log -S\'$regex\'` over `packages/spec/src` returns only doc comments describing how ' + + '`$contains` LOWERS to MongoDB (`Contains substring - SQL: LIKE %?% | MongoDB: ' + + '$regex`), plus #5701 itself, which added the name solely as `RETIRED_FILTER_OPERATORS` ' + + 'prescription data. ⚠️ But it differs from those two in the one way that decides the ' + + 'disposition, so a reader should not have to infer it: those were driver CALL ' + + 'ARGUMENTS, code and never stack metadata, whereas a filter IS stored metadata. ' + + '`FilterConditionSchema` is an OPEN RECORD (`z.record(z.string(), z.unknown())`) ' + + 'because a filter key is a field name, so a stored `{ name: { $regex: \'acme.*\' } }` ' + + 'parses GREEN and always will — a `retiredKey()` tombstone cannot exist on an open ' + + 'map, which is exactly why the ledger has to carry this. What such a stack used to ' + + 'get was four different answers from four backends: `driver-sql` and Turso\'s remote ' + + 'transport compiled it to a LIKE-escaped SUBSTRING (so `a.b` matched only the literal ' + + '`a.b` and the regex was silently never a regex), `driver-memory` and objectql\'s ' + + '`having` ran it as a real `RegExp` (so the same filter also matched `axb`, and an ' + + 'INVALID pattern was caught and answered `false` — zero rows, in silence), and ' + + '`driver-mongodb` refused it with a bare `Error` carrying no `code` and no `status`. ' + + 'It is now refused everywhere with INVALID_FILTER / 400 naming the replacement. ' + + 'There is deliberately NO D2 conversion and this sits in `semantic` rather than among ' + + 'the mechanical transforms: rewriting `$regex` to `$icontains` is NOT lossless in ' + + 'either direction — a regex metacharacter becomes a literal — so an auto-applied ' + + 'rewrite would silently change which rows a dashboard, report or permission filter ' + + 'selects, a wrong number rather than a missing one. Choosing the substring the ' + + 'pattern MEANT is a judgment about the query, not a transform. ⚠️ This entry covers ' + + 'BOTH HALVES of the #4706 ruling (B), not just the driver one: the contract half ' + + '(#5701 — the `$icontains` declaration, the `$contains` family pinned ' + + 'case-sensitive, and the `RETIRED_FILTER_OPERATORS` prescriptions) landed before the ' + + 'ADR-0087 disposition gate (#6148) existed and so was never asked for a ledger entry; ' + + 'the driver half (#5702) is where the refusal became executable. One surface, one ' + + 'entry, registered from the half that made it observable. ADR-0049 / ADR-0087, ' + + '#4706 / #5701 / #5702.', + acceptanceCriteria: + 'No stored filter and no request `where` spells `$regex` or `$options` — grep the ' + + 'stack for both. Each one is rewritten by asking what the pattern MEANT, not by ' + + 'transliterating it: a bare substring pattern becomes `$icontains` (or `$contains` ' + + 'when the match must stay case-sensitive), and its metacharacters are dropped rather ' + + 'than escaped, because they were never honoured as a regex on the SQL family in the ' + + 'first place. ⚠️ Expect the answer to CHANGE on any stack that ran on ' + + '`driver-memory`, `driver-mongodb` or objectql `having`, where the pattern really was ' + + 'evaluated as a regular expression; on the SQL family the rewritten filter returns ' + + 'what it always returned. A pattern that genuinely needs alternation, anchoring or ' + + 'character classes has no filter-level replacement — move that predicate into a ' + + 'formula field or a server-side view, or open an issue for it. Verify by loading the ' + + 'stack: a surviving `$regex` or `$options` is answered INVALID_FILTER / 400 with a ' + + 'message naming the replacement, on every backend.', + }, { id: 'http-server-runtime-vocabulary-retired', surface: diff --git a/scripts/check-driver-conformance.mjs b/scripts/check-driver-conformance.mjs index d2fe53b356..643674407a 100644 --- a/scripts/check-driver-conformance.mjs +++ b/scripts/check-driver-conformance.mjs @@ -279,33 +279,43 @@ const CASE_SETS = [ // fact instead of an assumption. They are cleared by #5702, one suite at a // time, exactly the way the five before them were. // -// What the case-set demands, and why nobody can answer it yet: -// -// 1. `$icontains` — a NEW operator (ASCII-only case fold). No backend has an -// arm for it. All five refuse it today, which is the fail-closed -// direction, so this is a missing capability rather than a live defect. +// What the case-set demands, and where each requirement stands: +// +// 1. `$icontains` — a NEW operator (ASCII-only case fold). **DONE on the SQL +// family** (#5702): driver-sql compiles `LOWER(col) LIKE LOWER(?) ESCAPE ?` +// through the same `applyLike` that carries the `%`/`_`/`\` class, turso's +// remote transport carries the twin in `pushLike`, and sqlite-wasm inherits +// and executes it on sql.js. Still REFUSED (fail-closed, an unimplemented +// capability rather than a live defect) on driver-memory and +// driver-mongodb, which are the #5499 frozen family — tracked as #6520, +// which also explains why the spec's `FILTER_OPERATORS` cannot take +// `$icontains` until those two have arms. // 2. `$contains` / `$startsWith` / `$endsWith` / `$notContains` must be // case-SENSITIVE (#4706 Q2 = A, superseding `filter.zod.ts`'s former -// "Case sensitivity should be handled at backend level"). NO driver -// delivers this on its live query path today: driver-memory and +// "Case sensitivity should be handled at backend level"). **STILL OPEN on +// every driver**, and it is the sole reason these five rows survive #5702. +// No driver delivers it on its live query path: driver-memory and // driver-mongodb fold the full Unicode range, and the SQL family follows // its dialect (SQLite — so also turso and sqlite-wasm — folds ASCII; // Postgres happens to be case-exact already; MySQL depends on collation). // The one surface that does compare case-sensitively is driver-memory's // REFERENCE matcher, which is not the path a query takes — see that row. -// 3. `$regex` / `$options` must be REFUSED, naming `$icontains`. Exactly one -// driver refuses `$regex` today (driver-mongodb, via its `default:` arm — -// though outside the ADR-0112 envelope the case-set requires; see its -// row). The other four accept it: driver-memory evaluates it as a real -// RegExp, and driver-sql / driver-sqlite-wasm / driver-turso compile it to -// a substring LIKE. That is deliberate, not neglect — `plugin-auth`'s -// ObjectQL adapter still emits it on the AUTHENTICATION path. **#5710 -// flips that producer before any of these four cells may be cleared**; a -// driver that refuses `$regex` first breaks sign-in. +// Tracked as #6518, filed separately rather than folded into #5702 because +// the lowering exists in THREE places that must move together — the two +// driver compilers and the RLS/analytics twins (`read-scope-sql`, +// `service-analytics`'s `like-pattern.ts`) — and a driver-only change +// would compile one permission rule into two row sets (#3948). +// 3. `$regex` / `$options` must be REFUSED, naming `$icontains`. **DONE on all +// five** (#5702), which was blocked until #5710 flipped the last live +// producer (`plugin-auth`'s ObjectQL adapter, on the AUTHENTICATION path). +// Each site now prints `RETIRED_FILTER_OPERATORS[op].why` verbatim, so the +// five refusals say one thing; driver-mongodb's `default:` arm was +// additionally routed through its own `INVALID_FILTER` helper, which is the +// `code` half the case-set requires and the last place a bare `new Error` +// escaped the ADR-0112 envelope. // // Each row's `why` is what that driver does TODAY, measured on this branch by -// reading the compiler and (for driver-memory) executing it. Nothing here is -// predicted. +// reading the compiler and executing it. Nothing here is predicted. const LEDGER = [ { @@ -313,7 +323,11 @@ const LEDGER = [ marker: 'FILTER_TEXT_CASES', kind: 'DEBT', why: - 'Measured, and the one row where "which face" changes the answer — do NOT take a single reading here. ' + 'Re-measured after #5702. Requirement 3 is DONE: `$regex`/`$options` are no longer in ' + + '`SUPPORTED_FIELD_OPERATORS`, the matcher\'s `$regex` arm (the only live regex evaluator in the ' + + 'repo, and the one that answered an ILLEGAL pattern with `false`) is deleted, and both faces refuse ' + + 'them with the spec prescription naming `$icontains`. What is left is requirement 2, and this is ' + + 'still the one row where "which face" changes the answer — do NOT take a single reading here. ' + 'The QUERY path (`find()` -> `normalizeFieldOperators`, and the analytics face via ' + '`filterSubstringPattern`) lowers `$contains` to `new RegExp(escapeRegex(v), "i")`: literal comparand ' + '(requirement 2\'s escaping half holds) but case-INSENSITIVE over the whole Unicode range, which ' @@ -321,68 +335,83 @@ const LEDGER = [ + '(`memory-matcher.ts` `match()`, the record-at-a-time evaluator `filter-logic-conformance.ts` counts ' + 'as a backend) uses String.prototype.includes and is case-SENSITIVE — i.e. this package answers one ' + '`$contains` two ways today, the divergence class #5374 fixed between the other two faces. Whichever ' - + 'suite clears this cell has to pick one and align both. `$icontains` is refused on both faces ' - + "(`SUPPORTED_FIELD_OPERATORS` derives from the spec's FILTER_OPERATORS, which deliberately does not " - + 'carry it yet) — unimplemented but fail-closed. `$regex`/`$options` are ACCEPTED and evaluated as a ' - + 'real RegExp, the opposite of requirement 3 and the only live regex evaluator in the repo; that arm ' - + "exists for plugin-auth's adapter and cannot be removed before #5710.", - issue: 'https://github.com/objectstack-ai/objectstack/issues/5702', + + 'suite clears this cell has to pick one and align both (#6518). `$icontains` is still refused on both ' + + "faces (`SUPPORTED_FIELD_OPERATORS` derives from the spec's FILTER_OPERATORS, which deliberately does " + + 'not carry it yet) — unimplemented but fail-closed, requirement 1 open here and tracked as #6520; ' + + 'this package is in the #5499 frozen family and #5702 left that half suspended by design.', + issue: 'https://github.com/objectstack-ai/objectstack/issues/6518', }, { driver: 'driver-sql', marker: 'FILTER_TEXT_CASES', kind: 'DEBT', why: - 'Measured: `applyLike` escapes `\\`, `%` and `_` and binds ESCAPE, so the literal-comparand cases would ' - + 'pass today. Case sensitivity is the DIALECT\'s, not the driver\'s — SQLite\'s LIKE folds ASCII, ' - + 'Postgres does not, MySQL follows its collation — so requirement 2 fails on two of three dialects and ' - + 'needs a case-exact comparison (GLOB / instr() / a binary collation), not a flag. `$icontains` hits the ' - + '`default:` arm and is refused in the ADR-0112 envelope. `$regex` is COMPILED (to the same substring ' - + 'LIKE), not refused.', - issue: 'https://github.com/objectstack-ai/objectstack/issues/5702', + 'Re-measured after #5702, which cleared two of the three requirements here. Requirement 3: `$regex` no ' + + 'longer has a `case` arm (it was a fallthrough onto `$contains`) and both retired spellings are ' + + 'refused with the spec prescription. Requirement 1: `$icontains` compiles to ' + + '`LOWER(col) LIKE LOWER(?) ESCAPE ?` through `applyLike`\'s `fold` parameter, so it shares the ' + + '`%`/`_`/`\\` class character-for-character (executed: `%`, `_`, `.` and `\\` all literal), and an ' + + 'empty or non-string comparand is refused on the VALIDATING walk beside `$null`/`$exists`. What is ' + + 'left is requirement 2: case sensitivity is the DIALECT\'s, not the driver\'s — SQLite\'s LIKE folds ' + + 'ASCII, Postgres does not, MySQL follows its collation — so `$contains` fails on two of three dialects ' + + 'and needs a case-exact comparison (GLOB / instr() / a binary collation), not a flag. Tracked as ' + + '#6518, which also carries the mirror defect the same axis produces on `$icontains`: `LOWER()` folds ' + + 'the whole Unicode range on Postgres/MySQL, so the ASCII-only boundary holds on SQLite (measured) and ' + + 'over-folds there. NOTE the consequence for reading this cell: on SQLite `LIKE` already folds ASCII, ' + + 'so `$contains` and `$icontains` return IDENTICAL rows for every comparand until #6518 lands — the ' + + 'fold is pinned by the compiled SQL, which is the only witness this dialect can give.', + issue: 'https://github.com/objectstack-ai/objectstack/issues/6518', }, { driver: 'driver-sqlite-wasm', marker: 'FILTER_TEXT_CASES', kind: 'DEBT', why: - 'Measured: `SqliteWasmDriver extends SqlDriver`, so every fact in the driver-sql row applies unchanged, ' - + 'with the dialect pinned to SQLite — i.e. requirement 2 fails here specifically because LIKE folds ' - + 'ASCII case. Tracked as DEBT rather than EXEMPT for the reason its FILTER_LOGIC row was: "inherits, ' - + 'therefore fine" is the assumption these suites exist to disprove, and what this one would add is the ' - + 'sql.js engine EXECUTING the compiled predicate, which is where a collation choice actually shows up.', - issue: 'https://github.com/objectstack-ai/objectstack/issues/5702', + 'Re-measured after #5702: `SqliteWasmDriver extends SqlDriver`, so every fact in the driver-sql row ' + + 'applies unchanged, with the dialect pinned to SQLite — requirements 1 and 3 are DONE and inherited, ' + + 'requirement 2 fails here specifically because LIKE folds ASCII case (#6518). The inheritance is not ' + + 'taken on faith: `sqlite-wasm-icontains-and-retired-operators.test.ts` executes the new predicate on ' + + 'sql.js, because `$icontains` is the first operator this package runs whose compiled form is a ' + + 'FUNCTION CALL on the column with a third bound argument, and a wasm dialect that mis-binds those ' + + 'three positions would fail in no other suite in the repo. Tracked as DEBT rather than EXEMPT for the ' + + 'reason its FILTER_LOGIC row was: "inherits, therefore fine" is the assumption these suites exist to ' + + 'disprove, and what a full run of this case-set would still add is the collation half.', + issue: 'https://github.com/objectstack-ai/objectstack/issues/6518', }, { driver: 'driver-turso', marker: 'FILTER_TEXT_CASES', kind: 'DEBT', why: - 'Measured: DUAL-TRANSPORT, so this cell needs TWO suites like its three predecessors. Local/replica ' - + 'inherits SqlDriver (see the driver-sql row) on the SQLite dialect. Remote does not go through knex at ' - + 'all: `remote-transport.ts` carries its own hand-written SUPPORTED_FILTER_OPERATORS — which lists ' - + '`$regex` and not `$icontains` — and its own LIKE assembly. Both transports therefore fail requirement ' - + '2 (SQLite LIKE folds ASCII) and requirement 3, and refuse `$icontains` today.', - issue: 'https://github.com/objectstack-ai/objectstack/issues/5702', + 'Re-measured after #5702. DUAL-TRANSPORT, so this cell needs TWO suites like its three predecessors, and ' + + 'both faces moved: local/replica inherits SqlDriver (see the driver-sql row) on the SQLite dialect, ' + + 'while remote does not go through knex at all — `remote-transport.ts` carries its own hand-written ' + + 'SUPPORTED_FILTER_OPERATORS and its own LIKE assembly, so the work had to be written twice. That list ' + + 'now carries `$icontains` and no longer carries `$regex`; `pushLike` grew the same `fold` parameter as ' + + '`applyLike` so the escape rule cannot fork between the two operators (executed against libSQL-shaped ' + + 'SQLite: `%`, `_` and `\\` literal under `$icontains`); a node-position `$regex` moved from the ' + + '"misplaced field operator" tail to the "declared at no level" one. Requirement 2 is what is left on ' + + 'both transports (SQLite LIKE folds ASCII) — #6518.', + issue: 'https://github.com/objectstack-ai/objectstack/issues/6518', }, { driver: 'driver-mongodb', marker: 'FILTER_TEXT_CASES', kind: 'DEBT', why: - 'Measured: the FURTHEST from the ruling. `translateFieldOperators` lowers `$contains`/`$startsWith`/' - + '`$endsWith`/`$notContains` to `$regex` with a HARDCODED `$options: "i"`, i.e. case-insensitive over ' - + 'the whole Unicode range — requirement 2 inverted, and requirement 1\'s ASCII-only boundary violated in ' - + 'the same expression. `escapeRegex` does escape metacharacters, so the literal-comparand cases hold. ' - + 'An incoming `$regex` reaches the `default:` arm and IS refused (mongo is the only backend that ' - + 'already satisfies requirement 3), and `$icontains` is refused there too — but that arm throws a bare ' - + '`new Error("[mongodb] unsupported filter operator ...")`, NOT the ADR-0112 envelope its own ' - + '`unsupportedFilterError` helper (same file, used by three other refusals here) produces. The ' - + "case-set requires `code: 'INVALID_FILTER'`, so clearing this cell means routing that arm through the " - + 'helper as well. Note this package is in the ' - + '#5499 frozen family: its real-mongod suites are opt-in, so whatever clears this cell needs a ' - + 'server-free half like `mongodb-filter-logic-translation.test.ts` has.', - issue: 'https://github.com/objectstack-ai/objectstack/issues/5702', + 'Re-measured after #5702: still the FURTHEST from the ruling, but the ENVELOPE half is now closed. That ' + + "arm used to throw a bare `new Error('[mongodb] unsupported filter operator …')` — no `code`, no " + + '`status` — three lines from this file\'s own `unsupportedFilterError` helper, which sets ' + + "`INVALID_FILTER` / 400 and which three other refusals here already used. It now routes through the " + + 'helper, and a RETIRED spelling additionally gets the spec prescription naming `$icontains`, so ' + + 'requirement 3 is DONE (mongo was already the only backend REFUSING `$regex`; what was missing was the ' + + 'shape of the refusal). Requirement 2 is inverted here and requirement 1\'s ASCII boundary violated in ' + + 'the same expression: `translateFieldOperators` lowers `$contains`/`$startsWith`/`$endsWith`/' + + '`$notContains` to `$regex` with a HARDCODED `$options: "i"` (#6518). `escapeRegex` does escape ' + + 'metacharacters, so the literal-comparand cases hold. `$icontains` is still refused (#6520). Note this ' + + 'package is in the #5499 frozen family: its real-mongod suites are opt-in, so whatever clears this ' + + 'cell needs a server-free half like `mongodb-filter-logic-translation.test.ts` has.', + issue: 'https://github.com/objectstack-ai/objectstack/issues/6518', }, ];