Extend relation filters: JSON paths and single-row matching ($elemMatch)

[rob-ghost]

Extending relation filters: JSON paths and single-row matching

Summary

Two additions to how mongo-knex filters a record by its related rows: reaching into a JSON value on a related row, and requiring several conditions to be satisfied by one and the same related row ($elemMatch). Both are opt-in and additive — existing filters are unchanged.

Motivation

NQL already lets a consumer filter a record by its relations — posts by their tags, members by their labels. Building "filter members by their custom fields" surfaced two things the relation grammar could not express. Custom fields are stored flexibly: a member has many key/value rows, and some values are structured (an address kept as a single JSON value).

Reaching inside a JSON value. A related row may hold structured data in one JSON column. A filter could match that column as a whole, but not a field inside it — you could not ask for "the address whose country is GB."

Requiring conditions to hold on the same related row. When a record has many rows for a relation, a compound condition is ambiguous. "field is company and value is not Ghost" can mean either one row (a company row whose value isn't Ghost) or across rows (has a company row, and separately has no row valued Ghost anywhere). NQL historically chose the second reading — the right default for tags, where "has tag A but not tag B" is genuinely about different rows. But for a key/value field the two halves describe a single row, and there was no way to say so. Crucially, "same row" is a property of the question being asked, not of the relation — so it cannot be inferred reliably from how the relation is configured.

What this adds

JSON path extraction. A dotted key can reach a value nested inside a JSON column of a related row — e.g. a member's address, stored as JSON, filtered by its country. A nested value then compares and matches like any ordinary column.

$elemMatch. An explicit operator meaning "a single related row satisfies all of these conditions at once." For example:

{ custom_fields: { $elemMatch: { key: 'company', value: { $ne: 'Ghost' } } } }

reads as "has a company field whose value is not Ghost" — both conditions bound to one row. Anything not wrapped in $elemMatch keeps the existing per-condition behavior.

Benefits

• Same-row intent is stated explicitly, in the query, rather than guessed from the relation's configuration.
• Additive and safe: existing filters emit exactly the same queries; only filters that opt into $elemMatch get the new grouping.
• General: $elemMatch works for any relation, not just the feature that prompted it.
• Correct and portable: JSON values compare consistently across the databases we support.

Tradeoffs and non-goals

• $elemMatch is opt-in by design. Changing the default relation behavior would silently alter the meaning of existing filters, so the default is left exactly as it was.
• JSON support is scalar path extraction for comparison, not a full JSON query language.
• This adds no new NQL surface syntax; these are query-object capabilities a caller (or a small translation layer) produces from whatever grammar it exposes to users.

Prior art

• MongoDB $elemMatch — same name and same meaning (a single array element matching all conditions). NQL already models filters as Mongo-style query objects, so this fits the existing vocabulary rather than inventing a concept.
• SQL JSON accessors — extraction relies on the standard unquoting accessor that both supported databases implement.

Backwards compatibility

Fully backwards compatible. Both capabilities are opt-in and additive; every existing filter produces identical output. Verified against both supported databases, including execution (not just generated SQL), so the JSON behavior is proven on each.

Draft: the consumer is Ghost's members custom-fields filtering; kept open so a pre-release artefact can be published for the Ghost PR to build against.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 786f120f-1270-4144-9f46-648f30e9593c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/mongo-knex-json-path-extraction

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.97207% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.26%. Comparing base (95f3927) to head (8da3e10).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
packages/mongo-knex/lib/convertor.js	94.97%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #192      +/-   ##
==========================================
+ Coverage   84.18%   85.26%   +1.07%     
==========================================
  Files           9        9              
  Lines        2074     2239     +165     
  Branches      428      470      +42     
==========================================
+ Hits         1746     1909     +163     
- Misses        322      324       +2     
  Partials        6        6              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

packages/mongo-knex/test/integration/relations.test.js (1)

1103-1127: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Consider adding cross-dialect coverage for numeric JSON-path comparisons.

These tests only exercise equality and regex (string) matching on extracted JSON scalars. SQLite's ->> returns a native INTEGER/REAL for a numeric JSON value, while MySQL's ->> always returns TEXT — the two dialects genuinely differ in the SQL type produced by the extraction, even though the comment at Line 1098-1102 emphasizes that only real query execution proves cross-dialect correctness. A numeric comparison ($gt/$lt) test, like the unit test at packages/mongo-knex/test/unit/convertor.test.js Line 843-846, run against both databases here would confirm type coercion behaves correctly on both engines and not just on the one currently exercised by string comparisons.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/mongo-knex/test/integration/relations.test.js` around lines 1103 -
1127, Add integration coverage in the JSON path extraction describe block for
numeric comparisons against a JSON subfield, using a numeric $gt or $lt query
and expected matching IDs. Ensure the test executes through the existing
makeQuery(...).select() flow against both supported database dialects, following
the numeric comparison pattern from the convertor tests.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/mongo-knex/lib/convertor.js`:
- Around line 195-238: The JSON-path null handling in applyComparison must use
the null-aware whereType (for example, matching the Null suffix) rather than
value === null or the operator, so relation-level array-ification still emits IS
NULL/IS NOT NULL instead of IN/NOT IN. In packages/mongo-knex/lib/convertor.js
lines 195-238, update applyComparison while preserving the existing raw SQL and
bindings; in packages/mongo-knex/test/unit/convertor.test.js lines 827-900, add
coverage asserting the generated SQL for a relation JSON-path {$ne: null} or
plain null comparison.

In `@packages/mongo-knex/test/unit/convertor.test.js`:
- Around line 827-900: The JSON-path relation regression suite lacks coverage
for null negation. Add a test in the “JSON path in relations” describe block
using runQuery with posts_meta.meta_json.country and {$ne: null}, asserting the
generated relation subquery uses IS NOT NULL rather than NOT IN (NULL).

---

Nitpick comments:
In `@packages/mongo-knex/test/integration/relations.test.js`:
- Around line 1103-1127: Add integration coverage in the JSON path extraction
describe block for numeric comparisons against a JSON subfield, using a numeric
$gt or $lt query and expected matching IDs. Ensure the test executes through the
existing makeQuery(...).select() flow against both supported database dialects,
following the numeric comparison pattern from the convertor tests.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a87aa15f-ee19-49a7-8d9a-6eaf7ef01bd1

📥 Commits

Reviewing files that changed from the base of the PR and between d0d7dc3 and 9f1d71e.

📒 Files selected for processing (5)

• packages/mongo-knex/lib/convertor.js
• packages/mongo-knex/test/integration/relations.test.js
• packages/mongo-knex/test/integration/suite1/fixtures/base.json
• packages/mongo-knex/test/integration/suite1/schema.js
• packages/mongo-knex/test/unit/convertor.test.js