fix(tesseract): render FILTER_PARAMS on the measure side of a join back - #11814
waralexrom wants to merge 10 commits into
Conversation
|
Claude finished @waralexrom's task in 3m 49s —— View job Review: sound, narrowly scoped fix — no blocking issues (1 medium, 3 low)Full review
VerdictThe diagnosis and the fix both hold up. Separating "the filters bindings resolve against" from "the WHERE clause" is the right seam: I checked the two claims the "cannot change results" argument rests on, since both are load-bearing:
Also verified: The I could not run Findings
Test coverageGood shape overall: two positive cases (one per Two gaps, neither blocking:
Minor: |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #11814 +/- ##
===========================================
+ Coverage 61.23% 80.52% +19.28%
===========================================
Files 247 508 +261
Lines 19824 107359 +87535
Branches 4043 4043
===========================================
+ Hits 12140 86454 +74314
- Misses 7112 20333 +13221
Partials 572 572
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
fc89e89 to
6750d5a
Compare
|
Claude finished @waralexrom's task in 5m 43s —— View job No new findings — the fix is sound; 3 prior threads re-affirmed, 0 new inline commentsReview details
Verification of the mechanism
The "cannot reference a cube that isn't joined" claim holds: Both sides resolve the same filter set:
No logical-plan pretty-print snapshots exist, so the new TestsThe unit tests pin the actual defect (both fact copies, both sources of the join back) and Not covered: a Not run hereNo Cargo cache on this runner, so the Rust suite was not executed — the analysis above is from reading. The Review threads3 of my own threads returned by the listing; 0 resolved (all three still live against the current code), 3 re-affirmed:
Both findings I had this round duplicate the first two threads, so nothing was posted inline. |
|
Claude finished @waralexrom's task in 6m 25s —— View job Approach is sound; 4 findings posted inline — 0 high, 1 medium, 3 lowReview details
VerdictThe diagnosis is right and the mechanism is the minimal one. The issue's reproduction is the Tests match the conventions of the neighbouring I could not run Findings
Review threadsThe listing returned 3 of my own threads (single page, no human replies). All 3 resolved as addressed by the current code — the |
A measure that needs the full-key plan is aggregated over a keys subquery joined back to a second copy of the fact source by primary key. The keys-side copy renders the cube's FILTER_PARAMS bindings as the query's predicates; the measure-side copy renders them as always-true, so the database builds the join against the whole unfiltered fact table. Both sources of the join back are covered: a bare cube, and the measure subquery a measure reaching another cube is aggregated over. Each test pins both copies, since the pushdown is only result-neutral while the two render the same predicate. A plain count, rewritten to a distinct count over a single filtered copy, anchors them. The Postgres tests state the equivalence in numbers: the same model with and without the bindings answers the same, so restricting the measure side can only shrink the build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A multiplied measure is aggregated over a keys subquery joined back to a second copy of the fact source by primary key. The keys subquery carries the query's filters, so the cube's FILTER_PARAMS bindings render as real predicates there. The measure side was built with no filter context at all, so every binding collapsed to always-true and the join was built against the whole unfiltered fact table - all tenants, all time. Results stayed correct, since the keys side restricts the output, but the hash build could outgrow the database's memory. Both copies read the same fact rows over the same columns and the join back is by primary key, so any row it can join already satisfies the predicates: rendering them on the measure side can only shrink the build. A select now takes the filters its sources' FILTER_PARAMS and FILTER_GROUP bindings resolve against separately from its own WHERE clause, since these selects carry no WHERE. Both sources of the join back are covered - a bare cube and a measure subquery - and both read the keys subquery's own filter, so the two copies cannot drift apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… key Review follow-ups on the join-back pushdown tests. Parameter bindings are now read off the rendered copies instead of being sliced at fixed offsets, so the assertion states "both copies are bound to the same values" rather than pinning the whole statement's parameter layout. Three cases added: - a FILTER_GROUP binding, which reaches the measure side through the filter subtree search rather than one binding at a time; - an OR filter spanning two cubes, where the subtree search keeps nothing and both copies must fall back to always-true together; - a primary key whose own sql carries a binding, whose two renderings meet in the join-back ON clause and have to be the same expression. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-ups on the join-back pushdown. The context the measure side of the join-back ON clause renders under was still built without filters, so a FILTER_PARAMS binding inside the primary key's own sql resolved on the keys side and fell back to always-true here - the two renderings of the join key stopped being the same expression. Both now read the one filter set the enclosing node already resolved. A debug assertion pins what the new setter means: it replaces the WHERE filter for binding resolution rather than adding to it, which is only right while the selects that use it carry no WHERE. The rationale for that now sits at the field it belongs to instead of being restated at each call site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c1d25ba to
be7a22c
Compare
|
Claude finished @waralexrom's task in 5m 20s —— View job Approach is sound and results-neutral for the fact copies — 1 new low finding posted, 3 prior threads re-affirmed, 0 resolved. Full review
VerdictThe fix matches the symptom in #11740: Tests are well-targeted: both branches of New finding
Re-affirmed from earlier rounds (not re-posted)
Review threads4 unresolved threads returned, all mine; 0 resolved (all four concerns still live in the working tree), 3 re-affirmed as duplicates of the findings above. Not run
|
The filters an enclosing construct applies on a select's behalf replaced the select's own WHERE filter for binding resolution, so a caller setting both would silently stop resolving bindings against its own predicates - the same failure mode as a binding falling back to always-true. Both are conjunctions of what constrains the rows the select emits, so a binding may push either into a source's scan. Conjoining them removes the choice, and with it the invariant a caller could get wrong. No behaviour change: the two callers that pass these filters carry no WHERE of their own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 6m 8s —— View job Approach is sound — 0 blocking, 2 medium, 3 prior threads still open. The fix targets exactly the gap #11740 reports, and the keys side is confirmed to render the identical filter set. Full review
VerdictThe diagnosis matches the issue. Checks that came back clean:
Tests are good: I could not run Findings posted
Review threadsThe listing returned 5 threads first-authored by
|
The measure subquery held a copy of the enclosing keys subquery's filter so that the physical builder could resolve its sources' bindings against it. The filter is not part of the plan, though: the tree is the same with and without it, and only rendering reads it. Holding it on the node also left the two sources of the join back disagreeing about where the filter comes from - one read it live from the keys subquery, the other a snapshot taken while planning. It now travels in the push-down build context, alongside the time shifts and render flags already carried there, and the node that was set for takes it so it is not visible further down. The logical plan keeps no copy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 8m 32s —— View job Fix is correctly targeted at the reported symptom — 0 high, 1 medium, 3 low, 4 posted inline. Review details
VerdictThe diagnosis in the issue is right and the fix matches it. The superset argument holds for the reason stated — a binding renders the column text written at the binding site, never the member's own SQL, so a query filter on a cube that is absent from the measure side simply matches no binding and cannot emit a dangling reference. The Not verified: I could not run Findings
Review threadsThe listing returned 6 of my own threads, no human replies on any. Resolved 2 — one asked to rename a |
The test located the join back by the last " ON " in the statement and then checked only the text after it, which is the measure side alone. The keys side's projection of that key - the expression the comparison is supposed to match - was never looked at, so a keys-side regression would have passed. It now collects every rendering of the key, blanks the parameter numbers and requires them all to agree, then checks that the join back compares the keys side's alias against that expression. The join back is located by the alias of the measure-side copy followed by its ON, which the keys-side copy does not carry. Also fixes the module doc, which claimed filtering the measure side can never change the result. That holds for a binding in the cube sql's top-level WHERE, not for one that restricts values rather than rows, and `expect` does not format, so the SQL in its message was printed literally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-ups. The filter the keys subquery applies was derived in two files that have to agree: the keys select's own WHERE, and the filter set the measure side's bindings resolve against. The whole correctness argument rests on those two being one expression, so `KeysSubQuery::where_filter` is now that expression and both sites call it. `Filter` and `FilterItem` gain a `Debug` forwarding to the `DebugSql` they already implement, which lets the build context keep its derive - printing it is what one does when a node renders against the wrong filters, the bug class this branch is about. The unreachable conjunction arm of `binding_filters` gains unit tests pinning what it renders, so its semantics are stated rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 7m 3s —— View job Review complete — approach is sound; 0 high, 1 medium (test gap), 2 low. 3 inline comments posted.Full review
VerdictThe diagnosis and the fix match the symptom in #11740. Both central claims in the PR body hold as written:
Coverage of the join back looks complete: the Fixture arithmetic checks out against both snapshots (t1 + the July 29–Aug 27 range keeps orders 1–3; Findings
Removing Not verified
Review threadsThe listing returned 8 unresolved threads, all opened by |
| /// Filters an enclosing construct applies on this select's behalf, for | ||
| /// the `FILTER_PARAMS` and `FILTER_GROUP` bindings of its sources to | ||
| /// resolve against. Set it where a select carries no WHERE of its own but | ||
| /// its sources still have to see the query's filters; it is conjoined | ||
| /// with the WHERE filter, never substituted for it. |
There was a problem hiding this comment.
Five lines where three would do. The load-bearing sentences are what the field is for and that it is conjoined rather than substituted; "Set it where a select carries no WHERE of its own but its sources still have to see the query's filters" restates the two call sites, which already say it in their own comments.
| /// Filters an enclosing construct applies on this select's behalf, for | |
| /// the `FILTER_PARAMS` and `FILTER_GROUP` bindings of its sources to | |
| /// resolve against. Set it where a select carries no WHERE of its own but | |
| /// its sources still have to see the query's filters; it is conjoined | |
| /// with the WHERE filter, never substituted for it. | |
| /// Filters an enclosing construct applies on this select's behalf, for the | |
| /// `FILTER_PARAMS` and `FILTER_GROUP` bindings of its sources to resolve | |
| /// against. Conjoined with the WHERE filter, never substituted for it. |
The suite covered only bindings in the cube sql's top-level WHERE, where the keys side applies the same predicate to the same rows and the measure side can only shrink its build. The two shapes that genuinely move a measure had no fixture, so the behaviour-changing half of this branch went unpinned. Both are now measured against Postgres: - a binding over a column of a table joined inside the cube's sql, which gates a projected value rather than a row, so the key set is untouched and the superset argument does not reach it; - a binding in a cube the measure join tree reaches and the keys tree does not, whose rows drop with nothing on the keys side to offset them. They answer differently - a gated value reads 0, a dropped row reads NULL - and both fail without the fix, returning the unfiltered measure side. The users fixture gains a signup date unrelated to any order, since a predicate over a fact column is implied by what the keys side already applies and cannot show any of this, and an order whose buyer is a second vip, so the results stay non-degenerate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 5m 56s —— View job No blocking defects. 1 low (test gap, inline), 1 note (no legacy-parity guard), 2 prior threads re-affirmed. Review details
VerdictThe mechanism is right and narrowly scoped. Both copies of the fact source now read the same accessor — the keys select's WHERE goes through FindingsLow — Note (no inline) — the legacy-parity claim is untested. The PR's risk argument rests on the moved values being the ones Re-affirmed, not re-posted
Threads4 of my own threads returned by the listing: 2 resolved ( Not verified
|
| // to resolve the same way in both, or the two stop being one expression and | ||
| // the join matches nothing. | ||
| #[test] | ||
| fn the_join_key_renders_the_same_expression_on_both_sides() { |
There was a problem hiding this comment.
The join-key equivalence is pinned only for the Cube source. the_join_key_renders_the_same_expression_on_both_sides queries orders_per_buyer, which takes AggregateMultipliedSubquerySource::Cube, where the measure-side key is rendered in the ON clause through join_visitor_context (aggregate_multiplied_subquery.rs:116). The MeasureSubquery branch reaches the key by a different path: the subquery projects it (measure_subquery.rs:45-47) and the ON clause compares aliases only (aggregate_multiplied_subquery.rs:169-173), so the binding inside the PK's sql resolves through select_builder.set_filter_params_filters at measure_subquery.rs:57 instead. That is the arm where a regression is silent: an unfiltered PK rendering there makes CASE WHEN 1 = 1 THEN id END on the measure side vs CASE WHEN (tenant_id = $1) THEN id END on the keys side, the join matches only the rows the binding kept, and the measure comes back NULL rather than wrong-and-obvious.
Same schema, one more test:
let ctx = TestContext::new(schema_with_key(
PUSHED_DOWN_SCAN,
"CASE WHEN {FILTER_PARAMS_COLUMN:fpmjb_orders.tenant_id:tenant_id} THEN id END",
))
.unwrap();
let (sql, _) = ctx
.build_sql_and_params(&query_for("fpmjb_orders.vip_amount"))
.unwrap();
let renderings = key_expressions(&sql);
assert_eq!(
renderings,
vec!["CASE WHEN (tenant_id = $) THEN id END".to_string(); renderings.len()],
"sql: {}",
sql
);The filters the measure subquery's bindings resolve against travelled in the push-down build context, which every processor clones and forwards. They are meaningful on exactly one edge - the aggregate that owns the subquery - and nothing but a comment and a take() kept them from reaching anywhere else. A second way into that processor would have rendered a subquery whose bindings resolve against filters its rows were never restricted by, and answered a wrong measure rather than failing. They are now an argument on that edge. `MeasureSubquery` is no longer a `ProcessableNode`, so the generic path that cannot carry them does not compile, and the context, the take() and the comments explaining them are gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @waralexrom's task in 6m 28s —— View job Approach is sound — no correctness defect found. 0 high, 0 medium, 3 low (2 comment/dead-branch cleanups, 1 test-assertion gap). 3 inline comments posted. Full review
What I verifiedThe fix reaches the symptom in #11740. The measure-side fact copy's Both sides really do read one filter set: I checked the PR's claim that a resolved binding can never emit a reference to a cube absent from the measure-side FROM, since that would be a hard SQL error rather than a moved number:
So the fallback-to-own-SQL branch is unreachable for anything the subtree search retains, and the blast radius really is confined to Seed data and the two "moves the measure" snapshots are self-consistent: for Removing Findings
One note on the PR description
The last commit ( Not verified
Review threadsThe listing returned 3 threads of mine. Resolved 1 ( |
| /// Deliberately not a `ProcessableNode`: a measure subquery is only meaningful | ||
| /// inside the aggregate that owns it, which is the only thing that knows the | ||
| /// filters its sources must resolve their bindings against. Going through the | ||
| /// generic `process_node` would lose them silently, so there is no way in. |
There was a problem hiding this comment.
Nine lines of doc comment across this struct and process below, most of it narrating the refactor rather than stating a reason a later edit could violate. "Going through the generic process_node would lose them silently, so there is no way in" describes a ProcessableNode impl that no longer exists — a reader of the current file has no way to reintroduce it by accident, because the only call site already passes the filters.
The load-bearing sentence is that this select carries no WHERE of its own and its sources' bindings resolve against the enclosing keys subquery's filters. One line on filter_params_filters says it; the struct doc says nothing the signature doesn't.
| /// Deliberately not a `ProcessableNode`: a measure subquery is only meaningful | |
| /// inside the aggregate that owns it, which is the only thing that knows the | |
| /// filters its sources must resolve their bindings against. Going through the | |
| /// generic `process_node` would lose them silently, so there is no way in. | |
| /// Not a `ProcessableNode`: only the enclosing aggregate knows the filters | |
| /// this subquery's sources must resolve their bindings against. |
Same on process (lines 25-29) — five lines for "these are the enclosing keys subquery's filters; this select has no WHERE of its own".
| /// Everything that constrains the rows this select emits, as one | ||
| /// conjunction: its own WHERE and whatever an enclosing construct applies | ||
| /// on its behalf. A binding may push any of it into a source's scan. | ||
| fn binding_filters(filter: Option<Filter>, from_enclosing: Option<Filter>) -> Option<Filter> { | ||
| match (filter, from_enclosing) { | ||
| (Some(filter), Some(from_enclosing)) => Some(Filter { | ||
| items: filter | ||
| .items | ||
| .into_iter() | ||
| .chain(from_enclosing.items) | ||
| .collect(), | ||
| }), | ||
| (filter, from_enclosing) => filter.or(from_enclosing), | ||
| } | ||
| } |
There was a problem hiding this comment.
The conjunction branch is unreachable in production. The only two set_filter_params_filters callers — aggregate_multiplied_subquery.rs:198 and measure_subquery.rs:61 — never call set_filter on the same builder, and the only set_filter callers (keys_sub_query.rs:114, query.rs:223/:235) never set the params filters. So binding_filters always takes the (x, None) / (None, y) arm; the merge is exercised only by the unit test right below it.
That is defensible as the safe composition for a future caller, but it means the doc on set_filter_params_filters ("it is conjoined with the WHERE filter, never substituted for it") documents behaviour no plan shape currently produces, and the unit test asserting item ordering pins a contract nothing reads. Worth either a one-line note that no call site sets both today, or dropping the branch to filter.or(from_enclosing) and letting the next caller that needs the merge add it with a real test behind it.
| #[tokio::test(flavor = "multi_thread")] | ||
| async fn a_binding_over_a_joined_column_moves_the_measure() { | ||
| let Some(result) = TestContext::new(schema(JOINED_VALUE_BINDING_SCAN)) | ||
| .unwrap() | ||
| .try_execute_pg(&query_for("fpmjb_orders.vip_amount"), SEED) | ||
| .await | ||
| else { | ||
| return; | ||
| }; | ||
| insta::assert_snapshot!(result); | ||
| } | ||
|
|
||
| #[tokio::test(flavor = "multi_thread")] | ||
| async fn a_binding_in_a_measure_side_cube_moves_the_measure() { | ||
| let Some(result) = TestContext::new(schema_with_users(PUSHED_DOWN_SCAN, USERS_BINDING_SCAN)) | ||
| .unwrap() | ||
| .try_execute_pg(&query_for("fpmjb_orders.vip_amount"), SEED) | ||
| .await | ||
| else { | ||
| return; | ||
| }; | ||
| insta::assert_snapshot!(result); |
There was a problem hiding this comment.
These two tests pin the moved value but not the claim that justifies moving it. The PR's argument for accepting the change is "the new value is the one the keys side and the legacy planner already produce" — an insta snapshot of a → NULL, b → 700 records only that Tesseract answers something, and a future regression that moves it again just re-blesses the snapshot.
The equivalence is expressible with the harness already here: the same model and filters without fpmjb_order_tags.tag is the non-multiplied query that renders one filtered copy of the fact source, and its per-measure total is what the multiplied plan must sum to. For a_binding_in_a_measure_side_cube_moves_the_measure, vip_amount over USERS_BINDING_SCAN with no tag dimension is 700 (order 6, user 3 signed up in range; orders 1-2's user 1 dropped) — the same 700 the tag rows total. assert_pushdown_is_result_neutral is already the pattern; a sibling that compares multiplied against non-multiplied would state it.
As written, if the pushdown ever started dropping a row on the measure side rather than a value, a → NULL would still be a green snapshot.
Fixes #11740.
Problem
When a measure that needs the full-key plan is requested together with a dimension of a
one_to_manyjoined cube, the planner builds a keys subquery joined back to a second copy of the fact source by primary key.The keys-side copy is planned with the query's filters, so the cube's
FILTER_PARAMSbindings render as real predicates. The measure side was planned with no filter context at all, so every binding collapsed to always-true:Results stayed correct — the keys side restricts the output — but the database built the join against the entire unfiltered fact table: all tenants, all time. On a large fact table the hash build outgrows the memory limit and the query fails.
The legacy planner is unaffected: its
FILTER_PARAMSproxy reads the query-levelallFiltersregardless of which sub-select is being rendered, so both copies come out restricted.CUBEJS_TESSERACT_SQL_PLANNER=falseis therefore a workaround, but the legacy planner is scheduled for removal.Cause
AggregateMultipliedSubquery's outer select andMeasureSubquery's select carry no WHERE clause of their own — the keys subquery already restricts the rows. In Tesseract the filter set thatFILTER_PARAMS/FILTER_GROUPbindings resolve against is taken from the select's WHERE filter, so for these two selects it was empty and every binding fell back toalways_true.What changed
SelectBuildernow takes the filters its sources'FILTER_PARAMS/FILTER_GROUPbindings resolve against separately from its WHERE clause (set_filter_params_filters), defaulting to the WHERE filter as before.KeysSubQuery::where_filter) that the keys select's WHERE also goes through, so the two copies of the fact source cannot drift apart:AggregateMultipliedSubquerySource::Cube— the bare cube;AggregateMultipliedSubquerySource::MeasureSubquery— reached by a measure whosefilters:reach another cube; it had the same gap.VisitorContextof the join-backONclause, so a binding inside the primary key's ownsqlrenders identically on both sides of the comparison.MeasureSubqueryis no longer aProcessableNode, so the generic path that cannot carry them does not compile.SelectBuilder::new_from_select, which had no callers and would have carried the two filter sets inconsistently.No WHERE clause is added anywhere, and no other plan shape changes.
On the reporter's proposal
@icoolguy1995 suggested pushing the subset of the query's filters whose members belong to the key cube into the bare cube source. That is the right diagnosis, and this change is the narrower form of it: rather than synthesising a predicate over the key cube's members, it lets the cube's existing
FILTER_PARAMSbindings resolve, which is what the legacy planner does and what the issue's SQL actually shows missing. It needs no member-ownership filtering (a binding renders the column stated at the binding site, never the member's own SQL, so a filter on another cube simply matches no binding) and cannot produce a reference to a cube that is not joined on the measure side.What this does to results
The measure side is brought into agreement with the keys side — and with what a plain, non-multiplied query over the same model already renders. That is the intended reading of a
FILTER_PARAMSbinding, and it is what the legacy planner produces. It is not, in general, a no-op:For a binding in the cube
sql's top-levelWHERE— the shape the issue reports — the result cannot move. Both copies read the same fact rows over the same columns, and the join back is by primary key. The keys side applies pushdown ∩ WHERE, the measure side applies pushdown only, so the measure side stays a superset of the key set the join looks up: no matched row can disappear and noLEFT JOINcan turn into a NULL. Only the hash build shrinks. The Postgres tests measure exactly this.Two shapes outside that argument can move a measure, both toward the value the keys side and the legacy planner already produce:
LEFT JOIN ... ONin the cube'ssql, or inside a projectedCASE. Row count is unchanged, so the superset argument says nothing; asumover the affected column answers differently.sqlnow resolves and can drop rows from that join, with nothing on the keys side to offset it.Cumulative and rolling measures are planned outside this branch, so no window-widening semantics are involved.
How it was verified
count, rewritten toCOUNT(DISTINCT id)over a single filtered copy, anchors them.1 = 1 AND 1 = 1) and pass after.ONclause renders the key as one expression on both sides, for a primary key whose ownsqlcarries a binding.cargo fmt,cargo clippy --all-targetsclean.Risks
The changed filter set feeds nothing but
FILTER_PARAMS/FILTER_GROUPresolution (VisitorContext::all_filters, read only bySqlCall::eval_filter_group), so no plan shape and no WHERE clause moves. What can move is a measure value, in the two shapes listed above plus a binding used negated (WHERE NOT (...)), whose measure side goes from empty to filtered. In every case the new value is the one the keys side and the legacy planner already produce, so a model that reads correctly underCUBEJS_TESSERACT_SQL_PLANNER=falseis unaffected; a model that had come to rely on the measure side being unfiltered under Tesseract will see its numbers change.🤖 Generated with Claude Code