Skip to content

fix: don't report equi-key equivalence for PiecewiseMergeJoin range joins - #24361

Open
viirya wants to merge 2 commits into
apache:mainfrom
viirya:pwmj-range-equivalence-fix
Open

fix: don't report equi-key equivalence for PiecewiseMergeJoin range joins#24361
viirya wants to merge 2 commits into
apache:mainfrom
viirya:pwmj-range-equivalence-fix

Conversation

@viirya

@viirya viirya commented Aug 14, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

PiecewiseMergeJoinExec::compute_properties passed the join's on pair to join_equivalence_properties as if it were an equijoin key. For an INNER join that registers left_on == right_on as an output equivalence — but PWMJ's on is a range predicate (l.v < r.v), not equality, so the equivalence is false. It let the optimizer treat a sort on the left key as also sorting the right key and drop a required ORDER BY, returning wrongly ordered rows.

set datafusion.optimizer.enable_piecewise_merge_join = true;
create table l(v int) as values (1),(2),(3),(5),(8);
create table r(v int) as values (4),(6),(9),(2);
select l.v, r.v from l join r on l.v < r.v where l.v = 2 order by r.v;
-- PWMJ: 2,9 / 2,6 / 2,4   (wrong order)
-- NLJ:  2,4 / 2,6 / 2,9   (correct)

The plans differ: PWMJ sorts only on l.v (SortExec: expr=[v@0 ASC]); NLJ sorts on both ([v@0 ASC, v@1 ASC]).

What changes are included in this PR?

  • compute_properties no longer passes the range on pair to join_equivalence_properties (a range join adds no column equivalences). The now-unused join_on parameter is dropped.
  • Existing pwmj.slt plans/results updated to the corrected (fully sorted) output — they previously encoded the wrong ordering — and a regression test is added.

Are these changes tested?

Yes.

  • New regression test in pwmj.slt (INNER l.v < r.v, WHERE l.v = 2, ORDER BY r.v) asserting the correctly ordered result.
  • The existing pwmj.slt cases that asserted the wrong ordering / a single-column sort plan are updated to the correct values (verified against NestedLoopJoin).
  • piecewise unit tests and joins.slt still pass. Only INNER is affected.

Are there any user-facing changes?

INNER range joins via PiecewiseMergeJoin (behind enable_piecewise_merge_join, default off) no longer drop a required sort, matching NestedLoopJoin. No API changes.

…oins

`PiecewiseMergeJoinExec::compute_properties` passed the join's `on` pair
to `join_equivalence_properties` as an equijoin key. For an INNER join
that registers `left_on == right_on` as an output equivalence — but PWMJ's
`on` is a range predicate (`l < r`), not equality, so the equivalence is
false. It let the optimizer treat a sort on the left key as also sorting
the right key and drop a required `ORDER BY`, returning wrongly ordered
rows.

A range join adds no column equivalences, so pass no `on` pairs. Existing
pwmj.slt plans/results updated to the now-correct (fully sorted) output;
adds a regression test.

Closes apache#24360.

Co-authored-by: Claude Code
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Aug 14, 2026
@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  75.034s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.141s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  37.667s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.151s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.651s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure method_parameter_count_changed: pub method parameter count changed ---

Description:
A publicly-visible method now takes a different number of parameters, not counting the receiver (self) parameter.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/method_parameter_count_changed.ron

Failed in:
  datafusion_physical_plan::joins::PiecewiseMergeJoinExec::compute_properties takes 5 parameters in /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/9fde98edbee831aa453ffcd4374749d5e630b645/datafusion/physical-plan/src/joins/piecewise_merge_join/exec.rs:423, but now takes 4 parameters in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/joins/piecewise_merge_join/exec.rs:422

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 115.290s] datafusion-physical-plan
    Building datafusion-sqllogictest v54.1.0 (current)
       Built [  99.942s] (current)
     Parsing datafusion-sqllogictest v54.1.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-sqllogictest v54.1.0 (baseline)
       Built [ 100.011s] (baseline)
     Parsing datafusion-sqllogictest v54.1.0 (baseline)
      Parsed [   0.022s] (baseline)
    Checking datafusion-sqllogictest v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.088s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 203.190s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Aug 14, 2026
@codecov-commenter

codecov-commenter commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.18%. Comparing base (571477c) to head (c1f9068).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24361      +/-   ##
==========================================
- Coverage   81.19%   81.18%   -0.01%     
==========================================
  Files        1110     1110              
  Lines      388616   388614       -2     
  Branches   388616   388614       -2     
==========================================
- Hits       315534   315512      -22     
- Misses      54500    54515      +15     
- Partials    18582    18587       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…nce-fix

# Conflicts:
#	datafusion/sqllogictest/test_files/pwmj.slt
@viirya
viirya requested a review from comphead August 14, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PiecewiseMergeJoin reports a false equi-key equivalence for INNER range joins, dropping a required sort

2 participants