fix: preserve scan order during sort-preserving merge limit pushdown - #24273
fix: preserve scan order during sort-preserving merge limit pushdown#24273Theodus wants to merge 3 commits into
Conversation
36adcf3 to
2003a43
Compare
|
@Nagato-Yuzuru thanks for the suggestion, added in 0dab6cf. |
Signed-off-by: Theo Butler <theodusbutler@gmail.com>
Signed-off-by: Theo Butler <theodusbutler@gmail.com>
Signed-off-by: Theo Butler <theodusbutler@gmail.com>
2003a43 to
e972cd3
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24273 +/- ##
==========================================
+ Coverage 81.03% 81.26% +0.22%
==========================================
Files 1107 1110 +3
Lines 385158 384908 -250
Branches 385158 384908 -250
==========================================
+ Hits 312132 312796 +664
+ Misses 54606 53642 -964
- Partials 18420 18470 +50 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
bvolpato
left a comment
There was a problem hiding this comment.
Other than this inline question, this looks good to me. Fix and regression coverage make sense.
| global_state.skip = skip; | ||
| global_state.fetch = fetch; | ||
| global_state.preserve_order = limit_info.preserve_order; | ||
| global_state.preserve_order |= limit_info.preserve_order; |
There was a problem hiding this comment.
Could we clear preserve_order when ordered fetch scope ends instead of carrying it into later limits? For example, with SortPreservingMergeExec(fetch=5) -> SortExec -> LocalLimitExec(fetch=10) -> DataSourceExec, SortExec clears outer fetch, but this |= keeps flag set and applies it to independent inner limit. That disables Parquet limit pruning and file-stream work stealing even though inner limit is not order-sensitive. Is this a concern, or am I missing where state is reset?
Which issue does this PR close?
LimitPushdowncan return incorrect rows for fetchedSortPreservingMergeExecover ordered Parquet scans #24272.Rationale for this change
Queries that filter ordered data and apply
ORDER BY ... LIMITcan return incorrect rows when the physical plan contains a fetchedSortPreservingMergeExec.When the fetch is pushed into an ordered Parquet scan,
LimitPushdowncurrently treats it as order-insensitive. This allows Parquet's limit-based row-group pruning to discard an earlier partially matched row group in favor of a later fully matched row group. The scan can therefore return later rows instead of the first rows in the requested ordering.For example, a query shaped like:
may skip the row group containing the lowest matching values and return values from a later row group.
What changes are included in this PR?
SortPreservingMergeExecas order-sensitive during limit pushdown.preserve_order = truewhen pushing its fetch into the underlying scan.limit = Some(5)andpreserve_order = true.This prevents order-insensitive Parquet limit pruning from changing which rows are eligible for an ordered limit.
Are these changes tested?
Yes. The new
preserves_order_when_pushing_fetch_from_sort_preserving_mergeregression test fails without this change because limit pushdown resets the scan'spreserve_orderflag tofalse.Are there any user-facing changes?
Yes. Ordered queries with a pushed-down limit now return the earliest matching rows instead of rows from a later Parquet row group.
There are no public API changes.