Skip to content

fix: Support RANGE window frames over binary ORDER BY keys - #24357

Open
fornwall wants to merge 1 commit into
apache:mainfrom
fornwall:range-window-bytes
Open

fix: Support RANGE window frames over binary ORDER BY keys#24357
fornwall wants to merge 1 commit into
apache:mainfrom
fornwall:range-window-bytes

Conversation

@fornwall

@fornwall fornwall commented Aug 14, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

A window function over a binary ORDER BY key fails even with no frame clause:

SELECT x, COUNT(*) OVER (ORDER BY x)
FROM (VALUES (arrow_cast('a', 'Binary')), (arrow_cast('b', 'Binary'))) t(x);

Internal error: Cannot run range queries on datatype: Binary.

ORDER BY without a frame defaults to RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and extract_window_frame_target_type had no arm for the binary types. LargeBinary, BinaryView, FixedSizeBinary and dictionary-wrapped binary are affected too. Those bounds only need to compare order key values, which binary supports just as Utf8 does.

A finite offset like 1 PRECEDING is different: the bound is computed as current_value - 1, which needs arithmetic on the order key type. Binary has none, and neither do Utf8, Boolean, List and Null -- yet offset frames over Utf8, Boolean and List keys were accepted, and since 54.0.0 the failing arithmetic is swallowed by the overflow handling from #22140, silently collapsing the bound to the partition edge:

SELECT x, COUNT(*) OVER (ORDER BY x RANGE BETWEEN 1 PRECEDING AND CURRENT ROW)
FROM (VALUES ('a'), ('b'), ('c')) t(x) ORDER BY x;

returns the running count 1, 2, 3 -- a silently widened frame -- instead of an error. This PR rejects such frames at planning time instead, matching PostgreSQL.

What changes are included in this PR?

  • Accept the binary types in extract_window_frame_target_type. The existing dictionary arm already recurses into the value type.
  • Reject finite RANGE offsets during planning when the target type has no arithmetic, meaning anything other than the numerics and Interval. The message follows PostgreSQL's RANGE with offset PRECEDING/FOLLOWING is not supported for column type ....
  • Make WindowFrame::free_range() pub, so the check reuses the crate's existing notion of "bounds that are UNBOUNDED or CURRENT ROW".
  • Document the restriction in the window functions guide and the 55.0.0 upgrade guide.

Are these changes tested?

Yes, in window.slt: all four binary types plus dictionary-wrapped binary, ascending and descending, an aggregate and RANK, with duplicate values asserting peer semantics rather than positional counting. Planning errors for offsets over Binary, Utf8, Boolean, List(Int64) and Null, on the end bound as well as the start. Free range frames over Utf8 and Boolean still return results, and Decimal128 keeps its offsets.

The full sqllogictest suite passes.

Are there any user-facing changes?

Yes, one breaking SQL change: RANGE BETWEEN <offset> PRECEDING/FOLLOWING over a Utf8, Binary, Boolean, List or Null ORDER BY key now fails at planning time with RANGE with offset PRECEDING/FOLLOWING is not supported for ORDER BY type .... Over Utf8, Boolean and List keys this replaces the silently widened frames described above; over Null and binary keys such queries already failed at planning time, so only the error message changes. The 55.0.0 upgrade guide entry covers the per-type history and the migration paths: use a ROWS frame to count rows, or state UNBOUNDED PRECEDING / UNBOUNDED FOLLOWING explicitly to compare against the partition edge.

In the other direction, binary order keys now work with free range frames, where they previously raised the internal error above.

WindowFrame::free_range() going from private to pub is additive, so not a breaking Rust API change.


AI usage: Created with Claude Code and Opus 5. I have reviewed the code and made modifications where it made sense. I have also tested this end to end in an internal project.

@github-actions github-actions Bot added documentation Improvements or additions to documentation logical-expr Logical plan and expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Aug 14, 2026
@fornwall
fornwall force-pushed the range-window-bytes branch 2 times, most recently from ce39143 to 5951847 Compare August 14, 2026 06:50
@github-actions github-actions Bot added the development-process Related to development process of DataFusion label Aug 14, 2026
@fornwall
fornwall force-pushed the range-window-bytes branch 2 times, most recently from cbcb22f to e0663ae Compare August 14, 2026 08:13
Closes apache#24327.

A window with an `ORDER BY` over a binary column and no explicit frame gets
the default `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`. Computing
that frame's bounds goes through `extract_window_frame_target_type`, which
had no arm for the binary types, so any window function whose leading
`ORDER BY` key was `Binary`/`LargeBinary`/`BinaryView`/`FixedSizeBinary`
failed with:

    Internal error: Cannot run range queries on datatype: Binary

Binary is orderable, so peer/range comparison is well defined for it in
exactly the same way it already is for `Utf8`. Accept the binary types
alongside the string types.

A finite RANGE offset such as `1 PRECEDING` asks for more than ordering: the
bound is computed as `current_value - 1`, so the order key type must also
support arithmetic. Binary does not, and neither do the `Utf8`, `Boolean`,
`List` and `Null` order keys that `extract_window_frame_target_type` already
accepted. For those, `ScalarValue::sub_checked` failed during execution and
the error was swallowed by the overflow handling from apache#22140, which collapses
a failed bound to the partition edge. So

    SELECT COUNT(*) OVER (ORDER BY x RANGE BETWEEN 1 PRECEDING AND CURRENT ROW)
    FROM (VALUES ('a'), ('b'), ('c')) t(x)

silently returned the running count 1, 2, 3 -- the whole partition -- rather
than an error.

Reject the offset form during planning for every target type that has no
arithmetic. For the string, boolean, list and null order keys this turns a
wrong result into a plan error, which matches PostgreSQL's "RANGE with offset
PRECEDING/FOLLOWING is not supported for column type ...".

`WindowFrame::free_range()` becomes public so that check can be expressed in
the vocabulary the crate already uses for "bounds that need no arithmetic".

Document the restriction in the window functions user guide, and the changed
SQL behavior in the 55.0.0 upgrade guide.

Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
@fornwall
fornwall force-pushed the range-window-bytes branch from e0663ae to 146f516 Compare August 14, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development-process Related to development process of DataFusion documentation Improvements or additions to documentation logical-expr Logical plan and expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Window functions fail when ORDER BY uses a binary column

1 participant