Skip to content

fix: correct list field inner type in array functions - #24345

Merged
timsaucer merged 2 commits into
apache:mainfrom
timsaucer:fix/inner-list-field
Aug 14, 2026
Merged

fix: correct list field inner type in array functions#24345
timsaucer merged 2 commits into
apache:mainfrom
timsaucer:fix/inner-list-field

Conversation

@timsaucer

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

There is a regression from 54.1.0 to 55.0.0rc2 where the spark function slice can fail as demonstrated in the updated unit test, and as described in the connected issue.

What changes are included in this PR?

In the nested functions, get the inner field directly from the input for array slicing operations. Also, since it is possible the inner list could be non-nullable emit an empty slice instead of a null child element for the outer lists's nulls.

Are these changes tested?

Added 6 new tests in the SLT suite.

Are there any user-facing changes?

None

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Aug 13, 2026
@timsaucer

Copy link
Copy Markdown
Member Author

If this is approved I'll create a PR targeting branch-55.

@codecov-commenter

codecov-commenter commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.17%. Comparing base (f171587) to head (7bd8af6).
⚠️ Report is 18 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/functions-nested/src/extract.rs 66.66% 3 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #24345    +/-   ##
========================================
  Coverage   81.17%   81.17%            
========================================
  Files        1109     1109            
  Lines      388033   388167   +134     
  Branches   388033   388167   +134     
========================================
+ Hits       314985   315107   +122     
- Misses      54509    54513     +4     
- Partials    18539    18547     +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@adriangb adriangb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks for the fix!

One finding out of scope for this PR: array_append, array_prepend and array_replace{,_n,_all} have the same defect and regressed in the same commit. Filed as #24347. They need a slightly different fix, since the appended element can be null and so the promised type is wrong at the source, not just the payload. array_concat, array_remove, array_distinct, array_union, array_intersect, array_sort and array_resize are all fine.

+1 on the branch-55 backport

Comment thread datafusion/functions-nested/src/extract.rs Outdated
Comment thread datafusion/functions-nested/src/extract.rs Outdated
Comment thread datafusion/sqllogictest/test_files/array/array_slice.slt
Comment thread datafusion/sqllogictest/test_files/array/array_pop.slt
Comment thread datafusion/functions-nested/src/extract.rs Outdated
Comment thread datafusion/functions-nested/src/extract.rs Outdated
- Pass `use_nulls: false` to `MutableArrayData` in both `general_array_slice`
  and `general_list_view_array_slice`. Neither calls `try_extend_nulls` any
  more, and arrow ORs the flag with the child array's null count internally,
  so this only avoids an unnecessary validity buffer allocation.
- Name the actual internal function in the `internal_err!` messages. These
  paths are also reachable from `array_pop_front` / `array_pop_back`, so
  reporting "array_slice" was misleading.
- Extend the LargeList slice and pop tests to cover empty and NULL rows so
  the null branch is exercised for i64 offsets, not just i32.
@timsaucer
timsaucer enabled auto-merge August 14, 2026 11:06
@timsaucer
timsaucer added this pull request to the merge queue Aug 14, 2026
Merged via the queue into apache:main with commit 9f377c4 Aug 14, 2026
37 checks passed
@timsaucer
timsaucer deleted the fix/inner-list-field branch August 14, 2026 11:40
timsaucer added a commit that referenced this pull request Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

array_slice discards the input list's inner field (name, nullability, metadata), contradicting its promised return type

4 participants