fix: correct list field inner type in array functions - #24345
Conversation
|
If this is approved I'll create a PR targeting |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24345 +/- ##
========================================
Coverage 81.17% 81.17%
========================================
Files 1109 1109
Lines 388033 388167 +134
Branches 388033 388167 +134
========================================
+ Hits 314985 315107 +122
- Misses 54509 54513 +4
- Partials 18539 18547 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
adriangb
left a comment
There was a problem hiding this comment.
Looks good to me, thanks for the fix!
One finding out of scope for this PR: array_append, array_prepend and array_replace{,_n,_all} have the same defect and regressed in the same commit. Filed as #24347. They need a slightly different fix, since the appended element can be null and so the promised type is wrong at the source, not just the payload. array_concat, array_remove, array_distinct, array_union, array_intersect, array_sort and array_resize are all fine.
+1 on the branch-55 backport
- Pass `use_nulls: false` to `MutableArrayData` in both `general_array_slice` and `general_list_view_array_slice`. Neither calls `try_extend_nulls` any more, and arrow ORs the flag with the child array's null count internally, so this only avoids an unnecessary validity buffer allocation. - Name the actual internal function in the `internal_err!` messages. These paths are also reachable from `array_pop_front` / `array_pop_back`, so reporting "array_slice" was misleading. - Extend the LargeList slice and pop tests to cover empty and NULL rows so the null branch is exercised for i64 offsets, not just i32.
Which issue does this PR close?
Rationale for this change
There is a regression from 54.1.0 to 55.0.0rc2 where the spark function
slicecan fail as demonstrated in the updated unit test, and as described in the connected issue.What changes are included in this PR?
In the nested functions, get the inner field directly from the input for array slicing operations. Also, since it is possible the inner list could be non-nullable emit an empty slice instead of a null child element for the outer lists's nulls.
Are these changes tested?
Added 6 new tests in the SLT suite.
Are there any user-facing changes?
None