Describe the bug
ArraySlice::return_type promises arg_types[0].clone(), i.e. the input list type verbatim — inner field name, nullability and metadata included. But the kernel rebuilds the output's inner field from scratch in general_array_slice (extract.rs:685):
Ok(Arc::new(GenericListArray::<O>::try_new(
Arc::new(Field::new_list_field(array.value_type(), true)),
...That drops the field name (any name → item), drops inner metadata, and forces nullable: true. Whenever the input list's inner field is anything other than Field("item", T, nullable: true), the returned array disagrees with the promised type. On debug builds this trips the return-type assertion added in #17515; on release builds it silently produces a batch whose field name/nullability disagrees with the schema the planner recorded.
This is the same defect fixed for array_sort in #19948 (issue #19947). The list-view path in this very function already handles it correctly — general_list_view_array_slice threads the input field through with ListView(field) | LargeListView(field) => Arc::clone(field) (extract.rs:705) — so the two paths inside array_slice disagree.
To Reproduce
Non-nullable inner field:
select array_slice(arrow_cast(make_array(1, 3, 5, -5), 'List(non-null Int32)'), 2, 3);
DataFusion error: Internal error: Assertion failed: result_data_type == *expected_type: Function 'array_slice' returned value of type 'List(Int32)' while the following type was promised at planning time and expected: 'List(non-null Int32)'.
Inner field name, via the Spark slice function (whose SparkSlice::return_field_from_args also copies the input field and delegates to array_slice_udf()), where datafusion-spark's array names its list field element:
slice(array(1, 2, 3, 4), 2, 2)
Internal error: Assertion failed: result_data_type == *expected_type: Function 'array_slice' returned value of type 'List(Int64)' while the following type was promised at planning time and expected: 'List(Int64, field: 'element')'.
Both reproduce on 55.0.0-rc2 (209fd9406).
Expected behavior
array_slice returns a list whose inner field matches the promised return type — the input list's field carried through unchanged, as array_sort does since #19948 and as the list-view path in array_slice already does. For the Spark slice case that also means the Spark element name element survives the slice, matching pyspark.
Additional context
Not a regression in array_slice itself — the kernel has always rebuilt the field. What changed is that #20945 ("avoid extraneous casts for equivalent nested types") stopped normalizing list arg types during signature coercion, so field-name differences now reach the function instead of being erased by an inserted cast. Before #20945, slice(array(...), ...) had a cast rewriting List(element) to List(item), which made promise and payload agree by accident. The non-nullable case (List(non-null Int32)) fails regardless of #20945.
Fix looks like #19948: thread the input list's FieldRef into GenericListArray::try_new instead of constructing Field::new_list_field(array.value_type(), true). That likely also addresses the inner-metadata loss described in #21982 for this function.
Would be good to land before 55.0.0 final — noticed while updating datafusion-python to 55.0.0-rc2, where the Spark slice case forced a local workaround.
Describe the bug
ArraySlice::return_typepromisesarg_types[0].clone(), i.e. the input list type verbatim — inner field name, nullability and metadata included. But the kernel rebuilds the output's inner field from scratch ingeneral_array_slice(extract.rs:685):That drops the field name (any name →
item), drops inner metadata, and forcesnullable: true. Whenever the input list's inner field is anything other thanField("item", T, nullable: true), the returned array disagrees with the promised type. On debug builds this trips the return-type assertion added in #17515; on release builds it silently produces a batch whose field name/nullability disagrees with the schema the planner recorded.This is the same defect fixed for
array_sortin #19948 (issue #19947). The list-view path in this very function already handles it correctly —general_list_view_array_slicethreads the input field through withListView(field) | LargeListView(field) => Arc::clone(field)(extract.rs:705) — so the two paths insidearray_slicedisagree.To Reproduce
Non-nullable inner field:
Inner field name, via the Spark
slicefunction (whoseSparkSlice::return_field_from_argsalso copies the input field and delegates toarray_slice_udf()), wheredatafusion-spark'sarraynames its list fieldelement:Both reproduce on
55.0.0-rc2(209fd9406).Expected behavior
array_slicereturns a list whose inner field matches the promised return type — the input list's field carried through unchanged, asarray_sortdoes since #19948 and as the list-view path inarray_slicealready does. For the Sparkslicecase that also means the Spark element nameelementsurvives the slice, matching pyspark.Additional context
Not a regression in
array_sliceitself — the kernel has always rebuilt the field. What changed is that #20945 ("avoid extraneous casts for equivalent nested types") stopped normalizing list arg types during signature coercion, so field-name differences now reach the function instead of being erased by an inserted cast. Before #20945,slice(array(...), ...)had a cast rewritingList(element)toList(item), which made promise and payload agree by accident. The non-nullable case (List(non-null Int32)) fails regardless of #20945.Fix looks like #19948: thread the input list's
FieldRefintoGenericListArray::try_newinstead of constructingField::new_list_field(array.value_type(), true). That likely also addresses the inner-metadata loss described in #21982 for this function.Would be good to land before 55.0.0 final — noticed while updating datafusion-python to
55.0.0-rc2, where the Sparkslicecase forced a local workaround.