[SPARK-58576][PYTHON][TESTS] Add tests for pa.Array.to_pandas with integer_object_nulls by Spenserrrr · Pull Request #57774 · apache/spark

Spenserrrr · 2026-08-04T22:08:13Z

What changes were proposed in this pull request?

Add PyArrowArrayToPandasIntegerObjectNullsTests to the existing test_pyarrow_arrow_to_pandas_non_default.py, pinning the behavior of pa.Array.to_pandas(integer_object_nulls=True) in a golden file. Sub-task of SPARK-54936.

numpy integer dtypes cannot represent a null, so PyArrow must choose a representation when an integer array contains one. By default it widens to float64 with NaN; with integer_object_nulls=True the result stays object dtype holding Python int and None.

The row set reuses all 121 rows of the sibling test_pyarrow_arrow_to_pandas_default.py via _build_source_arrays, so both golden files pin the same types, plus 18 appended integer rows that the shared rows do not reach:

Width extremes with a null (8 rows) — the shared nullable rows use small values that float64 represents exactly. At 64 bits the range exceeds float64's 53-bit mantissa and the value itself changes.
int64:all-null, int64:multi-chunk-nullable — no integer left to convert, and values split across buffers with the null in only one chunk. The latter is reachable in production: cogrouped applyInPandas is the one UDF path that does not call combine_chunks() first.
Nested types with a null integer element (7 rows) — list, large_list, fixed_size_list, list<list>, struct, map. The shared nested rows hold a missing sub-list, which this argument does not affect; a missing element inside one is what it changes. These are also the types convert_legacy still serves.
dictionary<int64>:nullable — control. Dictionary encoding stores the distinct values once plus an index per row, so the null lives in the indices and the int64 values hold none; with no null to represent, the argument has nothing to decide.

Three output columns per row: the argument off, on, and the full pandas_options dict convert_legacy passes. The last is not a duplicate of the second — coerce_temporal_nanoseconds shifts the temporal rows to ns — and pinning the call as Spark makes it also catches PyArrow changing the date_as_object=True default it relies on.

No dev/sparktestsupport/modules.py line is needed; the test file is already registered.

Why are the changes needed?

integer_object_nulls=True is passed unconditionally at python/pyspark/sql/conversion.py:1839, and again to _create_converter_to_pandas at :1850 so the fix-up stage knows what it received. The object Series is an intermediate, not the result: it is narrowed to a nullable extension dtype (Int8Dtype .. Int64Dtype) via _to_corrected_pandas_ext_type, and it is the only bridge from Arrow to those dtypes that avoids float64.

It was added by SPARK-54962 to fix a correctness bug where an identity pandas UDF returned a different number for a large nullable long, and its behavior has never been pinned by a test. Unlike types_mapper on the sibling convert_numpy path, which is gated by spark.sql.execution.pythonUDF.pandas.preferIntExtensionDtype (default false), this argument has no config to fall back on.

Scope is worth being precise about: this argument serves the pandas UDF path over the complex types convert_numpy does not handle yet (TODO(SPARK-55324) at conversion.py:1999-2007). df.toPandas() builds its own options at python/pyspark/sql/pandas/conversion.py:249-251 and does not pass it. That the covered type region is scheduled to move is itself an argument for recording the current semantics now.

Does this PR introduce any user-facing change?

No. Tests only.

How was this patch tested?

New golden-file test, 139 rows x 4 columns. The argument changes 25 rows; the remaining 114 pin that it does not affect them.

Validated across 16/16 environments — PyArrow 18/19/20/21/22/23/24/25 x pandas 2 and 3 — running the committed test file against the committed golden in a fresh venv per combination. Golden regeneration is byte-identical.

Two overrides tiers were needed and both were verified load-bearing by disabling each and observing the failure: pandas 3 renders non-empty Arrow string arrays with its dedicated string dtype, while empty ones only gain it from PyArrow 24.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

…teger_object_nulls Add a golden-file test pinning the behavior of pa.Array.to_pandas(integer_object_nulls=True), which PySpark passes unconditionally in ArrowArrayToPandasConversion.convert_legacy. Co-authored-by: Isaac

Spenserrrr · 2026-08-04T22:14:19Z

Hi @Yicong-Huang @zhengruifeng! Could you take a look when you have a moment? Thank you!

Yicong-Huang

Thanks @Spenserrrr, LGTM, one nit inline

Yicong-Huang · 2026-08-04T22:56:06Z

+        if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+
+            def override_all_outputs(row, expected):
+                for col in col_names[1:]:
+                    overrides[(row, col)] = expected
+
+            for row, expected in [
+                ("string:standard", "['hello', 'world', '']@Series[str]"),
+                ("string:nullable", "['hello', nan, 'world']@Series[str]"),
+                ("large_string:standard", "['hello', 'world']@Series[str]"),
+                ("large_string:nullable", "['hello', nan]@Series[str]"),
+            ]:
+                override_all_outputs(row, expected)
+
+            # Empty string arrays only gain that dtype from PyArrow 24; before that
+            # they stay object even on pandas 3, so the baseline still holds there.
+            if LooseVersion(pa.__version__) >= LooseVersion("24.0.0"):
+                for row in ["string:empty", "large_string:empty"]:
+                    override_all_outputs(row, "[]@Series[str]")


nit: I think it might be better to write this format for readability? I want to avoid nested if and loops.

if pandas > 3 and 18 <= arrow < 24:
overrides[(r1, c1)] = "xxx"
overrides[(r2, c2)] = "yyy"
if pandas > 3 and arrow > 24:
overrides[(r2, c2)] = "zzz"
...

Thanks for the review, @Yicong-Huang! I've just revised according to the suggestion.

Address review: replace the nested if / loop with two flat version predicates and a comprehension, so each version combination reads as one condition. Co-authored-by: Isaac

Drop the staging dict and the comprehension: each version combination is now one flat condition with one explicit call per overridden row, and the predicates are named for the `>=` comparisons they hold. Co-authored-by: Isaac

uros-b · 2026-08-05T08:45:53Z

Thank you @Spenserrrr and @Yicong-Huang @zhengruifeng!

…eger_object_nulls ### What changes were proposed in this pull request? Add `PyArrowArrayToPandasIntegerObjectNullsTests` to the existing `test_pyarrow_arrow_to_pandas_non_default.py`, pinning the behavior of `pa.Array.to_pandas(integer_object_nulls=True)` in a golden file. Sub-task of SPARK-54936. numpy integer dtypes cannot represent a null, so PyArrow must choose a representation when an integer array contains one. By default it widens to `float64` with `NaN`; with `integer_object_nulls=True` the result stays `object` dtype holding Python `int` and `None`. The row set reuses all 121 rows of the sibling `test_pyarrow_arrow_to_pandas_default.py` via `_build_source_arrays`, so both golden files pin the same types, plus 18 appended integer rows that the shared rows do not reach: - **Width extremes with a null** (8 rows) — the shared nullable rows use small values that `float64` represents exactly. At 64 bits the range exceeds float64's 53-bit mantissa and the value itself changes. - **`int64:all-null`, `int64:multi-chunk-nullable`** — no integer left to convert, and values split across buffers with the null in only one chunk. The latter is reachable in production: cogrouped `applyInPandas` is the one UDF path that does not call `combine_chunks()` first. - **Nested types with a null integer element** (7 rows) — `list`, `large_list`, `fixed_size_list`, `list<list>`, `struct`, `map`. The shared nested rows hold a missing *sub-list*, which this argument does not affect; a missing *element inside* one is what it changes. These are also the types `convert_legacy` still serves. - **`dictionary<int64>:nullable`** — control. Dictionary encoding stores the distinct values once plus an index per row, so the null lives in the indices and the int64 values hold none; with no null to represent, the argument has nothing to decide. Three output columns per row: the argument off, on, and the full `pandas_options` dict `convert_legacy` passes. The last is not a duplicate of the second — `coerce_temporal_nanoseconds` shifts the temporal rows to `ns` — and pinning the call as Spark makes it also catches PyArrow changing the `date_as_object=True` default it relies on. No `dev/sparktestsupport/modules.py` line is needed; the test file is already registered. ### Why are the changes needed? `integer_object_nulls=True` is passed unconditionally at `python/pyspark/sql/conversion.py:1839`, and again to `_create_converter_to_pandas` at `:1850` so the fix-up stage knows what it received. The object Series is an intermediate, not the result: it is narrowed to a nullable extension dtype (`Int8Dtype` .. `Int64Dtype`) via `_to_corrected_pandas_ext_type`, and it is the only bridge from Arrow to those dtypes that avoids `float64`. It was added by SPARK-54962 to fix a correctness bug where an identity pandas UDF returned a different number for a large nullable long, and its behavior has never been pinned by a test. Unlike `types_mapper` on the sibling `convert_numpy` path, which is gated by `spark.sql.execution.pythonUDF.pandas.preferIntExtensionDtype` (default false), this argument has no config to fall back on. Scope is worth being precise about: this argument serves the pandas UDF path over the complex types `convert_numpy` does not handle yet (`TODO(SPARK-55324)` at `conversion.py:1999-2007`). `df.toPandas()` builds its own options at `python/pyspark/sql/pandas/conversion.py:249-251` and does not pass it. That the covered type region is scheduled to move is itself an argument for recording the current semantics now. ### Does this PR introduce _any_ user-facing change? No. Tests only. ### How was this patch tested? New golden-file test, 139 rows x 4 columns. The argument changes 25 rows; the remaining 114 pin that it does not affect them. Validated across **16/16** environments — PyArrow 18/19/20/21/22/23/24/25 x pandas 2 and 3 — running the committed test file against the committed golden in a fresh venv per combination. Golden regeneration is byte-identical. Two `overrides` tiers were needed and both were verified load-bearing by disabling each and observing the failure: pandas 3 renders non-empty Arrow string arrays with its dedicated string dtype, while empty ones only gain it from PyArrow 24. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) Closes #57774 from Spenserrrr/integer-object-nulls-tests. Authored-by: Spenser Sun <hsun112358@gmail.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> (cherry picked from commit a8dcc97) Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>

uros-b · 2026-08-05T08:46:30Z

Merge Summary:

merged into master a8dcc97
merged into branch-4.x 083a70d

Posted by merge_spark_pr.py

Spenserrrr marked this pull request as ready for review August 4, 2026 22:13

Yicong-Huang approved these changes Aug 4, 2026

View reviewed changes

Spenserrrr added 2 commits August 4, 2026 23:21

Flatten the version overrides into explicit conditions

3d752a9

Address review: replace the nested if / loop with two flat version predicates and a comprehension, so each version combination reads as one condition. Co-authored-by: Isaac

Simplify the version overrides per review

ff86f53

Drop the staging dict and the comprehension: each version combination is now one flat condition with one explicit call per overridden row, and the predicates are named for the `>=` comparisons they hold. Co-authored-by: Isaac

zhengruifeng approved these changes Aug 5, 2026

View reviewed changes

uros-b approved these changes Aug 5, 2026

View reviewed changes

uros-b closed this in a8dcc97 Aug 5, 2026

Navigation Menu

Uh oh!

[SPARK-58576][PYTHON][TESTS] Add tests for pa.Array.to_pandas with integer_object_nulls - #57774

[SPARK-58576][PYTHON][TESTS] Add tests for pa.Array.to_pandas with integer_object_nulls#57774
Spenserrrr wants to merge 3 commits into
apache:masterfrom
Spenserrrr:integer-object-nulls-tests

Spenserrrr commented Aug 4, 2026

Uh oh!

Spenserrrr commented Aug 4, 2026

Uh oh!

Yicong-Huang left a comment

Uh oh!

Yicong-Huang Aug 4, 2026

Uh oh!

Spenserrrr Aug 4, 2026

Uh oh!

uros-b commented Aug 5, 2026

Uh oh!

uros-b commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Spenserrrr commented Aug 4, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Spenserrrr commented Aug 4, 2026

Uh oh!

Yicong-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

Yicong-Huang Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Spenserrrr Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

uros-b commented Aug 5, 2026

Uh oh!

uros-b commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants