[SPARK-58545][PYTHON][TEST] Add tests for pa.Array.to_pandas with types_mapper

[Spenserrrr]

What changes were proposed in this pull request?

This PR adds a golden-file test pinning the behavior of pa.Array.to_pandas(types_mapper=pd.ArrowDtype), under the umbrella SPARK-54936 (monitor behavior changes from upstream). It is a new test method on the existing PyArrowArrayToPandasZeroCopyTests, so it reuses that class's row set and the _verify_zero_copy / _arrow_buffers helpers rather than adding new machinery:

• test_to_pandas_zero_copy_only_arrow_backed, with the new golden file golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.{csv,md} (125 rows x 4 columns).
• Columns: pyarrow array | types_mapper=pd.ArrowDtype, zero_copy_only=False | types_mapper=pd.ArrowDtype, zero_copy_only=True | verified zero-copy.

Both zero_copy_only states are recorded even though the flag makes no observable difference on this backend today. Omitting a column because it looks redundant would bake that assumption into a monitoring test; recording both means a version where the flag starts to matter shows up as a diff instead of passing silently.

No dev/sparktestsupport/modules.py change is needed, since this is a new method in an already-registered test module.

Why are the changes needed?

PySpark calls this argument in production, at ArrowArrayToPandasConversion.convert_numpy (python/pyspark/sql/conversion.py), for ByteType / ShortType / IntegerType / LongType when spark.sql.execution.pythonUDF.pandas.preferIntExtensionDtype is enabled. That config exists because the integer dtype handed to a pandas UDF otherwise depends on whether the current batch happens to contain a null (SPARK-55788), so a change in how types_mapper maps Arrow types would change what user UDF code receives.

The existing goldens cover to_pandas on the NumPy backend. Asking pandas for Arrow-backed storage is a different conversion, and the point of that backend is that it can keep pointing at the Arrow buffers instead of materializing them into NumPy. Recording it in its own golden file lets the two be read side by side, and every row in this golden is zero-copy: strings, nested types and multi-chunk arrays, which golden_pyarrow_arrow_to_pandas_zero_copy records as copied on the NumPy backend, all reuse their buffers here. The tz-aware timestamp rows are the clearest contrast, since those are the ones where the NumPy backend reports zero_copy_only=True but pandas copies anyway; on this path the flag and the verified result agree.

Does this PR introduce any user-facing change?

No. Test-only.

How was this patch tested?

New golden-file test. Generated with SPARK_GENERATE_GOLDEN_FILES=1, then every cell was reviewed before being committed.

The whole python/pyspark/tests/upstream/pyarrow/ directory passes (49 passed, 1 skipped), and regeneration is byte-identical, so the golden is deterministic.

Validated across PyArrow 18-25 x pandas 2 and pandas 3 = 16/16 combinations pass, each in a fresh venv running the committed test file against the committed golden.

All 16 pass with an empty overrides dict, which is worth noting because the sibling methods in this file do need a pandas-3 override for their string rows. types_mapper=pd.ArrowDtype selects the dtype explicitly, so pandas' default never applies and the pandas-3 change to it does not reach these cells: the default string conversion goes object (pandas 2) -> str (pandas 3), while types_mapper reads string[pyarrow] on both. This was checked cell by cell, not just on the string rows -- all 125 cells are identical between pandas 2 and 3 (on both PyArrow 18 and 24) and between PyArrow 18 and 24.

Since this golden is uniformly zero-copy, the verified zero-copy column was sanity-checked rather than trusted: the empty rows perform a real buffer comparison rather than a vacuous zero-length one, and multi-chunk arrays preserve both chunk addresses instead of being combined. The same helper produces 86 copied rows in golden_pyarrow_arrow_to_pandas_zero_copy, so it is not rubber-stamping.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

[Spenserrrr]

Hi @Yicong-Huang @zhengruifeng! This is the PR testing the types_mapper = pd.ArrowDtype for pa.Array.to_pandas. Could you take a look when you have a moment? Thank you!

[uros-b]

Thank you @Spenserrrr! Adding @Yicong-Huang for further review

[uros-b]

Merge Summary:

• merged into master 055c0b9
• merged into branch-4.x a6e75da

Posted by merge_spark_pr.py