You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ur...@apache.org on 2026/08/05 09:37:43 UTC
(spark) branch branch-4.x updated: [SPARK-58545][PYTHON][TEST] Add tests for pa.Array.to_pandas with types_mapper
This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new a6e75dae63de [SPARK-58545][PYTHON][TEST] Add tests for pa.Array.to_pandas with types_mapper
a6e75dae63de is described below
commit a6e75dae63dee1322a9415419a1faad2285d73f2
Author: Spenser Sun <hs...@gmail.com>
AuthorDate: Wed Aug 5 09:37:20 2026 +0000
[SPARK-58545][PYTHON][TEST] Add tests for pa.Array.to_pandas with types_mapper
### What changes were proposed in this pull request?
This PR adds a golden-file test pinning the behavior of `pa.Array.to_pandas(types_mapper=pd.ArrowDtype)`, under the umbrella [SPARK-54936](https://issues.apache.org/jira/browse/SPARK-54936) (monitor behavior changes from upstream). It is a new test method on the existing `PyArrowArrayToPandasZeroCopyTests`, so it reuses that class's row set and the `_verify_zero_copy` / `_arrow_buffers` helpers rather than adding new machinery:
- `test_to_pandas_zero_copy_only_arrow_backed`, with the new golden file `golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.{csv,md}` (125 rows x 4 columns).
- Columns: `pyarrow array` | `types_mapper=pd.ArrowDtype, zero_copy_only=False` | `types_mapper=pd.ArrowDtype, zero_copy_only=True` | `verified zero-copy`.
Both `zero_copy_only` states are recorded even though the flag makes no observable difference on this backend today. Omitting a column because it looks redundant would bake that assumption into a monitoring test; recording both means a version where the flag starts to matter shows up as a diff instead of passing silently.
No `dev/sparktestsupport/modules.py` change is needed, since this is a new method in an already-registered test module.
### Why are the changes needed?
PySpark calls this argument in production, at `ArrowArrayToPandasConversion.convert_numpy` (`python/pyspark/sql/conversion.py`), for `ByteType` / `ShortType` / `IntegerType` / `LongType` when `spark.sql.execution.pythonUDF.pandas.preferIntExtensionDtype` is enabled. That config exists because the integer dtype handed to a pandas UDF otherwise depends on whether the current batch happens to contain a null (SPARK-55788), so a change in how `types_mapper` maps Arrow types would change wh [...]
The existing goldens cover `to_pandas` on the NumPy backend. Asking pandas for Arrow-backed storage is a different conversion, and the point of that backend is that it can keep pointing at the Arrow buffers instead of materializing them into NumPy. Recording it in its own golden file lets the two be read side by side, and every row in this golden is `zero-copy`: strings, nested types and multi-chunk arrays, which `golden_pyarrow_arrow_to_pandas_zero_copy` records as copied on the NumP [...]
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
New golden-file test. Generated with `SPARK_GENERATE_GOLDEN_FILES=1`, then every cell was reviewed before being committed.
The whole `python/pyspark/tests/upstream/pyarrow/` directory passes (49 passed, 1 skipped), and regeneration is byte-identical, so the golden is deterministic.
Validated across **PyArrow 18-25 x pandas 2 and pandas 3 = 16/16 combinations pass**, each in a fresh venv running the committed test file against the committed golden.
All 16 pass with an empty `overrides` dict, which is worth noting because the sibling methods in this file do need a pandas-3 override for their string rows. `types_mapper=pd.ArrowDtype` selects the dtype explicitly, so pandas' default never applies and the pandas-3 change to it does not reach these cells: the default string conversion goes `object` (pandas 2) -> `str` (pandas 3), while `types_mapper` reads `string[pyarrow]` on both. This was checked cell by cell, not just on the stri [...]
Since this golden is uniformly `zero-copy`, the `verified zero-copy` column was sanity-checked rather than trusted: the empty rows perform a real buffer comparison rather than a vacuous zero-length one, and multi-chunk arrays preserve both chunk addresses instead of being combined. The same helper produces 86 `copied` rows in `golden_pyarrow_arrow_to_pandas_zero_copy`, so it is not rubber-stamping.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
Closes #57771 from Spenserrrr/types-mapper-tests.
Authored-by: Spenser Sun <hs...@gmail.com>
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
(cherry picked from commit 055c0b9647b5a28ca5d590bdd2ce0cc2a32e3c98)
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
...rrow_arrow_to_pandas_zero_copy_arrow_backed.csv | 126 ++++++++++++++++++++
...arrow_arrow_to_pandas_zero_copy_arrow_backed.md | 127 +++++++++++++++++++++
.../test_pyarrow_arrow_to_pandas_non_default.py | 50 ++++++++
3 files changed, 303 insertions(+)
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
new file mode 100644
index 000000000000..11f2831ca359
--- /dev/null
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
@@ -0,0 +1,126 @@
+test case pyarrow array types_mapper=pd.ArrowDtype, zero_copy_only=False types_mapper=pd.ArrowDtype, zero_copy_only=True verified zero-copy
+int8:standard [0, 1, -1, 127, -128]@int8 [0, 1, -1, 127, -128]@Series[int8[pyarrow]] [0, 1, -1, 127, -128]@Series[int8[pyarrow]] zero-copy
+int8:nullable [0, 1, None]@int8 [0, 1, <NA>]@Series[int8[pyarrow]] [0, 1, <NA>]@Series[int8[pyarrow]] zero-copy
+int8:empty []@int8 []@Series[int8[pyarrow]] []@Series[int8[pyarrow]] zero-copy
+int16:standard [0, 1, -1, 32767, -32768]@int16 [0, 1, -1, 32767, -32768]@Series[int16[pyarrow]] [0, 1, -1, 32767, -32768]@Series[int16[pyarrow]] zero-copy
+int16:nullable [0, 1, None]@int16 [0, 1, <NA>]@Series[int16[pyarrow]] [0, 1, <NA>]@Series[int16[pyarrow]] zero-copy
+int16:empty []@int16 []@Series[int16[pyarrow]] []@Series[int16[pyarrow]] zero-copy
+int32:standard [0, 1, -1, 2147483647, -2147483648]@int32 [0, 1, -1, 2147483647, -2147483648]@Series[int32[pyarrow]] [0, 1, -1, 2147483647, -2147483648]@Series[int32[pyarrow]] zero-copy
+int32:nullable [0, 1, None]@int32 [0, 1, <NA>]@Series[int32[pyarrow]] [0, 1, <NA>]@Series[int32[pyarrow]] zero-copy
+int32:empty []@int32 []@Series[int32[pyarrow]] []@Series[int32[pyarrow]] zero-copy
+int64:standard [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 [0, 1, -1, 9223372036854775807, -9223372036854775808]@Series[int64[pyarrow]] [0, 1, -1, 9223372036854775807, -9223372036854775808]@Series[int64[pyarrow]] zero-copy
+int64:nullable [0, 1, None]@int64 [0, 1, <NA>]@Series[int64[pyarrow]] [0, 1, <NA>]@Series[int64[pyarrow]] zero-copy
+int64:empty []@int64 []@Series[int64[pyarrow]] []@Series[int64[pyarrow]] zero-copy
+uint8:standard [0, 1, 255]@uint8 [0, 1, 255]@Series[uint8[pyarrow]] [0, 1, 255]@Series[uint8[pyarrow]] zero-copy
+uint8:nullable [0, 1, None]@uint8 [0, 1, <NA>]@Series[uint8[pyarrow]] [0, 1, <NA>]@Series[uint8[pyarrow]] zero-copy
+uint8:empty []@uint8 []@Series[uint8[pyarrow]] []@Series[uint8[pyarrow]] zero-copy
+uint16:standard [0, 1, 65535]@uint16 [0, 1, 65535]@Series[uint16[pyarrow]] [0, 1, 65535]@Series[uint16[pyarrow]] zero-copy
+uint16:nullable [0, 1, None]@uint16 [0, 1, <NA>]@Series[uint16[pyarrow]] [0, 1, <NA>]@Series[uint16[pyarrow]] zero-copy
+uint16:empty []@uint16 []@Series[uint16[pyarrow]] []@Series[uint16[pyarrow]] zero-copy
+uint32:standard [0, 1, 4294967295]@uint32 [0, 1, 4294967295]@Series[uint32[pyarrow]] [0, 1, 4294967295]@Series[uint32[pyarrow]] zero-copy
+uint32:nullable [0, 1, None]@uint32 [0, 1, <NA>]@Series[uint32[pyarrow]] [0, 1, <NA>]@Series[uint32[pyarrow]] zero-copy
+uint32:empty []@uint32 []@Series[uint32[pyarrow]] []@Series[uint32[pyarrow]] zero-copy
+uint64:standard [0, 1, 18446744073709551615]@uint64 [0, 1, 18446744073709551615]@Series[uint64[pyarrow]] [0, 1, 18446744073709551615]@Series[uint64[pyarrow]] zero-copy
+uint64:nullable [0, 1, None]@uint64 [0, 1, <NA>]@Series[uint64[pyarrow]] [0, 1, <NA>]@Series[uint64[pyarrow]] zero-copy
+uint64:empty []@uint64 []@Series[uint64[pyarrow]] []@Series[uint64[pyarrow]] zero-copy
+float32:standard [0.0, 1.5, -1.5]@float32 [0.0, 1.5, -1.5]@Series[float[pyarrow]] [0.0, 1.5, -1.5]@Series[float[pyarrow]] zero-copy
+float32:nullable [0.0, 1.5, None]@float32 [0.0, 1.5, <NA>]@Series[float[pyarrow]] [0.0, 1.5, <NA>]@Series[float[pyarrow]] zero-copy
+float32:empty []@float32 []@Series[float[pyarrow]] []@Series[float[pyarrow]] zero-copy
+float64:standard [0.0, 1.5, -1.5]@float64 [0.0, 1.5, -1.5]@Series[double[pyarrow]] [0.0, 1.5, -1.5]@Series[double[pyarrow]] zero-copy
+float64:nullable [0.0, 1.5, None]@float64 [0.0, 1.5, <NA>]@Series[double[pyarrow]] [0.0, 1.5, <NA>]@Series[double[pyarrow]] zero-copy
+float64:special [nan, inf, -inf]@float64 [nan, inf, -inf]@Series[double[pyarrow]] [nan, inf, -inf]@Series[double[pyarrow]] zero-copy
+float64:empty []@float64 []@Series[double[pyarrow]] []@Series[double[pyarrow]] zero-copy
+bool:standard [True, False, True]@bool [True, False, True]@Series[bool[pyarrow]] [True, False, True]@Series[bool[pyarrow]] zero-copy
+bool:nullable [True, False, None]@bool [True, False, <NA>]@Series[bool[pyarrow]] [True, False, <NA>]@Series[bool[pyarrow]] zero-copy
+bool:empty []@bool []@Series[bool[pyarrow]] []@Series[bool[pyarrow]] zero-copy
+string:standard [hello, world, ]@string ['hello', 'world', '']@Series[string[pyarrow]] ['hello', 'world', '']@Series[string[pyarrow]] zero-copy
+string:nullable [hello, None, world]@string ['hello', <NA>, 'world']@Series[string[pyarrow]] ['hello', <NA>, 'world']@Series[string[pyarrow]] zero-copy
+string:empty []@string []@Series[string[pyarrow]] []@Series[string[pyarrow]] zero-copy
+large_string:standard [hello, world]@large_string ['hello', 'world']@Series[large_string[pyarrow]] ['hello', 'world']@Series[large_string[pyarrow]] zero-copy
+large_string:nullable [hello, None]@large_string ['hello', <NA>]@Series[large_string[pyarrow]] ['hello', <NA>]@Series[large_string[pyarrow]] zero-copy
+large_string:empty []@large_string []@Series[large_string[pyarrow]] []@Series[large_string[pyarrow]] zero-copy
+binary:standard [b'hello', b'world']@binary [b'hello', b'world']@Series[binary[pyarrow]] [b'hello', b'world']@Series[binary[pyarrow]] zero-copy
+binary:nullable [b'hello', None]@binary [b'hello', <NA>]@Series[binary[pyarrow]] [b'hello', <NA>]@Series[binary[pyarrow]] zero-copy
+binary:empty []@binary []@Series[binary[pyarrow]] []@Series[binary[pyarrow]] zero-copy
+large_binary:standard [b'hello', b'world']@large_binary [b'hello', b'world']@Series[large_binary[pyarrow]] [b'hello', b'world']@Series[large_binary[pyarrow]] zero-copy
+large_binary:nullable [b'hello', None]@large_binary [b'hello', <NA>]@Series[large_binary[pyarrow]] [b'hello', <NA>]@Series[large_binary[pyarrow]] zero-copy
+large_binary:empty []@large_binary []@Series[large_binary[pyarrow]] []@Series[large_binary[pyarrow]] zero-copy
+decimal128:standard [1.23, 4.56, -7.89]@decimal128(5, 2) [Decimal('1.23'), Decimal('4.56'), Decimal('-7.89')]@Series[decimal128(5, 2)[pyarrow]] [Decimal('1.23'), Decimal('4.56'), Decimal('-7.89')]@Series[decimal128(5, 2)[pyarrow]] zero-copy
+decimal128:nullable [1.23, None, 4.56]@decimal128(5, 2) [Decimal('1.23'), <NA>, Decimal('4.56')]@Series[decimal128(5, 2)[pyarrow]] [Decimal('1.23'), <NA>, Decimal('4.56')]@Series[decimal128(5, 2)[pyarrow]] zero-copy
+decimal128:empty []@decimal128(5, 2) []@Series[decimal128(5, 2)[pyarrow]] []@Series[decimal128(5, 2)[pyarrow]] zero-copy
+date32:standard [2024-01-01, 2024-06-15]@date32[day] [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[date32[day][pyarrow]] [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[date32[day][pyarrow]] zero-copy
+date32:nullable [2024-01-01, None]@date32[day] [datetime.date(2024, 1, 1), <NA>]@Series[date32[day][pyarrow]] [datetime.date(2024, 1, 1), <NA>]@Series[date32[day][pyarrow]] zero-copy
+date32:empty []@date32[day] []@Series[date32[day][pyarrow]] []@Series[date32[day][pyarrow]] zero-copy
+date64:standard [2024-01-01, 2024-06-15]@date64[ms] [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[date64[ms][pyarrow]] [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[date64[ms][pyarrow]] zero-copy
+date64:nullable [2024-01-01, None]@date64[ms] [datetime.date(2024, 1, 1), <NA>]@Series[date64[ms][pyarrow]] [datetime.date(2024, 1, 1), <NA>]@Series[date64[ms][pyarrow]] zero-copy
+date64:empty []@date64[ms] []@Series[date64[ms][pyarrow]] []@Series[date64[ms][pyarrow]] zero-copy
+timestamp[s]:standard [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[s] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[s][pyarrow]] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[s][pyarrow]] zero-copy
+timestamp[s]:nullable [2024-01-01 12:00:00, None]@timestamp[s] [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[s][pyarrow]] [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[s][pyarrow]] zero-copy
+timestamp[s]:empty []@timestamp[s] []@Series[timestamp[s][pyarrow]] []@Series[timestamp[s][pyarrow]] zero-copy
+timestamp[ms]:standard [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[ms] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[ms][pyarrow]] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[ms][pyarrow]] zero-copy
+timestamp[ms]:nullable [2024-01-01 12:00:00, None]@timestamp[ms] [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ms][pyarrow]] [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ms][pyarrow]] zero-copy
+timestamp[ms]:empty []@timestamp[ms] []@Series[timestamp[ms][pyarrow]] []@Series[timestamp[ms][pyarrow]] zero-copy
+timestamp[us]:standard [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[us] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[us][pyarrow]] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[us][pyarrow]] zero-copy
+timestamp[us]:nullable [2024-01-01 12:00:00, None]@timestamp[us] [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[us][pyarrow]] [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[us][pyarrow]] zero-copy
+timestamp[us]:empty []@timestamp[us] []@Series[timestamp[us][pyarrow]] []@Series[timestamp[us][pyarrow]] zero-copy
+timestamp[ns]:standard [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[ns] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[ns][pyarrow]] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[ns][pyarrow]] zero-copy
+timestamp[ns]:nullable [2024-01-01 12:00:00, None]@timestamp[ns] [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ns][pyarrow]] [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ns][pyarrow]] zero-copy
+timestamp[ns]:empty []@timestamp[ns] []@Series[timestamp[ns][pyarrow]] []@Series[timestamp[ns][pyarrow]] zero-copy
+timestamp[us,tz=UTC]:standard [2024-01-01 12:00:00+00:00, 2024-06-15 18:30:00+00:00]@timestamp[us, tz=UTC] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15 18:30:00+0000', tz='UTC')]@Series[timestamp[us, tz=UTC][pyarrow]] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15 18:30:00+0000', tz='UTC')]@Series[timestamp[us, tz=UTC][pyarrow]] zero-copy
+timestamp[us,tz=UTC]:nullable [2024-01-01 12:00:00+00:00, None]@timestamp[us, tz=UTC] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), <NA>]@Series[timestamp[us, tz=UTC][pyarrow]] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), <NA>]@Series[timestamp[us, tz=UTC][pyarrow]] zero-copy
+timestamp[us,tz=UTC]:empty []@timestamp[us, tz=UTC] []@Series[timestamp[us, tz=UTC][pyarrow]] []@Series[timestamp[us, tz=UTC][pyarrow]] zero-copy
+duration[s]:standard [1 day, 0:00:00, 2:30:00]@duration[s] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[s][pyarrow]] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[s][pyarrow]] zero-copy
+duration[s]:nullable [1 day, 0:00:00, None]@duration[s] [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[s][pyarrow]] [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[s][pyarrow]] zero-copy
+duration[s]:empty []@duration[s] []@Series[duration[s][pyarrow]] []@Series[duration[s][pyarrow]] zero-copy
+duration[ms]:standard [1 day, 0:00:00, 2:30:00]@duration[ms] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[ms][pyarrow]] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[ms][pyarrow]] zero-copy
+duration[ms]:nullable [1 day, 0:00:00, None]@duration[ms] [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[ms][pyarrow]] [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[ms][pyarrow]] zero-copy
+duration[ms]:empty []@duration[ms] []@Series[duration[ms][pyarrow]] []@Series[duration[ms][pyarrow]] zero-copy
+duration[us]:standard [1 day, 0:00:00, 2:30:00]@duration[us] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[us][pyarrow]] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[us][pyarrow]] zero-copy
+duration[us]:nullable [1 day, 0:00:00, None]@duration[us] [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[us][pyarrow]] [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[us][pyarrow]] zero-copy
+duration[us]:empty []@duration[us] []@Series[duration[us][pyarrow]] []@Series[duration[us][pyarrow]] zero-copy
+duration[ns]:standard [1 days 00:00:00, 0 days 02:30:00]@duration[ns] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[ns][pyarrow]] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[ns][pyarrow]] zero-copy
+duration[ns]:nullable [1 days 00:00:00, None]@duration[ns] [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[ns][pyarrow]] [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[ns][pyarrow]] zero-copy
+duration[ns]:empty []@duration[ns] []@Series[duration[ns][pyarrow]] []@Series[duration[ns][pyarrow]] zero-copy
+time32[s]:standard [12:30:00, 18:45:30]@time32[s] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time32[s][pyarrow]] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time32[s][pyarrow]] zero-copy
+time32[s]:nullable [12:30:00, None]@time32[s] [datetime.time(12, 30), <NA>]@Series[time32[s][pyarrow]] [datetime.time(12, 30), <NA>]@Series[time32[s][pyarrow]] zero-copy
+time32[s]:empty []@time32[s] []@Series[time32[s][pyarrow]] []@Series[time32[s][pyarrow]] zero-copy
+time32[ms]:standard [12:30:00, 18:45:30]@time32[ms] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time32[ms][pyarrow]] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time32[ms][pyarrow]] zero-copy
+time32[ms]:nullable [12:30:00, None]@time32[ms] [datetime.time(12, 30), <NA>]@Series[time32[ms][pyarrow]] [datetime.time(12, 30), <NA>]@Series[time32[ms][pyarrow]] zero-copy
+time32[ms]:empty []@time32[ms] []@Series[time32[ms][pyarrow]] []@Series[time32[ms][pyarrow]] zero-copy
+time64[us]:standard [12:30:00, 18:45:30]@time64[us] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time64[us][pyarrow]] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time64[us][pyarrow]] zero-copy
+time64[us]:nullable [12:30:00, None]@time64[us] [datetime.time(12, 30), <NA>]@Series[time64[us][pyarrow]] [datetime.time(12, 30), <NA>]@Series[time64[us][pyarrow]] zero-copy
+time64[us]:empty []@time64[us] []@Series[time64[us][pyarrow]] []@Series[time64[us][pyarrow]] zero-copy
+time64[ns]:standard [12:30:00, 18:45:30]@time64[ns] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time64[ns][pyarrow]] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time64[ns][pyarrow]] zero-copy
+time64[ns]:nullable [12:30:00, None]@time64[ns] [datetime.time(12, 30), <NA>]@Series[time64[ns][pyarrow]] [datetime.time(12, 30), <NA>]@Series[time64[ns][pyarrow]] zero-copy
+time64[ns]:empty []@time64[ns] []@Series[time64[ns][pyarrow]] []@Series[time64[ns][pyarrow]] zero-copy
+null:standard [None, None, None]@null [<NA>, <NA>, <NA>]@Series[null[pyarrow]] [<NA>, <NA>, <NA>]@Series[null[pyarrow]] zero-copy
+null:empty []@null []@Series[null[pyarrow]] []@Series[null[pyarrow]] zero-copy
+list<int64>:standard [[1, 2], [3, 4, 5]]@list<item: int64> [[1, 2], [3, 4, 5]]@Series[list<item: int64>[pyarrow]] [[1, 2], [3, 4, 5]]@Series[list<item: int64>[pyarrow]] zero-copy
+list<int64>:nullable [[1, 2], None, [3]]@list<item: int64> [[1, 2], <NA>, [3]]@Series[list<item: int64>[pyarrow]] [[1, 2], <NA>, [3]]@Series[list<item: int64>[pyarrow]] zero-copy
+list<int64>:empty []@list<item: int64> []@Series[list<item: int64>[pyarrow]] []@Series[list<item: int64>[pyarrow]] zero-copy
+list<string>:standard [['a', 'b'], ['c']]@list<item: string> [['a', 'b'], ['c']]@Series[list<item: string>[pyarrow]] [['a', 'b'], ['c']]@Series[list<item: string>[pyarrow]] zero-copy
+large_list<int64>:standard [[1, 2], [3, 4]]@large_list<item: int64> [[1, 2], [3, 4]]@Series[large_list<item: int64>[pyarrow]] [[1, 2], [3, 4]]@Series[large_list<item: int64>[pyarrow]] zero-copy
+large_list<int64>:empty []@large_list<item: int64> []@Series[large_list<item: int64>[pyarrow]] []@Series[large_list<item: int64>[pyarrow]] zero-copy
+fixed_size_list<int64>[3]:standard [[1, 2, 3], [4, 5, 6]]@fixed_size_list<item: int64>[3] [[1, 2, 3], [4, 5, 6]]@Series[fixed_size_list<item: int64>[3][pyarrow]] [[1, 2, 3], [4, 5, 6]]@Series[fixed_size_list<item: int64>[3][pyarrow]] zero-copy
+fixed_size_list<int64>[3]:empty []@fixed_size_list<item: int64>[3] []@Series[fixed_size_list<item: int64>[3][pyarrow]] []@Series[fixed_size_list<item: int64>[3][pyarrow]] zero-copy
+struct:standard [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]] [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]] zero-copy
+struct:nullable [[('x', 1), ('y', 'a')], None]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, <NA>]@Series[struct<x: int64, y: string>[pyarrow]] [{'x': 1, 'y': 'a'}, <NA>]@Series[struct<x: int64, y: string>[pyarrow]] zero-copy
+struct:empty []@struct<x: int64, y: string> []@Series[struct<x: int64, y: string>[pyarrow]] []@Series[struct<x: int64, y: string>[pyarrow]] zero-copy
+map<string,int64>:standard [[('a', 1), ('b', 2)], [('c', 3)]]@map<string, int64> [[('a', 1), ('b', 2)], [('c', 3)]]@Series[map<string, int64>[pyarrow]] [[('a', 1), ('b', 2)], [('c', 3)]]@Series[map<string, int64>[pyarrow]] zero-copy
+map<string,int64>:empty []@map<string, int64> []@Series[map<string, int64>[pyarrow]] []@Series[map<string, int64>[pyarrow]] zero-copy
+list<list<int64>>:standard [[[1, 2], [3]], [[4, 5, 6]]]@list<item: list<item: int64>> [[[1, 2], [3]], [[4, 5, 6]]]@Series[list<item: list<item: int64>>[pyarrow]] [[[1, 2], [3]], [[4, 5, 6]]]@Series[list<item: list<item: int64>>[pyarrow]] zero-copy
+list<struct>:standard [[{'x': 1}, {'x': 2}], [{'x': 3}]]@list<item: struct<x: int64>> [[{'x': 1}, {'x': 2}], [{'x': 3}]]@Series[list<item: struct<x: int64>>[pyarrow]] [[{'x': 1}, {'x': 2}], [{'x': 3}]]@Series[list<item: struct<x: int64>>[pyarrow]] zero-copy
+list<map<string,int64>>:standard [[[('a', 1)], [('b', 2)]], [[('c', 3)]]]@list<item: map<string, int64>> [[[('a', 1)], [('b', 2)]], [[('c', 3)]]]@Series[list<item: map<string, int64>>[pyarrow]] [[[('a', 1)], [('b', 2)]], [[('c', 3)]]]@Series[list<item: map<string, int64>>[pyarrow]] zero-copy
+struct<struct>:standard [[('outer', {'inner': 1})], [('outer', {'inner': 2})]]@struct<outer: struct<inner: int64>> [{'outer': {'inner': 1}}, {'outer': {'inner': 2}}]@Series[struct<outer: struct<inner: int64>>[pyarrow]] [{'outer': {'inner': 1}}, {'outer': {'inner': 2}}]@Series[struct<outer: struct<inner: int64>>[pyarrow]] zero-copy
+struct<list<int64>>:standard [[('items', [1, 2, 3])], [('items', [4, 5])]]@struct<items: list<item: int64>> [{'items': [1, 2, 3]}, {'items': [4, 5]}]@Series[struct<items: list<item: int64>>[pyarrow]] [{'items': [1, 2, 3]}, {'items': [4, 5]}]@Series[struct<items: list<item: int64>>[pyarrow]] zero-copy
+struct<map<string,int64>>:standard [[('mapping', [('a', 1)])], [('mapping', [('b', 2)])]]@struct<mapping: map<string, int64>> [{'mapping': [('a', 1)]}, {'mapping': [('b', 2)]}]@Series[struct<mapping: map<string, int64>>[pyarrow]] [{'mapping': [('a', 1)]}, {'mapping': [('b', 2)]}]@Series[struct<mapping: map<string, int64>>[pyarrow]] zero-copy
+map<string,list<int64>>:standard [[('a', [1, 2]), ('b', [3])], [('c', [4, 5, 6])]]@map<string, list<item: int64>> [[('a', [1, 2]), ('b', [3])], [('c', [4, 5, 6])]]@Series[map<string, list<item: int64>>[pyarrow]] [[('a', [1, 2]), ('b', [3])], [('c', [4, 5, 6])]]@Series[map<string, list<item: int64>>[pyarrow]] zero-copy
+map<string,struct>:standard [[('a', {'v': 1}), ('b', {'v': 2})], [('c', {'v': 3})]]@map<string, struct<v: int64>> [[('a', {'v': 1}), ('b', {'v': 2})], [('c', {'v': 3})]]@Series[map<string, struct<v: int64>>[pyarrow]] [[('a', {'v': 1}), ('b', {'v': 2})], [('c', {'v': 3})]]@Series[map<string, struct<v: int64>>[pyarrow]] zero-copy
+map<string,map<string,int64>>:standard [[('a', [('x', 1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@map<string, map<string, int64>> [[('a', [('x', 1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@Series[map<string, map<string, int64>>[pyarrow]] [[('a', [('x', 1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@Series[map<string, map<string, int64>>[pyarrow]] zero-copy
+dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] ['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] zero-copy
+dictionary<int32,string>:nullable [a, b, None, a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] ['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] zero-copy
+dictionary<int32,string>:empty []@dictionary<values=string, indices=int32, ordered=0> []@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] []@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] zero-copy
+int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64[pyarrow]] [2, 3, 4]@Series[int64[pyarrow]] zero-copy
+int64:sliced-with-null [2, None, 4]@int64 [2, <NA>, 4]@Series[int64[pyarrow]] [2, <NA>, 4]@Series[int64[pyarrow]] zero-copy
+int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64[pyarrow]] [1, 2, 3]@Series[int64[pyarrow]] zero-copy
+int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64[pyarrow]] [1, 2, 3, 4]@Series[int64[pyarrow]] zero-copy
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
new file mode 100644
index 000000000000..ac177461a2ed
--- /dev/null
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
@@ -0,0 +1,127 @@
+| test case | pyarrow array | types_mapper=pd.ArrowDtype, zero_copy_only=False | types_mapper=pd.ArrowDtype, zero_copy_only=True | verified zero-copy |
+|----------------------------------------|-----------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|----------------------|
+| int8:standard | [0, 1, -1, 127, -128]@int8 | [0, 1, -1, 127, -128]@Series[int8[pyarrow]] | [0, 1, -1, 127, -128]@Series[int8[pyarrow]] | zero-copy |
+| int8:nullable | [0, 1, None]@int8 | [0, 1, <NA>]@Series[int8[pyarrow]] | [0, 1, <NA>]@Series[int8[pyarrow]] | zero-copy |
+| int8:empty | []@int8 | []@Series[int8[pyarrow]] | []@Series[int8[pyarrow]] | zero-copy |
+| int16:standard | [0, 1, -1, 32767, -32768]@int16 | [0, 1, -1, 32767, -32768]@Series[int16[pyarrow]] | [0, 1, -1, 32767, -32768]@Series[int16[pyarrow]] | zero-copy |
+| int16:nullable | [0, 1, None]@int16 | [0, 1, <NA>]@Series[int16[pyarrow]] | [0, 1, <NA>]@Series[int16[pyarrow]] | zero-copy |
+| int16:empty | []@int16 | []@Series[int16[pyarrow]] | []@Series[int16[pyarrow]] | zero-copy |
+| int32:standard | [0, 1, -1, 2147483647, -2147483648]@int32 | [0, 1, -1, 2147483647, -2147483648]@Series[int32[pyarrow]] | [0, 1, -1, 2147483647, -2147483648]@Series[int32[pyarrow]] | zero-copy |
+| int32:nullable | [0, 1, None]@int32 | [0, 1, <NA>]@Series[int32[pyarrow]] | [0, 1, <NA>]@Series[int32[pyarrow]] | zero-copy |
+| int32:empty | []@int32 | []@Series[int32[pyarrow]] | []@Series[int32[pyarrow]] | zero-copy |
+| int64:standard | [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 | [0, 1, -1, 9223372036854775807, -9223372036854775808]@Series[int64[pyarrow]] | [0, 1, -1, 9223372036854775807, -9223372036854775808]@Series[int64[pyarrow]] | zero-copy |
+| int64:nullable | [0, 1, None]@int64 | [0, 1, <NA>]@Series[int64[pyarrow]] | [0, 1, <NA>]@Series[int64[pyarrow]] | zero-copy |
+| int64:empty | []@int64 | []@Series[int64[pyarrow]] | []@Series[int64[pyarrow]] | zero-copy |
+| uint8:standard | [0, 1, 255]@uint8 | [0, 1, 255]@Series[uint8[pyarrow]] | [0, 1, 255]@Series[uint8[pyarrow]] | zero-copy |
+| uint8:nullable | [0, 1, None]@uint8 | [0, 1, <NA>]@Series[uint8[pyarrow]] | [0, 1, <NA>]@Series[uint8[pyarrow]] | zero-copy |
+| uint8:empty | []@uint8 | []@Series[uint8[pyarrow]] | []@Series[uint8[pyarrow]] | zero-copy |
+| uint16:standard | [0, 1, 65535]@uint16 | [0, 1, 65535]@Series[uint16[pyarrow]] | [0, 1, 65535]@Series[uint16[pyarrow]] | zero-copy |
+| uint16:nullable | [0, 1, None]@uint16 | [0, 1, <NA>]@Series[uint16[pyarrow]] | [0, 1, <NA>]@Series[uint16[pyarrow]] | zero-copy |
+| uint16:empty | []@uint16 | []@Series[uint16[pyarrow]] | []@Series[uint16[pyarrow]] | zero-copy |
+| uint32:standard | [0, 1, 4294967295]@uint32 | [0, 1, 4294967295]@Series[uint32[pyarrow]] | [0, 1, 4294967295]@Series[uint32[pyarrow]] | zero-copy |
+| uint32:nullable | [0, 1, None]@uint32 | [0, 1, <NA>]@Series[uint32[pyarrow]] | [0, 1, <NA>]@Series[uint32[pyarrow]] | zero-copy |
+| uint32:empty | []@uint32 | []@Series[uint32[pyarrow]] | []@Series[uint32[pyarrow]] | zero-copy |
+| uint64:standard | [0, 1, 18446744073709551615]@uint64 | [0, 1, 18446744073709551615]@Series[uint64[pyarrow]] | [0, 1, 18446744073709551615]@Series[uint64[pyarrow]] | zero-copy |
+| uint64:nullable | [0, 1, None]@uint64 | [0, 1, <NA>]@Series[uint64[pyarrow]] | [0, 1, <NA>]@Series[uint64[pyarrow]] | zero-copy |
+| uint64:empty | []@uint64 | []@Series[uint64[pyarrow]] | []@Series[uint64[pyarrow]] | zero-copy |
+| float32:standard | [0.0, 1.5, -1.5]@float32 | [0.0, 1.5, -1.5]@Series[float[pyarrow]] | [0.0, 1.5, -1.5]@Series[float[pyarrow]] | zero-copy |
+| float32:nullable | [0.0, 1.5, None]@float32 | [0.0, 1.5, <NA>]@Series[float[pyarrow]] | [0.0, 1.5, <NA>]@Series[float[pyarrow]] | zero-copy |
+| float32:empty | []@float32 | []@Series[float[pyarrow]] | []@Series[float[pyarrow]] | zero-copy |
+| float64:standard | [0.0, 1.5, -1.5]@float64 | [0.0, 1.5, -1.5]@Series[double[pyarrow]] | [0.0, 1.5, -1.5]@Series[double[pyarrow]] | zero-copy |
+| float64:nullable | [0.0, 1.5, None]@float64 | [0.0, 1.5, <NA>]@Series[double[pyarrow]] | [0.0, 1.5, <NA>]@Series[double[pyarrow]] | zero-copy |
+| float64:special | [nan, inf, -inf]@float64 | [nan, inf, -inf]@Series[double[pyarrow]] | [nan, inf, -inf]@Series[double[pyarrow]] | zero-copy |
+| float64:empty | []@float64 | []@Series[double[pyarrow]] | []@Series[double[pyarrow]] | zero-copy |
+| bool:standard | [True, False, True]@bool | [True, False, True]@Series[bool[pyarrow]] | [True, False, True]@Series[bool[pyarrow]] | zero-copy |
+| bool:nullable | [True, False, None]@bool | [True, False, <NA>]@Series[bool[pyarrow]] | [True, False, <NA>]@Series[bool[pyarrow]] | zero-copy |
+| bool:empty | []@bool | []@Series[bool[pyarrow]] | []@Series[bool[pyarrow]] | zero-copy |
+| string:standard | [hello, world, ]@string | ['hello', 'world', '']@Series[string[pyarrow]] | ['hello', 'world', '']@Series[string[pyarrow]] | zero-copy |
+| string:nullable | [hello, None, world]@string | ['hello', <NA>, 'world']@Series[string[pyarrow]] | ['hello', <NA>, 'world']@Series[string[pyarrow]] | zero-copy |
+| string:empty | []@string | []@Series[string[pyarrow]] | []@Series[string[pyarrow]] | zero-copy |
+| large_string:standard | [hello, world]@large_string | ['hello', 'world']@Series[large_string[pyarrow]] | ['hello', 'world']@Series[large_string[pyarrow]] | zero-copy |
+| large_string:nullable | [hello, None]@large_string | ['hello', <NA>]@Series[large_string[pyarrow]] | ['hello', <NA>]@Series[large_string[pyarrow]] | zero-copy |
+| large_string:empty | []@large_string | []@Series[large_string[pyarrow]] | []@Series[large_string[pyarrow]] | zero-copy |
+| binary:standard | [b'hello', b'world']@binary | [b'hello', b'world']@Series[binary[pyarrow]] | [b'hello', b'world']@Series[binary[pyarrow]] | zero-copy |
+| binary:nullable | [b'hello', None]@binary | [b'hello', <NA>]@Series[binary[pyarrow]] | [b'hello', <NA>]@Series[binary[pyarrow]] | zero-copy |
+| binary:empty | []@binary | []@Series[binary[pyarrow]] | []@Series[binary[pyarrow]] | zero-copy |
+| large_binary:standard | [b'hello', b'world']@large_binary | [b'hello', b'world']@Series[large_binary[pyarrow]] | [b'hello', b'world']@Series[large_binary[pyarrow]] | zero-copy |
+| large_binary:nullable | [b'hello', None]@large_binary | [b'hello', <NA>]@Series[large_binary[pyarrow]] | [b'hello', <NA>]@Series[large_binary[pyarrow]] | zero-copy |
+| large_binary:empty | []@large_binary | []@Series[large_binary[pyarrow]] | []@Series[large_binary[pyarrow]] | zero-copy |
+| decimal128:standard | [1.23, 4.56, -7.89]@decimal128(5, 2) | [Decimal('1.23'), Decimal('4.56'), Decimal('-7.89')]@Series[decimal128(5, 2)[pyarrow]] | [Decimal('1.23'), Decimal('4.56'), Decimal('-7.89')]@Series[decimal128(5, 2)[pyarrow]] | zero-copy |
+| decimal128:nullable | [1.23, None, 4.56]@decimal128(5, 2) | [Decimal('1.23'), <NA>, Decimal('4.56')]@Series[decimal128(5, 2)[pyarrow]] | [Decimal('1.23'), <NA>, Decimal('4.56')]@Series[decimal128(5, 2)[pyarrow]] | zero-copy |
+| decimal128:empty | []@decimal128(5, 2) | []@Series[decimal128(5, 2)[pyarrow]] | []@Series[decimal128(5, 2)[pyarrow]] | zero-copy |
+| date32:standard | [2024-01-01, 2024-06-15]@date32[day] | [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[date32[day][pyarrow]] | [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[date32[day][pyarrow]] | zero-copy |
+| date32:nullable | [2024-01-01, None]@date32[day] | [datetime.date(2024, 1, 1), <NA>]@Series[date32[day][pyarrow]] | [datetime.date(2024, 1, 1), <NA>]@Series[date32[day][pyarrow]] | zero-copy |
+| date32:empty | []@date32[day] | []@Series[date32[day][pyarrow]] | []@Series[date32[day][pyarrow]] | zero-copy |
+| date64:standard | [2024-01-01, 2024-06-15]@date64[ms] | [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[date64[ms][pyarrow]] | [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[date64[ms][pyarrow]] | zero-copy |
+| date64:nullable | [2024-01-01, None]@date64[ms] | [datetime.date(2024, 1, 1), <NA>]@Series[date64[ms][pyarrow]] | [datetime.date(2024, 1, 1), <NA>]@Series[date64[ms][pyarrow]] | zero-copy |
+| date64:empty | []@date64[ms] | []@Series[date64[ms][pyarrow]] | []@Series[date64[ms][pyarrow]] | zero-copy |
+| timestamp[s]:standard | [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[s] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[s][pyarrow]] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[s][pyarrow]] | zero-copy |
+| timestamp[s]:nullable | [2024-01-01 12:00:00, None]@timestamp[s] | [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[s][pyarrow]] | [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[s][pyarrow]] | zero-copy |
+| timestamp[s]:empty | []@timestamp[s] | []@Series[timestamp[s][pyarrow]] | []@Series[timestamp[s][pyarrow]] | zero-copy |
+| timestamp[ms]:standard | [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[ms] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[ms][pyarrow]] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[ms][pyarrow]] | zero-copy |
+| timestamp[ms]:nullable | [2024-01-01 12:00:00, None]@timestamp[ms] | [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ms][pyarrow]] | [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ms][pyarrow]] | zero-copy |
+| timestamp[ms]:empty | []@timestamp[ms] | []@Series[timestamp[ms][pyarrow]] | []@Series[timestamp[ms][pyarrow]] | zero-copy |
+| timestamp[us]:standard | [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[us] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[us][pyarrow]] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[us][pyarrow]] | zero-copy |
+| timestamp[us]:nullable | [2024-01-01 12:00:00, None]@timestamp[us] | [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[us][pyarrow]] | [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[us][pyarrow]] | zero-copy |
+| timestamp[us]:empty | []@timestamp[us] | []@Series[timestamp[us][pyarrow]] | []@Series[timestamp[us][pyarrow]] | zero-copy |
+| timestamp[ns]:standard | [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[ns] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[ns][pyarrow]] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[ns][pyarrow]] | zero-copy |
+| timestamp[ns]:nullable | [2024-01-01 12:00:00, None]@timestamp[ns] | [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ns][pyarrow]] | [Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ns][pyarrow]] | zero-copy |
+| timestamp[ns]:empty | []@timestamp[ns] | []@Series[timestamp[ns][pyarrow]] | []@Series[timestamp[ns][pyarrow]] | zero-copy |
+| timestamp[us,tz=UTC]:standard | [2024-01-01 12:00:00+00:00, 2024-06-15 18:30:00+00:00]@timestamp[us, tz=UTC] | [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15 18:30:00+0000', tz='UTC')]@Series[timestamp[us, tz=UTC][pyarrow]] | [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15 18:30:00+0000', tz='UTC')]@Series[timestamp[us, tz=UTC][pyarrow]] | zero-copy |
+| timestamp[us,tz=UTC]:nullable | [2024-01-01 12:00:00+00:00, None]@timestamp[us, tz=UTC] | [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), <NA>]@Series[timestamp[us, tz=UTC][pyarrow]] | [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), <NA>]@Series[timestamp[us, tz=UTC][pyarrow]] | zero-copy |
+| timestamp[us,tz=UTC]:empty | []@timestamp[us, tz=UTC] | []@Series[timestamp[us, tz=UTC][pyarrow]] | []@Series[timestamp[us, tz=UTC][pyarrow]] | zero-copy |
+| duration[s]:standard | [1 day, 0:00:00, 2:30:00]@duration[s] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[s][pyarrow]] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[s][pyarrow]] | zero-copy |
+| duration[s]:nullable | [1 day, 0:00:00, None]@duration[s] | [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[s][pyarrow]] | [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[s][pyarrow]] | zero-copy |
+| duration[s]:empty | []@duration[s] | []@Series[duration[s][pyarrow]] | []@Series[duration[s][pyarrow]] | zero-copy |
+| duration[ms]:standard | [1 day, 0:00:00, 2:30:00]@duration[ms] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[ms][pyarrow]] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[ms][pyarrow]] | zero-copy |
+| duration[ms]:nullable | [1 day, 0:00:00, None]@duration[ms] | [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[ms][pyarrow]] | [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[ms][pyarrow]] | zero-copy |
+| duration[ms]:empty | []@duration[ms] | []@Series[duration[ms][pyarrow]] | []@Series[duration[ms][pyarrow]] | zero-copy |
+| duration[us]:standard | [1 day, 0:00:00, 2:30:00]@duration[us] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[us][pyarrow]] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[us][pyarrow]] | zero-copy |
+| duration[us]:nullable | [1 day, 0:00:00, None]@duration[us] | [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[us][pyarrow]] | [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[us][pyarrow]] | zero-copy |
+| duration[us]:empty | []@duration[us] | []@Series[duration[us][pyarrow]] | []@Series[duration[us][pyarrow]] | zero-copy |
+| duration[ns]:standard | [1 days 00:00:00, 0 days 02:30:00]@duration[ns] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[ns][pyarrow]] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[ns][pyarrow]] | zero-copy |
+| duration[ns]:nullable | [1 days 00:00:00, None]@duration[ns] | [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[ns][pyarrow]] | [Timedelta('1 days 00:00:00'), <NA>]@Series[duration[ns][pyarrow]] | zero-copy |
+| duration[ns]:empty | []@duration[ns] | []@Series[duration[ns][pyarrow]] | []@Series[duration[ns][pyarrow]] | zero-copy |
+| time32[s]:standard | [12:30:00, 18:45:30]@time32[s] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time32[s][pyarrow]] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time32[s][pyarrow]] | zero-copy |
+| time32[s]:nullable | [12:30:00, None]@time32[s] | [datetime.time(12, 30), <NA>]@Series[time32[s][pyarrow]] | [datetime.time(12, 30), <NA>]@Series[time32[s][pyarrow]] | zero-copy |
+| time32[s]:empty | []@time32[s] | []@Series[time32[s][pyarrow]] | []@Series[time32[s][pyarrow]] | zero-copy |
+| time32[ms]:standard | [12:30:00, 18:45:30]@time32[ms] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time32[ms][pyarrow]] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time32[ms][pyarrow]] | zero-copy |
+| time32[ms]:nullable | [12:30:00, None]@time32[ms] | [datetime.time(12, 30), <NA>]@Series[time32[ms][pyarrow]] | [datetime.time(12, 30), <NA>]@Series[time32[ms][pyarrow]] | zero-copy |
+| time32[ms]:empty | []@time32[ms] | []@Series[time32[ms][pyarrow]] | []@Series[time32[ms][pyarrow]] | zero-copy |
+| time64[us]:standard | [12:30:00, 18:45:30]@time64[us] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time64[us][pyarrow]] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time64[us][pyarrow]] | zero-copy |
+| time64[us]:nullable | [12:30:00, None]@time64[us] | [datetime.time(12, 30), <NA>]@Series[time64[us][pyarrow]] | [datetime.time(12, 30), <NA>]@Series[time64[us][pyarrow]] | zero-copy |
+| time64[us]:empty | []@time64[us] | []@Series[time64[us][pyarrow]] | []@Series[time64[us][pyarrow]] | zero-copy |
+| time64[ns]:standard | [12:30:00, 18:45:30]@time64[ns] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time64[ns][pyarrow]] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[time64[ns][pyarrow]] | zero-copy |
+| time64[ns]:nullable | [12:30:00, None]@time64[ns] | [datetime.time(12, 30), <NA>]@Series[time64[ns][pyarrow]] | [datetime.time(12, 30), <NA>]@Series[time64[ns][pyarrow]] | zero-copy |
+| time64[ns]:empty | []@time64[ns] | []@Series[time64[ns][pyarrow]] | []@Series[time64[ns][pyarrow]] | zero-copy |
+| null:standard | [None, None, None]@null | [<NA>, <NA>, <NA>]@Series[null[pyarrow]] | [<NA>, <NA>, <NA>]@Series[null[pyarrow]] | zero-copy |
+| null:empty | []@null | []@Series[null[pyarrow]] | []@Series[null[pyarrow]] | zero-copy |
+| list<int64>:standard | [[1, 2], [3, 4, 5]]@list<item: int64> | [[1, 2], [3, 4, 5]]@Series[list<item: int64>[pyarrow]] | [[1, 2], [3, 4, 5]]@Series[list<item: int64>[pyarrow]] | zero-copy |
+| list<int64>:nullable | [[1, 2], None, [3]]@list<item: int64> | [[1, 2], <NA>, [3]]@Series[list<item: int64>[pyarrow]] | [[1, 2], <NA>, [3]]@Series[list<item: int64>[pyarrow]] | zero-copy |
+| list<int64>:empty | []@list<item: int64> | []@Series[list<item: int64>[pyarrow]] | []@Series[list<item: int64>[pyarrow]] | zero-copy |
+| list<string>:standard | [['a', 'b'], ['c']]@list<item: string> | [['a', 'b'], ['c']]@Series[list<item: string>[pyarrow]] | [['a', 'b'], ['c']]@Series[list<item: string>[pyarrow]] | zero-copy |
+| large_list<int64>:standard | [[1, 2], [3, 4]]@large_list<item: int64> | [[1, 2], [3, 4]]@Series[large_list<item: int64>[pyarrow]] | [[1, 2], [3, 4]]@Series[large_list<item: int64>[pyarrow]] | zero-copy |
+| large_list<int64>:empty | []@large_list<item: int64> | []@Series[large_list<item: int64>[pyarrow]] | []@Series[large_list<item: int64>[pyarrow]] | zero-copy |
+| fixed_size_list<int64>[3]:standard | [[1, 2, 3], [4, 5, 6]]@fixed_size_list<item: int64>[3] | [[1, 2, 3], [4, 5, 6]]@Series[fixed_size_list<item: int64>[3][pyarrow]] | [[1, 2, 3], [4, 5, 6]]@Series[fixed_size_list<item: int64>[3][pyarrow]] | zero-copy |
+| fixed_size_list<int64>[3]:empty | []@fixed_size_list<item: int64>[3] | []@Series[fixed_size_list<item: int64>[3][pyarrow]] | []@Series[fixed_size_list<item: int64>[3][pyarrow]] | zero-copy |
+| struct:standard | [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]] | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]] | zero-copy |
+| struct:nullable | [[('x', 1), ('y', 'a')], None]@struct<x: int64, y: string> | [{'x': 1, 'y': 'a'}, <NA>]@Series[struct<x: int64, y: string>[pyarrow]] | [{'x': 1, 'y': 'a'}, <NA>]@Series[struct<x: int64, y: string>[pyarrow]] | zero-copy |
+| struct:empty | []@struct<x: int64, y: string> | []@Series[struct<x: int64, y: string>[pyarrow]] | []@Series[struct<x: int64, y: string>[pyarrow]] | zero-copy |
+| map<string,int64>:standard | [[('a', 1), ('b', 2)], [('c', 3)]]@map<string, int64> | [[('a', 1), ('b', 2)], [('c', 3)]]@Series[map<string, int64>[pyarrow]] | [[('a', 1), ('b', 2)], [('c', 3)]]@Series[map<string, int64>[pyarrow]] | zero-copy |
+| map<string,int64>:empty | []@map<string, int64> | []@Series[map<string, int64>[pyarrow]] | []@Series[map<string, int64>[pyarrow]] | zero-copy |
+| list<list<int64>>:standard | [[[1, 2], [3]], [[4, 5, 6]]]@list<item: list<item: int64>> | [[[1, 2], [3]], [[4, 5, 6]]]@Series[list<item: list<item: int64>>[pyarrow]] | [[[1, 2], [3]], [[4, 5, 6]]]@Series[list<item: list<item: int64>>[pyarrow]] | zero-copy |
+| list<struct>:standard | [[{'x': 1}, {'x': 2}], [{'x': 3}]]@list<item: struct<x: int64>> | [[{'x': 1}, {'x': 2}], [{'x': 3}]]@Series[list<item: struct<x: int64>>[pyarrow]] | [[{'x': 1}, {'x': 2}], [{'x': 3}]]@Series[list<item: struct<x: int64>>[pyarrow]] | zero-copy |
+| list<map<string,int64>>:standard | [[[('a', 1)], [('b', 2)]], [[('c', 3)]]]@list<item: map<string, int64>> | [[[('a', 1)], [('b', 2)]], [[('c', 3)]]]@Series[list<item: map<string, int64>>[pyarrow]] | [[[('a', 1)], [('b', 2)]], [[('c', 3)]]]@Series[list<item: map<string, int64>>[pyarrow]] | zero-copy |
+| struct<struct>:standard | [[('outer', {'inner': 1})], [('outer', {'inner': 2})]]@struct<outer: struct<inner: int64>> | [{'outer': {'inner': 1}}, {'outer': {'inner': 2}}]@Series[struct<outer: struct<inner: int64>>[pyarrow]] | [{'outer': {'inner': 1}}, {'outer': {'inner': 2}}]@Series[struct<outer: struct<inner: int64>>[pyarrow]] | zero-copy |
+| struct<list<int64>>:standard | [[('items', [1, 2, 3])], [('items', [4, 5])]]@struct<items: list<item: int64>> | [{'items': [1, 2, 3]}, {'items': [4, 5]}]@Series[struct<items: list<item: int64>>[pyarrow]] | [{'items': [1, 2, 3]}, {'items': [4, 5]}]@Series[struct<items: list<item: int64>>[pyarrow]] | zero-copy |
+| struct<map<string,int64>>:standard | [[('mapping', [('a', 1)])], [('mapping', [('b', 2)])]]@struct<mapping: map<string, int64>> | [{'mapping': [('a', 1)]}, {'mapping': [('b', 2)]}]@Series[struct<mapping: map<string, int64>>[pyarrow]] | [{'mapping': [('a', 1)]}, {'mapping': [('b', 2)]}]@Series[struct<mapping: map<string, int64>>[pyarrow]] | zero-copy |
+| map<string,list<int64>>:standard | [[('a', [1, 2]), ('b', [3])], [('c', [4, 5, 6])]]@map<string, list<item: int64>> | [[('a', [1, 2]), ('b', [3])], [('c', [4, 5, 6])]]@Series[map<string, list<item: int64>>[pyarrow]] | [[('a', [1, 2]), ('b', [3])], [('c', [4, 5, 6])]]@Series[map<string, list<item: int64>>[pyarrow]] | zero-copy |
+| map<string,struct>:standard | [[('a', {'v': 1}), ('b', {'v': 2})], [('c', {'v': 3})]]@map<string, struct<v: int64>> | [[('a', {'v': 1}), ('b', {'v': 2})], [('c', {'v': 3})]]@Series[map<string, struct<v: int64>>[pyarrow]] | [[('a', {'v': 1}), ('b', {'v': 2})], [('c', {'v': 3})]]@Series[map<string, struct<v: int64>>[pyarrow]] | zero-copy |
+| map<string,map<string,int64>>:standard | [[('a', [('x', 1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@map<string, map<string, int64>> | [[('a', [('x', 1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@Series[map<string, map<string, int64>>[pyarrow]] | [[('a', [('x', 1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@Series[map<string, map<string, int64>>[pyarrow]] | zero-copy |
+| dictionary<int32,string>:standard | [a, b, a, b]@dictionary<values=string, indices=int32, ordered=0> | ['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | ['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | zero-copy |
+| dictionary<int32,string>:nullable | [a, b, None, a]@dictionary<values=string, indices=int32, ordered=0> | ['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | ['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | zero-copy |
+| dictionary<int32,string>:empty | []@dictionary<values=string, indices=int32, ordered=0> | []@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | []@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | zero-copy |
+| int64:sliced | [2, 3, 4]@int64 | [2, 3, 4]@Series[int64[pyarrow]] | [2, 3, 4]@Series[int64[pyarrow]] | zero-copy |
+| int64:sliced-with-null | [2, None, 4]@int64 | [2, <NA>, 4]@Series[int64[pyarrow]] | [2, <NA>, 4]@Series[int64[pyarrow]] | zero-copy |
+| int64:single-chunk | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64[pyarrow]] | [1, 2, 3]@Series[int64[pyarrow]] | zero-copy |
+| int64:multi-chunk | [1, 2, 3, 4]@int64 | [1, 2, 3, 4]@Series[int64[pyarrow]] | [1, 2, 3, 4]@Series[int64[pyarrow]] | zero-copy |
\ No newline at end of file
diff --git a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
index 742b7db7389f..7e0dc3a0823a 100644
--- a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
+++ b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
@@ -384,6 +384,13 @@ class PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
so both golden files pin the same types, and appends the layout variants that
only matter for zero-copy: sliced (offset) arrays and single- vs multi-chunk
ChunkedArrays.
+
+ A second test method repeats those rows with ``types_mapper=pd.ArrowDtype``, which
+ asks pandas to keep pointing at the Arrow buffers instead of materializing them
+ into NumPy -- avoiding the copy is the point of that backend. Its own golden file
+ records the result, so the two can be read side by side. PySpark takes this path
+ in ``ArrowArrayToPandasConversion.convert_numpy``
+ (``python/pyspark/sql/conversion.py``).
"""
def _build_source_arrays(self):
@@ -492,6 +499,49 @@ class PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
overrides=overrides,
)
+ # Output columns for the Arrow-backed conversion. Both zero_copy_only states are
+ # recorded even though the flag is expected to make no difference here
+ COL_ARROW_ZERO_COPY_OFF = "types_mapper=pd.ArrowDtype, zero_copy_only=False"
+ COL_ARROW_ZERO_COPY_ON = "types_mapper=pd.ArrowDtype, zero_copy_only=True"
+ COL_ARROW_VERIFIED = "verified zero-copy"
+
+ def test_to_pandas_zero_copy_only_arrow_backed(self):
+ """Test pa.Array.to_pandas(types_mapper=pd.ArrowDtype) against golden file."""
+ sources = self._build_source_arrays()
+ row_names = list(sources.keys())
+ col_names = [
+ "pyarrow array",
+ self.COL_ARROW_ZERO_COPY_OFF,
+ self.COL_ARROW_ZERO_COPY_ON,
+ self.COL_ARROW_VERIFIED,
+ ]
+
+ # Version-specific expected values go here, keyed by (row, col), when a
+ # newer pandas/PyArrow/NumPy legitimately changes a cell's output.
+ overrides: dict[tuple[str, str], str] = {}
+
+ def compute_cell(row_name, col_name):
+ arr = sources[row_name]
+ if col_name == "pyarrow array":
+ return self.repr_value(arr, max_len=0)
+ elif col_name == self.COL_ARROW_ZERO_COPY_OFF:
+ return self._to_pandas_cell(arr, types_mapper=pd.ArrowDtype, zero_copy_only=False)
+ elif col_name == self.COL_ARROW_ZERO_COPY_ON:
+ return self._to_pandas_cell(arr, types_mapper=pd.ArrowDtype, zero_copy_only=True)
+ elif col_name == self.COL_ARROW_VERIFIED:
+ return self._verify_zero_copy(arr, types_mapper=pd.ArrowDtype)
+ else:
+ raise ValueError(f"unknown column: {col_name}")
+
+ self.compare_or_generate_golden_matrix(
+ row_names=row_names,
+ col_names=col_names,
+ compute_cell=compute_cell,
+ golden_file_prefix="golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed",
+ index_name="test case",
+ overrides=overrides,
+ )
+
@unittest.skipIf(
not have_pyarrow or not have_pandas or not have_numpy,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org