You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/08/11 03:41:48 UTC
(spark) branch master updated: [SPARK-58656][PYTHON][TEST][FOLLOWUP] Add tests for pa.ChunkedArray.to_pandas
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7f8403aa1f79 [SPARK-58656][PYTHON][TEST][FOLLOWUP] Add tests for pa.ChunkedArray.to_pandas
7f8403aa1f79 is described below
commit 7f8403aa1f793c569eb8cc05eab917541ba4f143
Author: Spenser Sun <hs...@gmail.com>
AuthorDate: Tue Aug 11 11:41:35 2026 +0800
[SPARK-58656][PYTHON][TEST][FOLLOWUP] Add tests for pa.ChunkedArray.to_pandas
### What changes were proposed in this pull request?
This is a follow-up to SPARK-58656, which centralized the `to_pandas` golden test inventory into a shared `_PyArrowToPandasTestBase`. It extends that inventory to cover `pa.ChunkedArray.to_pandas` across the existing golden tests, plus the memory-tuning arguments that only apply on the chunked (Table->columns) path.
Concretely:
- Adds a `_chunked_sources()` group method to the shared base covering the chunk-count axis (zero-chunk, empty-chunk, single-chunk, and multi-chunk variants) across representative types (int64, float64, string, list, struct). Because the base's `_build_source_arrays()` unions all groups, these rows flow into the default, zero_copy, zero_copy_arrow_backed, and integer_object_nulls goldens automatically, so each records how chunking interacts with its argument.
- Consolidates the chunk rows that previously lived separately in the `ZeroCopy` and `IntegerObjectNulls` subclasses into the shared group.
- Adds `PyArrowChunkedArrayToPandasMemoryFlagsTests`, pinning `to_pandas` under the `self_destruct` / `split_blocks` / `use_threads` bundle that `toPandas()` uses. The three are recorded as one column because Spark only ever sets them together, and a final column records that the source ChunkedArray stays readable after `self_destruct=True` (freeing is an optimization, not a contract).
- Adds chunked `timestamp[us]` rows to the coerce_temporal test, where per-chunk unit coercion is uniquely observable.
Spark converts a `pa.ChunkedArray` (not a plain Array) on the Table->columns path in `python/pyspark/sql/pandas/conversion.py` (`df.toPandas()`, the empty-dataset branch, and the pandas-UDF paths after `combine_chunks()`), so these tests pin the upstream behavior PySpark depends on there.
### Why are the changes needed?
These golden tests act as a drift canary for upstream PyArrow/pandas behavior under umbrella SPARK-54936. `pa.ChunkedArray.to_pandas` was previously covered only incidentally by a few scattered rows; this makes the coverage systematic so a behavior change in a future PyArrow/pandas release fails CI loudly instead of silently changing what PySpark returns.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
New and updated golden-file tests, verified across a sweep of PyArrow 18-25 x pandas 2 and 3 (16 combinations), all passing. Existing goldens whose row set changed were regenerated and diffed to confirm the moved chunk cells are value-identical.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
This pull request and its description were written by Isaac.
Closes #57909 from Spenserrrr/chunkedarray-tests.
Authored-by: Spenser Sun <hs...@gmail.com>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
...den_pyarrow_arrow_to_pandas_coerce_temporal.csv | 2 +
...lden_pyarrow_arrow_to_pandas_coerce_temporal.md | 2 +
.../golden_pyarrow_arrow_to_pandas_default.csv | 12 ++
.../golden_pyarrow_arrow_to_pandas_default.md | 14 +-
...yarrow_arrow_to_pandas_integer_object_nulls.csv | 13 +-
...pyarrow_arrow_to_pandas_integer_object_nulls.md | 13 +-
.../golden_pyarrow_arrow_to_pandas_zero_copy.csv | 14 +-
.../golden_pyarrow_arrow_to_pandas_zero_copy.md | 16 +-
...rrow_arrow_to_pandas_zero_copy_arrow_backed.csv | 14 +-
...arrow_arrow_to_pandas_zero_copy_arrow_backed.md | 16 +-
...yarrow_chunked_array_to_pandas_memory_flags.csv | 11 ++
...pyarrow_chunked_array_to_pandas_memory_flags.md | 12 ++
.../test_pyarrow_arrow_to_pandas_default.py | 72 +++++++-
.../test_pyarrow_arrow_to_pandas_non_default.py | 199 ++++++++++++++++++---
14 files changed, 370 insertions(+), 40 deletions(-)
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
index 4f8439cd1d2e..afc22fdfac99 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
@@ -46,6 +46,8 @@ time64[ns]:nullable [12:30:00, None]@time64[ns] [datetime.time(12, 30), None]@Se
time64[ns]:empty []@time64[ns] []@Series[object] []@Series[object]
timestamp[s]:overflow [2500-01-01 00:00:00]@timestamp[s] ERR@ArrowInvalid ERR@ArrowInvalid
duration[s]:overflow [109500 days, 0:00:00]@duration[s] [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]] [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
+timestamp[us]:single-chunk [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[us] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
+timestamp[us]:multi-chunk [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[us] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
int64:standard [0, 1, -1]@int64 [0, 1, -1]@Series[int64] [0, 1, -1]@Series[int64]
int64:nullable [0, 1, None]@int64 [0.0, 1.0, nan]@Series[float64] [0.0, 1.0, nan]@Series[float64]
float64:standard [0.0, 1.5, -1.5]@float64 [0.0, 1.5, -1.5]@Series[float64] [0.0, 1.5, -1.5]@Series[float64]
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
index e6b3a061c97d..3e3e500fc52e 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
@@ -47,6 +47,8 @@
| time64[ns]:empty | []@time64[ns] | []@Series[object] | []@Series[object] |
| timestamp[s]:overflow | [2500-01-01 00:00:00]@timestamp[s] | ERR@ArrowInvalid | ERR@ArrowInvalid |
| duration[s]:overflow | [109500 days, 0:00:00]@duration[s] | [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]] | [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]] |
+| timestamp[us]:single-chunk | [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[us] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]] |
+| timestamp[us]:multi-chunk | [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[us] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]] |
| int64:standard | [0, 1, -1]@int64 | [0, 1, -1]@Series[int64] | [0, 1, -1]@Series[int64] |
| int64:nullable | [0, 1, None]@int64 | [0.0, 1.0, nan]@Series[float64] | [0.0, 1.0, nan]@Series[float64] |
| float64:standard | [0.0, 1.5, -1.5]@float64 | [0.0, 1.5, -1.5]@Series[float64] | [0.0, 1.5, -1.5]@Series[float64] |
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.csv b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.csv
index 13450e5634ed..5015d7c396f8 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.csv
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.csv
@@ -120,3 +120,15 @@ map<string,map<string,int64>>:standard [[('a', [('x', 1)]), ('b', [('y', 2)])],
dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', 'a', 'b']@Series[category]
dictionary<int32,string>:nullable [a, b, None, a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', nan, 'a']@Series[category]
dictionary<int32,string>:empty []@dictionary<values=string, indices=int32, ordered=0> []@Series[category]
+int64:zero-chunk []@int64 []@Series[int64]
+int64:empty-chunk []@int64 []@Series[int64]
+int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64]
+int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64]
+int64:multi-chunk-nullable [1, None, 2]@int64 [1.0, nan, 2.0]@Series[float64]
+int64:multi-chunk-with-empty [1, 2, 3]@int64 [1, 2, 3]@Series[int64]
+float64:multi-chunk [1.5, 2.5, 3.5]@float64 [1.5, 2.5, 3.5]@Series[float64]
+string:single-chunk [a, b]@string ['a', 'b']@Series[object]
+string:multi-chunk [a, b, c]@string ['a', 'b', 'c']@Series[object]
+string:multi-chunk-nullable [a, None, c]@string ['a', None, 'c']@Series[object]
+list<int64>:multi-chunk [[1, 2], [3], [4]]@list<item: int64> [array([1, 2]), array([3]), array([4])]@Series[object]
+struct:multi-chunk [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object]
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.md b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.md
index 04debdc77e03..48d561f5963a 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.md
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.md
@@ -120,4 +120,16 @@
| map<string,map<string,int64>>:standard | [[('a', [('x', 1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@map<string, map<string, int64>> | [[('a', [('x', 1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@Series[object] |
| dictionary<int32,string>:standard | [a, b, a, b]@dictionary<values=string, indices=int32, ordered=0> | ['a', 'b', 'a', 'b']@Series[category] |
| dictionary<int32,string>:nullable | [a, b, None, a]@dictionary<values=string, indices=int32, ordered=0> | ['a', 'b', nan, 'a']@Series[category] |
-| dictionary<int32,string>:empty | []@dictionary<values=string, indices=int32, ordered=0> | []@Series[category] |
\ No newline at end of file
+| dictionary<int32,string>:empty | []@dictionary<values=string, indices=int32, ordered=0> | []@Series[category] |
+| int64:zero-chunk | []@int64 | []@Series[int64] |
+| int64:empty-chunk | []@int64 | []@Series[int64] |
+| int64:single-chunk | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64] |
+| int64:multi-chunk | [1, 2, 3, 4]@int64 | [1, 2, 3, 4]@Series[int64] |
+| int64:multi-chunk-nullable | [1, None, 2]@int64 | [1.0, nan, 2.0]@Series[float64] |
+| int64:multi-chunk-with-empty | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64] |
+| float64:multi-chunk | [1.5, 2.5, 3.5]@float64 | [1.5, 2.5, 3.5]@Series[float64] |
+| string:single-chunk | [a, b]@string | ['a', 'b']@Series[object] |
+| string:multi-chunk | [a, b, c]@string | ['a', 'b', 'c']@Series[object] |
+| string:multi-chunk-nullable | [a, None, c]@string | ['a', None, 'c']@Series[object] |
+| list<int64>:multi-chunk | [[1, 2], [3], [4]]@list<item: int64> | [array([1, 2]), array([3]), array([4])]@Series[object] |
+| struct:multi-chunk | [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] |
\ No newline at end of file
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.csv b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.csv
index e28e175f4107..e9ef08af1bd0 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.csv
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.csv
@@ -120,6 +120,18 @@ map<string,map<string,int64>>:standard [[('a', [('x', 1)]), ('b', [('y', 2)])],
dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', 'a', 'b']@Series[category] ['a', 'b', 'a', 'b']@Series[category] ['a', 'b', 'a', 'b']@Series[category]
dictionary<int32,string>:nullable [a, b, None, a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', nan, 'a']@Series[category] ['a', 'b', nan, 'a']@Series[category] ['a', 'b', nan, 'a']@Series[category]
dictionary<int32,string>:empty []@dictionary<values=string, indices=int32, ordered=0> []@Series[category] []@Series[category] []@Series[category]
+int64:zero-chunk []@int64 []@Series[int64] []@Series[int64] []@Series[int64]
+int64:empty-chunk []@int64 []@Series[int64] []@Series[int64] []@Series[int64]
+int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2, 3]@Series[int64] [1, 2, 3]@Series[int64]
+int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64] [1, 2, 3, 4]@Series[int64] [1, 2, 3, 4]@Series[int64]
+int64:multi-chunk-nullable [1, None, 2]@int64 [1.0, nan, 2.0]@Series[float64] [1, None, 2]@Series[object] [1, None, 2]@Series[object]
+int64:multi-chunk-with-empty [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2, 3]@Series[int64] [1, 2, 3]@Series[int64]
+float64:multi-chunk [1.5, 2.5, 3.5]@float64 [1.5, 2.5, 3.5]@Series[float64] [1.5, 2.5, 3.5]@Series[float64] [1.5, 2.5, 3.5]@Series[float64]
+string:single-chunk [a, b]@string ['a', 'b']@Series[object] ['a', 'b']@Series[object] ['a', 'b']@Series[object]
+string:multi-chunk [a, b, c]@string ['a', 'b', 'c']@Series[object] ['a', 'b', 'c']@Series[object] ['a', 'b', 'c']@Series[object]
+string:multi-chunk-nullable [a, None, c]@string ['a', None, 'c']@Series[object] ['a', None, 'c']@Series[object] ['a', None, 'c']@Series[object]
+list<int64>:multi-chunk [[1, 2], [3], [4]]@list<item: int64> [array([1, 2]), array([3]), array([4])]@Series[object] [array([1, 2]), array([3]), array([4])]@Series[object] [array([1, 2]), array([3]), array([4])]@Series[object]
+struct:multi-chunk [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object]
int8:extremes-nullable [127, -128, None]@int8 [127.0, -128.0, nan]@Series[float64] [127, -128, None]@Series[object] [127, -128, None]@Series[object]
int16:extremes-nullable [32767, -32768, None]@int16 [32767.0, -32768.0, nan]@Series[float64] [32767, -32768, None]@Series[object] [32767, -32768, None]@Series[object]
int32:extremes-nullable [2147483647, -2147483648, None]@int32 [2147483647.0, -2147483648.0, nan]@Series[float64] [2147483647, -2147483648, None]@Series[object] [2147483647, -2147483648, None]@Series[object]
@@ -129,7 +141,6 @@ uint16:extremes-nullable [65535, 0, None]@uint16 [65535.0, 0.0, nan]@Series[floa
uint32:extremes-nullable [4294967295, 0, None]@uint32 [4294967295.0, 0.0, nan]@Series[float64] [4294967295, 0, None]@Series[object] [4294967295, 0, None]@Series[object]
uint64:extremes-nullable [18446744073709551615, 0, None]@uint64 [1.8446744073709552e+19, 0.0, nan]@Series[float64] [18446744073709551615, 0, None]@Series[object] [18446744073709551615, 0, None]@Series[object]
int64:all-null [None, None]@int64 [nan, nan]@Series[float64] [None, None]@Series[object] [None, None]@Series[object]
-int64:multi-chunk-nullable [1, None, 2]@int64 [1.0, nan, 2.0]@Series[float64] [1, None, 2]@Series[object] [1, None, 2]@Series[object]
list<int64>:null-element [[1, None], [2, 3]]@list<item: int64> [array([ 1., nan]), array([2., 3.])]@Series[object] [array([1, None], dtype=object), array([2, 3], dtype=object)]@Series[object] [array([1, None], dtype=object), array([2, 3], dtype=object)]@Series[object]
list<int64>:null-element-extreme [[9223372036854775807, None]]@list<item: int64> [array([9.22337204e+18, nan])]@Series[object] [array([9223372036854775807, None], dtype=object)]@Series[object] [array([9223372036854775807, None], dtype=object)]@Series[object]
large_list<int64>:null-element [[1, None]]@large_list<item: int64> [array([ 1., nan])]@Series[object] [array([1, None], dtype=object)]@Series[object] [array([1, None], dtype=object)]@Series[object]
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.md b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.md
index 9545c071aa08..661016dc3df1 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.md
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.md
@@ -121,6 +121,18 @@
| dictionary<int32,string>:standard | [a, b, a, b]@dictionary<values=string, indices=int32, ordered=0> | ['a', 'b', 'a', 'b']@Series[category] | ['a', 'b', 'a', 'b']@Series[category] | ['a', 'b', 'a', 'b']@Series[category] [...]
| dictionary<int32,string>:nullable | [a, b, None, a]@dictionary<values=string, indices=int32, ordered=0> | ['a', 'b', nan, 'a']@Series[category] | ['a', 'b', nan, 'a']@Series[category] | ['a', 'b', nan, 'a']@Series[category] [...]
| dictionary<int32,string>:empty | []@dictionary<values=string, indices=int32, ordered=0> | []@Series[category] | []@Series[category] | []@Series[category] [...]
+| int64:zero-chunk | []@int64 | []@Series[int64] | []@Series[int64] | []@Series[int64] [...]
+| int64:empty-chunk | []@int64 | []@Series[int64] | []@Series[int64] | []@Series[int64] [...]
+| int64:single-chunk | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64] | [1, 2, 3]@Series[int64] | [1, 2, 3]@Series[int64] [...]
+| int64:multi-chunk | [1, 2, 3, 4]@int64 | [1, 2, 3, 4]@Series[int64] | [1, 2, 3, 4]@Series[int64] | [1, 2, 3, 4]@Series[int64] [...]
+| int64:multi-chunk-nullable | [1, None, 2]@int64 | [1.0, nan, 2.0]@Series[float64] | [1, None, 2]@Series[object] | [1, None, 2]@Series[object] [...]
+| int64:multi-chunk-with-empty | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64] | [1, 2, 3]@Series[int64] | [1, 2, 3]@Series[int64] [...]
+| float64:multi-chunk | [1.5, 2.5, 3.5]@float64 | [1.5, 2.5, 3.5]@Series[float64] | [1.5, 2.5, 3.5]@Series[float64] | [1.5, 2.5, 3.5]@Series[float64] [...]
+| string:single-chunk | [a, b]@string | ['a', 'b']@Series[object] | ['a', 'b']@Series[object] | ['a', 'b']@Series[object] [...]
+| string:multi-chunk | [a, b, c]@string | ['a', 'b', 'c']@Series[object] | ['a', 'b', 'c']@Series[object] | ['a', 'b', 'c']@Series[object] [...]
+| string:multi-chunk-nullable | [a, None, c]@string | ['a', None, 'c']@Series[object] | ['a', None, 'c']@Series[object] | ['a', None, 'c']@Series[object] [...]
+| list<int64>:multi-chunk | [[1, 2], [3], [4]]@list<item: int64> | [array([1, 2]), array([3]), array([4])]@Series[object] | [array([1, 2]), array([3]), array([4])]@Series[object] | [array([1, 2]), array([3]), array([4])]@Series[object] [...]
+| struct:multi-chunk | [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] [...]
| int8:extremes-nullable | [127, -128, None]@int8 | [127.0, -128.0, nan]@Series[float64] | [127, -128, None]@Series[object] | [127, -128, None]@Series[object] [...]
| int16:extremes-nullable | [32767, -32768, None]@int16 | [32767.0, -32768.0, nan]@Series[float64] | [32767, -32768, None]@Series[object] | [32767, -32768, None]@Series[object] [...]
| int32:extremes-nullable | [2147483647, -2147483648, None]@int32 | [2147483647.0, -2147483648.0, nan]@Series[float64] | [2147483647, -2147483648, None]@Series[object] | [2147483647, -2147483648, None]@Series[object] [...]
@@ -130,7 +142,6 @@
| uint32:extremes-nullable | [4294967295, 0, None]@uint32 | [4294967295.0, 0.0, nan]@Series[float64] | [4294967295, 0, None]@Series[object] | [4294967295, 0, None]@Series[object] [...]
| uint64:extremes-nullable | [18446744073709551615, 0, None]@uint64 | [1.8446744073709552e+19, 0.0, nan]@Series[float64] | [18446744073709551615, 0, None]@Series[object] | [18446744073709551615, 0, None]@Series[object] [...]
| int64:all-null | [None, None]@int64 | [nan, nan]@Series[float64] | [None, None]@Series[object] | [None, None]@Series[object] [...]
-| int64:multi-chunk-nullable | [1, None, 2]@int64 | [1.0, nan, 2.0]@Series[float64] | [1, None, 2]@Series[object] | [1, None, 2]@Series[object] [...]
| list<int64>:null-element | [[1, None], [2, 3]]@list<item: int64> | [array([ 1., nan]), array([2., 3.])]@Series[object] | [array([1, None], dtype=object), array([2, 3], dtype=object)]@Series[object] | [array([1, None], dtype=object), array([2, 3], dtype=object)]@Series[object] [...]
| list<int64>:null-element-extreme | [[9223372036854775807, None]]@list<item: int64> | [array([9.22337204e+18, nan])]@Series[object] | [array([9223372036854775807, None], dtype=object)]@Series[object] | [array([9223372036854775807, None], dtype=object)]@Series[object] [...]
| large_list<int64>:null-element | [[1, None]]@large_list<item: int64> | [array([ 1., nan])]@Series[object] | [array([1, None], dtype=object)]@Series[object] | [array([1, None], dtype=object)]@Series[object] [...]
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv
index f2648eccf094..f587c3753a27 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv
@@ -120,7 +120,17 @@ map<string,map<string,int64>>:standard [[('a', [('x', 1)]), ('b', [('y', 2)])],
dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', 'a', 'b']@Series[category] ERR@ArrowInvalid copied
dictionary<int32,string>:nullable [a, b, None, a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', nan, 'a']@Series[category] ERR@ArrowInvalid copied
dictionary<int32,string>:empty []@dictionary<values=string, indices=int32, ordered=0> []@Series[category] ERR@ArrowInvalid copied
-int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64] [2, 3, 4]@Series[int64] zero-copy
-int64:sliced-with-null [2, None, 4]@int64 [2.0, nan, 4.0]@Series[float64] ERR@ArrowInvalid copied
+int64:zero-chunk []@int64 []@Series[int64] ERR@ArrowInvalid copied
+int64:empty-chunk []@int64 []@Series[int64] []@Series[int64] zero-copy
int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2, 3]@Series[int64] zero-copy
int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64] ERR@ArrowInvalid copied
+int64:multi-chunk-nullable [1, None, 2]@int64 [1.0, nan, 2.0]@Series[float64] ERR@ArrowInvalid copied
+int64:multi-chunk-with-empty [1, 2, 3]@int64 [1, 2, 3]@Series[int64] ERR@ArrowInvalid copied
+float64:multi-chunk [1.5, 2.5, 3.5]@float64 [1.5, 2.5, 3.5]@Series[float64] ERR@ArrowInvalid copied
+string:single-chunk [a, b]@string ['a', 'b']@Series[object] ERR@ArrowInvalid copied
+string:multi-chunk [a, b, c]@string ['a', 'b', 'c']@Series[object] ERR@ArrowInvalid copied
+string:multi-chunk-nullable [a, None, c]@string ['a', None, 'c']@Series[object] ERR@ArrowInvalid copied
+list<int64>:multi-chunk [[1, 2], [3], [4]]@list<item: int64> [array([1, 2]), array([3]), array([4])]@Series[object] ERR@ArrowInvalid copied
+struct:multi-chunk [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] ERR@ArrowInvalid copied
+int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64] [2, 3, 4]@Series[int64] zero-copy
+int64:sliced-with-null [2, None, 4]@int64 [2.0, nan, 4.0]@Series[float64] ERR@ArrowInvalid copied
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md
index b1954bdceb27..1091d1e5d3d6 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md
@@ -121,7 +121,17 @@
| dictionary<int32,string>:standard | [a, b, a, b]@dictionary<values=string, indices=int32, ordered=0> | ['a', 'b', 'a', 'b']@Series[category] | ERR@ArrowInvalid | copied |
| dictionary<int32,string>:nullable | [a, b, None, a]@dictionary<values=string, indices=int32, ordered=0> | ['a', 'b', nan, 'a']@Series[category] | ERR@ArrowInvalid | copied |
| dictionary<int32,string>:empty | []@dictionary<values=string, indices=int32, ordered=0> | []@Series[category] | ERR@ArrowInvalid | copied |
-| int64:sliced | [2, 3, 4]@int64 | [2, 3, 4]@Series[int64] | [2, 3, 4]@Series[int64] | zero-copy |
-| int64:sliced-with-null | [2, None, 4]@int64 | [2.0, nan, 4.0]@Series[float64] | ERR@ArrowInvalid | copied |
+| int64:zero-chunk | []@int64 | []@Series[int64] | ERR@ArrowInvalid | copied |
+| int64:empty-chunk | []@int64 | []@Series[int64] | []@Series[int64] | zero-copy |
| int64:single-chunk | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64] | [1, 2, 3]@Series[int64] | zero-copy |
-| int64:multi-chunk | [1, 2, 3, 4]@int64 | [1, 2, 3, 4]@Series[int64] | ERR@ArrowInvalid | copied |
\ No newline at end of file
+| int64:multi-chunk | [1, 2, 3, 4]@int64 | [1, 2, 3, 4]@Series[int64] | ERR@ArrowInvalid | copied |
+| int64:multi-chunk-nullable | [1, None, 2]@int64 | [1.0, nan, 2.0]@Series[float64] | ERR@ArrowInvalid | copied |
+| int64:multi-chunk-with-empty | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64] | ERR@ArrowInvalid | copied |
+| float64:multi-chunk | [1.5, 2.5, 3.5]@float64 | [1.5, 2.5, 3.5]@Series[float64] | ERR@ArrowInvalid | copied |
+| string:single-chunk | [a, b]@string | ['a', 'b']@Series[object] | ERR@ArrowInvalid | copied |
+| string:multi-chunk | [a, b, c]@string | ['a', 'b', 'c']@Series[object] | ERR@ArrowInvalid | copied |
+| string:multi-chunk-nullable | [a, None, c]@string | ['a', None, 'c']@Series[object] | ERR@ArrowInvalid | copied |
+| list<int64>:multi-chunk | [[1, 2], [3], [4]]@list<item: int64> | [array([1, 2]), array([3]), array([4])]@Series[object] | ERR@ArrowInvalid | copied |
+| struct:multi-chunk | [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] | ERR@ArrowInvalid | copied |
+| int64:sliced | [2, 3, 4]@int64 | [2, 3, 4]@Series[int64] | [2, 3, 4]@Series[int64] | zero-copy |
+| int64:sliced-with-null | [2, None, 4]@int64 | [2.0, nan, 4.0]@Series[float64] | ERR@ArrowInvalid | copied |
\ No newline at end of file
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
index 11f2831ca359..55a84ce933e3 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
@@ -120,7 +120,17 @@ map<string,map<string,int64>>:standard [[('a', [('x', 1)]), ('b', [('y', 2)])],
dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] ['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] zero-copy
dictionary<int32,string>:nullable [a, b, None, a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] ['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] zero-copy
dictionary<int32,string>:empty []@dictionary<values=string, indices=int32, ordered=0> []@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] []@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] zero-copy
-int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64[pyarrow]] [2, 3, 4]@Series[int64[pyarrow]] zero-copy
-int64:sliced-with-null [2, None, 4]@int64 [2, <NA>, 4]@Series[int64[pyarrow]] [2, <NA>, 4]@Series[int64[pyarrow]] zero-copy
+int64:zero-chunk []@int64 []@Series[int64[pyarrow]] []@Series[int64[pyarrow]] zero-copy
+int64:empty-chunk []@int64 []@Series[int64[pyarrow]] []@Series[int64[pyarrow]] zero-copy
int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64[pyarrow]] [1, 2, 3]@Series[int64[pyarrow]] zero-copy
int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64[pyarrow]] [1, 2, 3, 4]@Series[int64[pyarrow]] zero-copy
+int64:multi-chunk-nullable [1, None, 2]@int64 [1, <NA>, 2]@Series[int64[pyarrow]] [1, <NA>, 2]@Series[int64[pyarrow]] zero-copy
+int64:multi-chunk-with-empty [1, 2, 3]@int64 [1, 2, 3]@Series[int64[pyarrow]] [1, 2, 3]@Series[int64[pyarrow]] zero-copy
+float64:multi-chunk [1.5, 2.5, 3.5]@float64 [1.5, 2.5, 3.5]@Series[double[pyarrow]] [1.5, 2.5, 3.5]@Series[double[pyarrow]] zero-copy
+string:single-chunk [a, b]@string ['a', 'b']@Series[string[pyarrow]] ['a', 'b']@Series[string[pyarrow]] zero-copy
+string:multi-chunk [a, b, c]@string ['a', 'b', 'c']@Series[string[pyarrow]] ['a', 'b', 'c']@Series[string[pyarrow]] zero-copy
+string:multi-chunk-nullable [a, None, c]@string ['a', <NA>, 'c']@Series[string[pyarrow]] ['a', <NA>, 'c']@Series[string[pyarrow]] zero-copy
+list<int64>:multi-chunk [[1, 2], [3], [4]]@list<item: int64> [[1, 2], [3], [4]]@Series[list<item: int64>[pyarrow]] [[1, 2], [3], [4]]@Series[list<item: int64>[pyarrow]] zero-copy
+struct:multi-chunk [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]] [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]] zero-copy
+int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64[pyarrow]] [2, 3, 4]@Series[int64[pyarrow]] zero-copy
+int64:sliced-with-null [2, None, 4]@int64 [2, <NA>, 4]@Series[int64[pyarrow]] [2, <NA>, 4]@Series[int64[pyarrow]] zero-copy
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
index ac177461a2ed..daa25761c7b4 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
@@ -121,7 +121,17 @@
| dictionary<int32,string>:standard | [a, b, a, b]@dictionary<values=string, indices=int32, ordered=0> | ['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | ['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | zero-copy |
| dictionary<int32,string>:nullable | [a, b, None, a]@dictionary<values=string, indices=int32, ordered=0> | ['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | ['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | zero-copy |
| dictionary<int32,string>:empty | []@dictionary<values=string, indices=int32, ordered=0> | []@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | []@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]] | zero-copy |
-| int64:sliced | [2, 3, 4]@int64 | [2, 3, 4]@Series[int64[pyarrow]] | [2, 3, 4]@Series[int64[pyarrow]] | zero-copy |
-| int64:sliced-with-null | [2, None, 4]@int64 | [2, <NA>, 4]@Series[int64[pyarrow]] | [2, <NA>, 4]@Series[int64[pyarrow]] | zero-copy |
+| int64:zero-chunk | []@int64 | []@Series[int64[pyarrow]] | []@Series[int64[pyarrow]] | zero-copy |
+| int64:empty-chunk | []@int64 | []@Series[int64[pyarrow]] | []@Series[int64[pyarrow]] | zero-copy |
| int64:single-chunk | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64[pyarrow]] | [1, 2, 3]@Series[int64[pyarrow]] | zero-copy |
-| int64:multi-chunk | [1, 2, 3, 4]@int64 | [1, 2, 3, 4]@Series[int64[pyarrow]] | [1, 2, 3, 4]@Series[int64[pyarrow]] | zero-copy |
\ No newline at end of file
+| int64:multi-chunk | [1, 2, 3, 4]@int64 | [1, 2, 3, 4]@Series[int64[pyarrow]] | [1, 2, 3, 4]@Series[int64[pyarrow]] | zero-copy |
+| int64:multi-chunk-nullable | [1, None, 2]@int64 | [1, <NA>, 2]@Series[int64[pyarrow]] | [1, <NA>, 2]@Series[int64[pyarrow]] | zero-copy |
+| int64:multi-chunk-with-empty | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64[pyarrow]] | [1, 2, 3]@Series[int64[pyarrow]] | zero-copy |
+| float64:multi-chunk | [1.5, 2.5, 3.5]@float64 | [1.5, 2.5, 3.5]@Series[double[pyarrow]] | [1.5, 2.5, 3.5]@Series[double[pyarrow]] | zero-copy |
+| string:single-chunk | [a, b]@string | ['a', 'b']@Series[string[pyarrow]] | ['a', 'b']@Series[string[pyarrow]] | zero-copy |
+| string:multi-chunk | [a, b, c]@string | ['a', 'b', 'c']@Series[string[pyarrow]] | ['a', 'b', 'c']@Series[string[pyarrow]] | zero-copy |
+| string:multi-chunk-nullable | [a, None, c]@string | ['a', <NA>, 'c']@Series[string[pyarrow]] | ['a', <NA>, 'c']@Series[string[pyarrow]] | zero-copy |
+| list<int64>:multi-chunk | [[1, 2], [3], [4]]@list<item: int64> | [[1, 2], [3], [4]]@Series[list<item: int64>[pyarrow]] | [[1, 2], [3], [4]]@Series[list<item: int64>[pyarrow]] | zero-copy |
+| struct:multi-chunk | [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]] | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]] | zero-copy |
+| int64:sliced | [2, 3, 4]@int64 | [2, 3, 4]@Series[int64[pyarrow]] | [2, 3, 4]@Series[int64[pyarrow]] | zero-copy |
+| int64:sliced-with-null | [2, None, 4]@int64 | [2, <NA>, 4]@Series[int64[pyarrow]] | [2, <NA>, 4]@Series[int64[pyarrow]] | zero-copy |
\ No newline at end of file
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.csv b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.csv
new file mode 100644
index 000000000000..7ab114cfbf89
--- /dev/null
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.csv
@@ -0,0 +1,11 @@
+test case pyarrow array default spark memory options source readable after self_destruct
+int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2, 3]@Series[int64] readable
+int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64] [1, 2, 3, 4]@Series[int64] readable
+int64:multi-chunk-nullable [1, None, 2]@int64 [1.0, nan, 2.0]@Series[float64] [1.0, nan, 2.0]@Series[float64] readable
+int64:multi-chunk-with-empty [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2, 3]@Series[int64] readable
+int64:empty-chunk []@int64 []@Series[int64] []@Series[int64] readable
+float64:multi-chunk [1.5, 2.5, 3.5]@float64 [1.5, 2.5, 3.5]@Series[float64] [1.5, 2.5, 3.5]@Series[float64] readable
+string:multi-chunk [a, b, c]@string ['a', 'b', 'c']@Series[object] ['a', 'b', 'c']@Series[object] readable
+string:multi-chunk-nullable [a, None, c]@string ['a', None, 'c']@Series[object] ['a', None, 'c']@Series[object] readable
+list<int64>:multi-chunk [[1, 2], [3], [4]]@list<item: int64> [array([1, 2]), array([3]), array([4])]@Series[object] [array([1, 2]), array([3]), array([4])]@Series[object] readable
+struct:multi-chunk [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] readable
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.md b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.md
new file mode 100644
index 000000000000..e41a51201015
--- /dev/null
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.md
@@ -0,0 +1,12 @@
+| test case | pyarrow array | default | spark memory options | source readable after self_destruct |
+|------------------------------|------------------------------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------|---------------------------------------|
+| int64:single-chunk | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64] | [1, 2, 3]@Series[int64] | readable |
+| int64:multi-chunk | [1, 2, 3, 4]@int64 | [1, 2, 3, 4]@Series[int64] | [1, 2, 3, 4]@Series[int64] | readable |
+| int64:multi-chunk-nullable | [1, None, 2]@int64 | [1.0, nan, 2.0]@Series[float64] | [1.0, nan, 2.0]@Series[float64] | readable |
+| int64:multi-chunk-with-empty | [1, 2, 3]@int64 | [1, 2, 3]@Series[int64] | [1, 2, 3]@Series[int64] | readable |
+| int64:empty-chunk | []@int64 | []@Series[int64] | []@Series[int64] | readable |
+| float64:multi-chunk | [1.5, 2.5, 3.5]@float64 | [1.5, 2.5, 3.5]@Series[float64] | [1.5, 2.5, 3.5]@Series[float64] | readable |
+| string:multi-chunk | [a, b, c]@string | ['a', 'b', 'c']@Series[object] | ['a', 'b', 'c']@Series[object] | readable |
+| string:multi-chunk-nullable | [a, None, c]@string | ['a', None, 'c']@Series[object] | ['a', None, 'c']@Series[object] | readable |
+| list<int64>:multi-chunk | [[1, 2], [3], [4]]@list<item: int64> | [array([1, 2]), array([3]), array([4])]@Series[object] | [array([1, 2]), array([3]), array([4])]@Series[object] | readable |
+| struct:multi-chunk | [[('x', 1), ('y', 'a')], [('x', 2), ('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object] | readable |
\ No newline at end of file
diff --git a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
index a7e2b6e056d1..68c9c4e5de51 100644
--- a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
+++ b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
@@ -80,8 +80,9 @@ class _PyArrowToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
Shared machinery for pa.Array.to_pandas() golden file tests.
Holds the conversion helper and the source-array inventory, split into type-family
- group methods reusable whole (via ``_build_source_arrays``) or one at a time.
- Defines no ``test_*`` of its own.
+ group methods reusable whole (via ``_build_source_arrays``) or one at a time, plus a
+ ``_chunked_sources`` layout group of ChunkedArrays (chunk count is orthogonal to
+ type). Defines no ``test_*`` of its own.
"""
def _to_pandas_cell(self, arr, **to_pandas_kwargs) -> str:
@@ -358,6 +359,69 @@ class _PyArrowToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
return sources
+ def _chunked_sources(self):
+ """
+ ChunkedArray inputs covering the chunk-count axis.
+
+ pa.Array and pa.ChunkedArray share one ``to_pandas``, and Spark feeds a
+ ChunkedArray into it on the Table->columns path
+ (``python/pyspark/sql/pandas/conversion.py``): ``df.toPandas()`` builds one per
+ partition (no ``combine_chunks``), the empty-dataset branch uses
+ ``empty_table()`` (a single empty chunk), and the UDF paths ``combine_chunks()``
+ first (one chunk). Chunk count is orthogonal to type, so these rows reuse a few
+ representative types and vary only how the values are split.
+ """
+ sources = {}
+
+ # =====================================================================
+ # Numeric chunk layouts (int64: the type most Spark columns reduce to)
+ # =====================================================================
+ sources["int64:zero-chunk"] = pa.chunked_array([], type=pa.int64())
+ sources["int64:empty-chunk"] = pa.chunked_array([pa.array([], pa.int64())])
+ sources["int64:single-chunk"] = pa.chunked_array([pa.array([1, 2, 3], pa.int64())])
+ sources["int64:multi-chunk"] = pa.chunked_array(
+ [pa.array([1, 2], pa.int64()), pa.array([3, 4], pa.int64())]
+ )
+ # Null in one chunk, not the other. Reachable via cogrouped applyInPandas, the
+ # one UDF path that does not combine_chunks() first.
+ sources["int64:multi-chunk-nullable"] = pa.chunked_array(
+ [pa.array([1, None], pa.int64()), pa.array([2], pa.int64())]
+ )
+ # Empty chunk between two non-empty ones: concatenation must skip it cleanly.
+ sources["int64:multi-chunk-with-empty"] = pa.chunked_array(
+ [pa.array([1, 2], pa.int64()), pa.array([], pa.int64()), pa.array([3], pa.int64())]
+ )
+ sources["float64:multi-chunk"] = pa.chunked_array(
+ [pa.array([1.5, 2.5], pa.float64()), pa.array([3.5], pa.float64())]
+ )
+
+ # =====================================================================
+ # Variable-width chunk layouts (offset buffers, not just values)
+ # =====================================================================
+ sources["string:single-chunk"] = pa.chunked_array([pa.array(["a", "b"], pa.string())])
+ sources["string:multi-chunk"] = pa.chunked_array(
+ [pa.array(["a", "b"], pa.string()), pa.array(["c"], pa.string())]
+ )
+ sources["string:multi-chunk-nullable"] = pa.chunked_array(
+ [pa.array(["a", None], pa.string()), pa.array(["c"], pa.string())]
+ )
+
+ # =====================================================================
+ # Nested chunk layouts (child arrays split across chunks)
+ # =====================================================================
+ sources["list<int64>:multi-chunk"] = pa.chunked_array(
+ [pa.array([[1, 2], [3]], pa.list_(pa.int64())), pa.array([[4]], pa.list_(pa.int64()))]
+ )
+ struct_type = pa.struct([("x", pa.int64()), ("y", pa.string())])
+ sources["struct:multi-chunk"] = pa.chunked_array(
+ [
+ pa.array([{"x": 1, "y": "a"}], struct_type),
+ pa.array([{"x": 2, "y": "b"}], struct_type),
+ ]
+ )
+
+ return sources
+
def _build_source_arrays(self):
"""Build an ordered dict of named source PyArrow arrays for testing."""
sources = {}
@@ -367,6 +431,7 @@ class _PyArrowToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
self._temporal_sources(),
self._nested_sources(),
self._dictionary_sources(),
+ self._chunked_sources(),
]:
sources.update(group)
return sources
@@ -400,6 +465,9 @@ class PyArrowArrayToPandasDefaultTests(_PyArrowToPandasTestBase):
("string:nullable", "pandas series"): ("['hello', nan, 'world']@Series[str]"),
("large_string:standard", "pandas series"): ("['hello', 'world']@Series[str]"),
("large_string:nullable", "pandas series"): "['hello', nan]@Series[str]",
+ ("string:single-chunk", "pandas series"): "['a', 'b']@Series[str]",
+ ("string:multi-chunk", "pandas series"): "['a', 'b', 'c']@Series[str]",
+ ("string:multi-chunk-nullable", "pandas series"): "['a', nan, 'c']@Series[str]",
}
)
# PyArrow 24 extends the pandas string dtype conversion to empty arrays.
diff --git a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
index b619a443912b..84d711156f03 100644
--- a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
+++ b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
@@ -106,7 +106,7 @@ class PyArrowArrayToPandasCoerceTemporalTests(_PyArrowToPandasTestBase):
def _build_source_arrays(self):
"""
Reuse the base's temporal group (the types this argument targets), then add
- coercion-specific overflow rows and a few non-temporal controls.
+ coercion-specific overflow rows, chunked timestamps, and non-temporal controls.
"""
sources = self._temporal_sources()
@@ -122,6 +122,18 @@ class PyArrowArrayToPandasCoerceTemporalTests(_PyArrowToPandasTestBase):
[datetime.timedelta(days=300 * 365)], pa.duration("s")
)
+ # Chunked timestamps: the base's _chunked_sources has no temporal rows, so pin
+ # here that a chunked (non-ns) timestamp coerces to the same datetime64[ns] as
+ # the equivalent contiguous Array.
+ dt1 = datetime.datetime(2024, 1, 1, 12, 0, 0)
+ dt2 = datetime.datetime(2024, 6, 15, 18, 30, 0)
+ sources["timestamp[us]:single-chunk"] = pa.chunked_array(
+ [pa.array([dt1, dt2], pa.timestamp("us"))]
+ )
+ sources["timestamp[us]:multi-chunk"] = pa.chunked_array(
+ [pa.array([dt1], pa.timestamp("us")), pa.array([dt2], pa.timestamp("us"))]
+ )
+
# Non-temporal controls (unaffected by coerce_temporal_nanoseconds).
sources["int64:standard"] = pa.array([0, 1, -1], pa.int64())
sources["int64:nullable"] = pa.array([0, 1, None], pa.int64())
@@ -218,10 +230,10 @@ class PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
behind an assertion.
The row set mirrors ``test_pyarrow_arrow_to_pandas_default.py``'s rows exactly
- -- every Arrow type it covers, in its standard / nullable / empty variants --
- so both golden files pin the same types, and appends the layout variants that
- only matter for zero-copy: sliced (offset) arrays and single- vs multi-chunk
- ChunkedArrays.
+ -- every Arrow type it covers, in its standard / nullable / empty variants, plus
+ the shared ChunkedArray layouts -- so both golden files pin the same types, and
+ appends the one layout variant that only matters for zero-copy: sliced (offset)
+ arrays.
A second test method repeats those rows with ``types_mapper=pd.ArrowDtype``, which
asks pandas to keep pointing at the Arrow buffers instead of materializing them
@@ -233,25 +245,16 @@ class PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
def _build_source_arrays(self):
"""
- Reuse the base's full inventory, then add the layout variants that only
- matter for zero-copy.
-
- Sharing the row set keeps this and the default golden pinning the same types,
- and a type added to the base is covered here automatically.
- A slice still views a contiguous no-null region, so it stays zero-copy even
- though its data starts partway into the parent buffer. A single chunk is
- zero-copy, but multiple chunks must be concatenated into one contiguous
- numpy buffer -- a copy. Multi-chunk is the common shape in real PySpark,
- where each partition arrives as its own chunk.
+ Reuse the base's full inventory, then add the sliced layout variants that only
+ matter for zero-copy: a slice views a contiguous no-null region, so it stays
+ zero-copy even though its data starts partway into the parent buffer. The
+ chunk-count variants live in the base's ``_chunked_sources`` so every golden
+ records their zero-copy behavior, not just this one.
"""
sources = super()._build_source_arrays()
sources["int64:sliced"] = pa.array(list(range(10)), pa.int64()).slice(2, 3)
sources["int64:sliced-with-null"] = pa.array([1, 2, None, 4, 5], pa.int64()).slice(1, 3)
- sources["int64:single-chunk"] = pa.chunked_array([pa.array([1, 2, 3], pa.int64())])
- sources["int64:multi-chunk"] = pa.chunked_array(
- [pa.array([1, 2], pa.int64()), pa.array([3, 4], pa.int64())]
- )
return sources
@@ -371,6 +374,9 @@ class PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
("string:nullable", "['hello', nan, 'world']@Series[str]", "partial-copy"),
("large_string:standard", "['hello', 'world']@Series[str]", "zero-copy"),
("large_string:nullable", "['hello', nan]@Series[str]", "zero-copy"),
+ ("string:single-chunk", "['a', 'b']@Series[str]", "partial-copy"),
+ ("string:multi-chunk", "['a', 'b', 'c']@Series[str]", "partial-copy"),
+ ("string:multi-chunk-nullable", "['a', nan, 'c']@Series[str]", "partial-copy"),
]
for row, expected, _ in non_empty_strings:
overrides[(row, self.COL_ZERO_COPY_OFF)] = expected
@@ -519,12 +525,6 @@ class PyArrowArrayToPandasIntegerObjectNullsTests(_PyArrowToPandasTestBase):
# Every value is null, so there is no integer left to convert.
sources["int64:all-null"] = pa.array([None, None], pa.int64())
- # Values split across buffers, with the null in one chunk and not the other.
- # Reachable in production: cogrouped applyInPandas is the one UDF path that does
- # not call combine_chunks() first.
- sources["int64:multi-chunk-nullable"] = pa.chunked_array(
- [pa.array([1, None], pa.int64()), pa.array([2], pa.int64())]
- )
# Nested types whose null is an integer ELEMENT, not a missing sub-list: the
# shared rows only cover the latter, which this argument does not affect.
@@ -600,6 +600,9 @@ class PyArrowArrayToPandasIntegerObjectNullsTests(_PyArrowToPandasTestBase):
override_outputs("string:nullable", "['hello', nan, 'world']@Series[str]")
override_outputs("large_string:standard", "['hello', 'world']@Series[str]")
override_outputs("large_string:nullable", "['hello', nan]@Series[str]")
+ override_outputs("string:single-chunk", "['a', 'b']@Series[str]")
+ override_outputs("string:multi-chunk", "['a', 'b', 'c']@Series[str]")
+ override_outputs("string:multi-chunk-nullable", "['a', nan, 'c']@Series[str]")
# Empty ones stay object until PyArrow 24, so the baseline holds before that.
# Spark supports PyArrow 18+, so both branches are reachable.
@@ -630,6 +633,152 @@ class PyArrowArrayToPandasIntegerObjectNullsTests(_PyArrowToPandasTestBase):
)
+@unittest.skipIf(
+ not have_pyarrow or not have_pandas or not have_numpy,
+ pyarrow_requirement_message or pandas_requirement_message or numpy_requirement_message,
+)
+class PyArrowChunkedArrayToPandasMemoryFlagsTests(_PyArrowToPandasTestBase):
+ """
+ Tests pa.ChunkedArray.to_pandas() under the memory-tuning arguments
+ ``self_destruct`` / ``split_blocks`` / ``use_threads`` via golden file comparison.
+
+ Spark sets all three together for ``df.toPandas()`` when Arrow self-destruct is
+ enabled (``python/pyspark/sql/pandas/conversion.py``), freeing each column's buffers
+ as it is converted to keep peak memory near one column. They form one unit --
+ freeing (``self_destruct``) needs per-column blocks (``split_blocks``) and
+ single-threaded conversion (``use_threads=False``) -- so this golden records the
+ bundle (``spark memory options``) against ``default`` rather than a column per flag.
+ The path always converts a ``pa.ChunkedArray`` (``Table.column(i)``), hence the rows.
+
+ The arguments tune HOW the conversion runs, not WHAT it produces, so the bundle is
+ expected to match the default. Pinning that equivalence catches drift: if these
+ arguments ever alter the output -- or ``self_destruct`` actually consumes the input,
+ which today it does not at this level -- a cell moves. A final column records whether
+ the source is still readable after ``self_destruct=True``, since Spark treats freeing
+ as an optional optimization, not a contract. Each cell rebuilds its source from a
+ factory so a destructive ``self_destruct`` cannot corrupt another cell.
+ """
+
+ def _memory_flag_source_factories(self):
+ """
+ Named factories, each returning a FRESH ChunkedArray per call, since
+ ``self_destruct=True`` may consume its input. Rows span the chunk-count axis
+ across numeric, variable-width, and nested types (buffer layout differs by type).
+ """
+ struct_type = pa.struct([("x", pa.int64()), ("y", pa.string())])
+ return {
+ "int64:single-chunk": lambda: pa.chunked_array([pa.array([1, 2, 3], pa.int64())]),
+ "int64:multi-chunk": lambda: pa.chunked_array(
+ [pa.array([1, 2], pa.int64()), pa.array([3, 4], pa.int64())]
+ ),
+ "int64:multi-chunk-nullable": lambda: pa.chunked_array(
+ [pa.array([1, None], pa.int64()), pa.array([2], pa.int64())]
+ ),
+ "int64:multi-chunk-with-empty": lambda: pa.chunked_array(
+ [pa.array([1, 2], pa.int64()), pa.array([], pa.int64()), pa.array([3], pa.int64())]
+ ),
+ "int64:empty-chunk": lambda: pa.chunked_array([pa.array([], pa.int64())]),
+ "float64:multi-chunk": lambda: pa.chunked_array(
+ [pa.array([1.5, 2.5], pa.float64()), pa.array([3.5], pa.float64())]
+ ),
+ "string:multi-chunk": lambda: pa.chunked_array(
+ [pa.array(["a", "b"], pa.string()), pa.array(["c"], pa.string())]
+ ),
+ "string:multi-chunk-nullable": lambda: pa.chunked_array(
+ [pa.array(["a", None], pa.string()), pa.array(["c"], pa.string())]
+ ),
+ "list<int64>:multi-chunk": lambda: pa.chunked_array(
+ [
+ pa.array([[1, 2], [3]], pa.list_(pa.int64())),
+ pa.array([[4]], pa.list_(pa.int64())),
+ ]
+ ),
+ "struct:multi-chunk": lambda: pa.chunked_array(
+ [
+ pa.array([{"x": 1, "y": "a"}], struct_type),
+ pa.array([{"x": 2, "y": "b"}], struct_type),
+ ]
+ ),
+ }
+
+ # Default (no memory arguments) -- the baseline the bundle should match.
+ COL_DEFAULT = "default"
+ # All three flags as convert_arrow_table_to_pandas passes them together.
+ COL_SPARK_MEMORY_OPTIONS = "spark memory options"
+ # Whether the source ChunkedArray is still readable after self_destruct=True.
+ COL_SOURCE_READABLE = "source readable after self_destruct"
+
+ # Kept as one dict so the column cannot drift from the call site it mirrors
+ # (python/pyspark/sql/pandas/conversion.py).
+ SPARK_MEMORY_OPTIONS = {
+ "self_destruct": True,
+ "split_blocks": True,
+ "use_threads": False,
+ }
+
+ def _source_readable_after_self_destruct(self, factory) -> str:
+ """
+ Convert a fresh source with ``self_destruct=True``, then report whether it is
+ still readable: ``readable``, or ``unreadable@<ExceptionClass>`` if freed.
+ """
+ arr = factory()
+ try:
+ arr.to_pandas(self_destruct=True)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+ try:
+ arr.to_pylist()
+ return "readable"
+ except Exception as e:
+ return f"unreadable@{type(e).__name__}"
+
+ def test_to_pandas_memory_flags(self):
+ """Test pa.ChunkedArray.to_pandas() under the memory-tuning arguments."""
+ factories = self._memory_flag_source_factories()
+ row_names = list(factories.keys())
+ col_names = [
+ "pyarrow array",
+ self.COL_DEFAULT,
+ self.COL_SPARK_MEMORY_OPTIONS,
+ self.COL_SOURCE_READABLE,
+ ]
+
+ # Version-specific expected values go here, keyed by (row, col), when a newer
+ # pandas/PyArrow/NumPy legitimately changes a cell's output. These arguments do
+ # not touch strings, so a string row shifts in both value columns at once.
+ overrides: dict[tuple[str, str], str] = {}
+ if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+ value_cols = [self.COL_DEFAULT, self.COL_SPARK_MEMORY_OPTIONS]
+ for row, expected in [
+ ("string:multi-chunk", "['a', 'b', 'c']@Series[str]"),
+ ("string:multi-chunk-nullable", "['a', nan, 'c']@Series[str]"),
+ ]:
+ for col in value_cols:
+ overrides[(row, col)] = expected
+
+ def compute_cell(row_name, col_name):
+ factory = factories[row_name]
+ if col_name == "pyarrow array":
+ return self.repr_value(factory(), max_len=0)
+ elif col_name == self.COL_DEFAULT:
+ return self._to_pandas_cell(factory())
+ elif col_name == self.COL_SPARK_MEMORY_OPTIONS:
+ return self._to_pandas_cell(factory(), **self.SPARK_MEMORY_OPTIONS)
+ elif col_name == self.COL_SOURCE_READABLE:
+ return self._source_readable_after_self_destruct(factory)
+ else:
+ raise ValueError(f"unknown column: {col_name}")
+
+ self.compare_or_generate_golden_matrix(
+ row_names=row_names,
+ col_names=col_names,
+ compute_cell=compute_cell,
+ golden_file_prefix="golden_pyarrow_chunked_array_to_pandas_memory_flags",
+ index_name="test case",
+ overrides=overrides,
+ )
+
+
if __name__ == "__main__":
from pyspark.testing import main
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org