You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/08/10 04:09:07 UTC
(spark) branch master updated: [SPARK-58656][PYTHON][TEST] Centralize the to_pandas golden test inventory into a shared base
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 661188e8fd26 [SPARK-58656][PYTHON][TEST] Centralize the to_pandas golden test inventory into a shared base
661188e8fd26 is described below
commit 661188e8fd26924a3fd44acb281b1f82066ea71b
Author: Spenser Sun <hs...@gmail.com>
AuthorDate: Mon Aug 10 12:08:54 2026 +0800
[SPARK-58656][PYTHON][TEST] Centralize the to_pandas golden test inventory into a shared base
### What changes were proposed in this pull request?
This is a **pure refactor** of the existing `pa.Array.to_pandas` golden tests, done first so that `pa.ChunkedArray.to_pandas` coverage can be added cleanly in a follow-up. It adds no new test cases and changes no behavior; it only reorganizes the shared machinery.
- Move `_PyArrowToPandasTestBase` into `test_pyarrow_arrow_to_pandas_default.py` and have `PyArrowArrayToPandasDefaultTests` extend it, so the base class is the single home for the machinery both test files share.
- Split the source-array inventory into type-family group methods on the base -- `_numeric_sources`, `_string_binary_sources`, `_temporal_sources`, `_nested_sources`, `_dictionary_sources` -- unioned by `_build_source_arrays`. A concrete test can now reuse the whole inventory or one group.
- `PyArrowArrayToPandasCoerceTemporalTests` builds its rows from `_temporal_sources()` instead of rewriting the temporal types, removing the duplicated definitions. Its golden file is regenerated: **same rows and same cell values, only the row order changes** to match the shared temporal group.
- The default test class reuses the base's `_to_pandas_cell` instead of reimplementing the conversion inline.
- Move `_verify_zero_copy` / `_arrow_buffers` (used only by the zero-copy tests) out of the base and into `PyArrowArrayToPandasZeroCopyTests`, so no class inherits helpers it does not use.
- The non-default file now imports the shared base directly rather than importing the whole default module to avoid test re-collection; a base with no `test_*` methods is not collected.
Class hierarchy after this change:
```
GoldenFileTestMixin
<- _PyArrowToPandasTestBase (in ..._default.py; _to_pandas_cell + 5 group methods + _build_source_arrays; no test_*)
<- PyArrowArrayToPandasDefaultTests (..._default.py)
<- PyArrowArrayToPandasCoerceTemporalTests (..._non_default.py; uses _temporal_sources())
<- PyArrowArrayToPandasZeroCopyTests (..._non_default.py; owns _verify_zero_copy / _arrow_buffers)
<- PyArrowArrayToPandasIntegerObjectNullsTests (..._non_default.py)
```
### Why are the changes needed?
`pa.ChunkedArray.to_pandas` shares `to_pandas`'s entire keyword surface and returns a Series, exactly like `pa.Array.to_pandas`, so its tests belong in these same files. Adding them requires the source inventory to be reusable by group (the ChunkedArray rows extend the existing ones) and the shared machinery to live in one place. Doing this reorganization on its own keeps the follow-up focused purely on the new coverage, and keeps this diff easy to verify as behavior-preserving.
I will add the `pa.ChunkedArray.to_pandas` tests in a follow-up PR after this one is merged.
### Does this PR introduce _any_ user-facing change?
No. Test-only refactor.
### How was this patch tested?
- The four to_pandas golden files were regenerated. The `default`, `zero_copy`, `zero_copy_arrow_backed`, and `integer_object_nulls` goldens are **byte-identical**; the `coerce_temporal` golden differs **only in row order** (same rows, same cell values, confirmed by a sorted diff).
- The full to_pandas test suite passes without `SPARK_GENERATE_GOLDEN_FILES`, and passes across a sweep of pyarrow 18/19/20/21/22/23/24/25 x pandas 2 and 3 (16/16).
- `dev/lint-python --ruff` is clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57861 from Spenserrrr/chunkedarray-to-pandas-tests.
Authored-by: Spenser Sun <hs...@gmail.com>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
...den_pyarrow_arrow_to_pandas_coerce_temporal.csv | 16 +-
...lden_pyarrow_arrow_to_pandas_coerce_temporal.md | 16 +-
.../test_pyarrow_arrow_to_pandas_default.py | 92 +++++--
.../test_pyarrow_arrow_to_pandas_non_default.py | 295 ++++++++-------------
4 files changed, 195 insertions(+), 224 deletions(-)
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
index d4a8df1a2a62..4f8439cd1d2e 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
@@ -1,4 +1,10 @@
test case pyarrow array pandas series pandas series (date_as_object=False)
+date32:standard [2024-01-01, 2024-06-15]@date32[day] [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object] [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15 00:00:00')]@Series[datetime64[ns]]
+date32:nullable [2024-01-01, None]@date32[day] [datetime.date(2024, 1, 1), None]@Series[object] [Timestamp('2024-01-01 00:00:00'), NaT]@Series[datetime64[ns]]
+date32:empty []@date32[day] []@Series[object] []@Series[datetime64[ns]]
+date64:standard [2024-01-01, 2024-06-15]@date64[ms] [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object] [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15 00:00:00')]@Series[datetime64[ns]]
+date64:nullable [2024-01-01, None]@date64[ms] [datetime.date(2024, 1, 1), None]@Series[object] [Timestamp('2024-01-01 00:00:00'), NaT]@Series[datetime64[ns]]
+date64:empty []@date64[ms] []@Series[object] []@Series[datetime64[ns]]
timestamp[s]:standard [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[s] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]] [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
timestamp[s]:nullable [2024-01-01 12:00:00, None]@timestamp[s] [Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]] [Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
timestamp[s]:empty []@timestamp[s] []@Series[datetime64[ns]] []@Series[datetime64[ns]]
@@ -14,7 +20,6 @@ timestamp[ns]:empty []@timestamp[ns] []@Series[datetime64[ns]] []@Series[datetim
timestamp[us,tz=UTC]:standard [2024-01-01 12:00:00+00:00, 2024-06-15 18:30:00+00:00]@timestamp[us, tz=UTC] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15 18:30:00+0000', tz='UTC')]@Series[datetime64[ns, UTC]] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15 18:30:00+0000', tz='UTC')]@Series[datetime64[ns, UTC]]
timestamp[us,tz=UTC]:nullable [2024-01-01 12:00:00+00:00, None]@timestamp[us, tz=UTC] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), NaT]@Series[datetime64[ns, UTC]] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), NaT]@Series[datetime64[ns, UTC]]
timestamp[us,tz=UTC]:empty []@timestamp[us, tz=UTC] []@Series[datetime64[ns, UTC]] []@Series[datetime64[ns, UTC]]
-timestamp[s]:overflow [2500-01-01 00:00:00]@timestamp[s] ERR@ArrowInvalid ERR@ArrowInvalid
duration[s]:standard [1 day, 0:00:00, 2:30:00]@duration[s] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]]
duration[s]:nullable [1 day, 0:00:00, None]@duration[s] [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]] [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]]
duration[s]:empty []@duration[s] []@Series[timedelta64[ns]] []@Series[timedelta64[ns]]
@@ -27,13 +32,6 @@ duration[us]:empty []@duration[us] []@Series[timedelta64[ns]] []@Series[timedelt
duration[ns]:standard [1 days 00:00:00, 0 days 02:30:00]@duration[ns] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]] [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]]
duration[ns]:nullable [1 days 00:00:00, None]@duration[ns] [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]] [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]]
duration[ns]:empty []@duration[ns] []@Series[timedelta64[ns]] []@Series[timedelta64[ns]]
-duration[s]:overflow [109500 days, 0:00:00]@duration[s] [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]] [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
-date32:standard [2024-01-01, 2024-06-15]@date32[day] [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object] [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15 00:00:00')]@Series[datetime64[ns]]
-date32:nullable [2024-01-01, None]@date32[day] [datetime.date(2024, 1, 1), None]@Series[object] [Timestamp('2024-01-01 00:00:00'), NaT]@Series[datetime64[ns]]
-date32:empty []@date32[day] []@Series[object] []@Series[datetime64[ns]]
-date64:standard [2024-01-01, 2024-06-15]@date64[ms] [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object] [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15 00:00:00')]@Series[datetime64[ns]]
-date64:nullable [2024-01-01, None]@date64[ms] [datetime.date(2024, 1, 1), None]@Series[object] [Timestamp('2024-01-01 00:00:00'), NaT]@Series[datetime64[ns]]
-date64:empty []@date64[ms] []@Series[object] []@Series[datetime64[ns]]
time32[s]:standard [12:30:00, 18:45:30]@time32[s] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object]
time32[s]:nullable [12:30:00, None]@time32[s] [datetime.time(12, 30), None]@Series[object] [datetime.time(12, 30), None]@Series[object]
time32[s]:empty []@time32[s] []@Series[object] []@Series[object]
@@ -46,6 +44,8 @@ time64[us]:empty []@time64[us] []@Series[object] []@Series[object]
time64[ns]:standard [12:30:00, 18:45:30]@time64[ns] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object] [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object]
time64[ns]:nullable [12:30:00, None]@time64[ns] [datetime.time(12, 30), None]@Series[object] [datetime.time(12, 30), None]@Series[object]
time64[ns]:empty []@time64[ns] []@Series[object] []@Series[object]
+timestamp[s]:overflow [2500-01-01 00:00:00]@timestamp[s] ERR@ArrowInvalid ERR@ArrowInvalid
+duration[s]:overflow [109500 days, 0:00:00]@duration[s] [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]] [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
int64:standard [0, 1, -1]@int64 [0, 1, -1]@Series[int64] [0, 1, -1]@Series[int64]
int64:nullable [0, 1, None]@int64 [0.0, 1.0, nan]@Series[float64] [0.0, 1.0, nan]@Series[float64]
float64:standard [0.0, 1.5, -1.5]@float64 [0.0, 1.5, -1.5]@Series[float64] [0.0, 1.5, -1.5]@Series[float64]
diff --git a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
index 45eec337f059..e6b3a061c97d 100644
--- a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
@@ -1,5 +1,11 @@
| test case | pyarrow array | pandas series | pandas series (date_as_object=False) |
|-------------------------------|------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
+| date32:standard | [2024-01-01, 2024-06-15]@date32[day] | [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object] | [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15 00:00:00')]@Series[datetime64[ns]] |
+| date32:nullable | [2024-01-01, None]@date32[day] | [datetime.date(2024, 1, 1), None]@Series[object] | [Timestamp('2024-01-01 00:00:00'), NaT]@Series[datetime64[ns]] |
+| date32:empty | []@date32[day] | []@Series[object] | []@Series[datetime64[ns]] |
+| date64:standard | [2024-01-01, 2024-06-15]@date64[ms] | [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object] | [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15 00:00:00')]@Series[datetime64[ns]] |
+| date64:nullable | [2024-01-01, None]@date64[ms] | [datetime.date(2024, 1, 1), None]@Series[object] | [Timestamp('2024-01-01 00:00:00'), NaT]@Series[datetime64[ns]] |
+| date64:empty | []@date64[ms] | []@Series[object] | []@Series[datetime64[ns]] |
| timestamp[s]:standard | [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[s] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]] | [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]] |
| timestamp[s]:nullable | [2024-01-01 12:00:00, None]@timestamp[s] | [Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]] | [Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]] |
| timestamp[s]:empty | []@timestamp[s] | []@Series[datetime64[ns]] | []@Series[datetime64[ns]] |
@@ -15,7 +21,6 @@
| timestamp[us,tz=UTC]:standard | [2024-01-01 12:00:00+00:00, 2024-06-15 18:30:00+00:00]@timestamp[us, tz=UTC] | [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15 18:30:00+0000', tz='UTC')]@Series[datetime64[ns, UTC]] | [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15 18:30:00+0000', tz='UTC')]@Series[datetime64[ns, UTC]] |
| timestamp[us,tz=UTC]:nullable | [2024-01-01 12:00:00+00:00, None]@timestamp[us, tz=UTC] | [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), NaT]@Series[datetime64[ns, UTC]] | [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), NaT]@Series[datetime64[ns, UTC]] |
| timestamp[us,tz=UTC]:empty | []@timestamp[us, tz=UTC] | []@Series[datetime64[ns, UTC]] | []@Series[datetime64[ns, UTC]] |
-| timestamp[s]:overflow | [2500-01-01 00:00:00]@timestamp[s] | ERR@ArrowInvalid | ERR@ArrowInvalid |
| duration[s]:standard | [1 day, 0:00:00, 2:30:00]@duration[s] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]] |
| duration[s]:nullable | [1 day, 0:00:00, None]@duration[s] | [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]] | [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]] |
| duration[s]:empty | []@duration[s] | []@Series[timedelta64[ns]] | []@Series[timedelta64[ns]] |
@@ -28,13 +33,6 @@
| duration[ns]:standard | [1 days 00:00:00, 0 days 02:30:00]@duration[ns] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]] | [Timedelta('1 days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]] |
| duration[ns]:nullable | [1 days 00:00:00, None]@duration[ns] | [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]] | [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]] |
| duration[ns]:empty | []@duration[ns] | []@Series[timedelta64[ns]] | []@Series[timedelta64[ns]] |
-| duration[s]:overflow | [109500 days, 0:00:00]@duration[s] | [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]] | [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]] |
-| date32:standard | [2024-01-01, 2024-06-15]@date32[day] | [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object] | [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15 00:00:00')]@Series[datetime64[ns]] |
-| date32:nullable | [2024-01-01, None]@date32[day] | [datetime.date(2024, 1, 1), None]@Series[object] | [Timestamp('2024-01-01 00:00:00'), NaT]@Series[datetime64[ns]] |
-| date32:empty | []@date32[day] | []@Series[object] | []@Series[datetime64[ns]] |
-| date64:standard | [2024-01-01, 2024-06-15]@date64[ms] | [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object] | [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15 00:00:00')]@Series[datetime64[ns]] |
-| date64:nullable | [2024-01-01, None]@date64[ms] | [datetime.date(2024, 1, 1), None]@Series[object] | [Timestamp('2024-01-01 00:00:00'), NaT]@Series[datetime64[ns]] |
-| date64:empty | []@date64[ms] | []@Series[object] | []@Series[datetime64[ns]] |
| time32[s]:standard | [12:30:00, 18:45:30]@time32[s] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object] |
| time32[s]:nullable | [12:30:00, None]@time32[s] | [datetime.time(12, 30), None]@Series[object] | [datetime.time(12, 30), None]@Series[object] |
| time32[s]:empty | []@time32[s] | []@Series[object] | []@Series[object] |
@@ -47,6 +45,8 @@
| time64[ns]:standard | [12:30:00, 18:45:30]@time64[ns] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object] | [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object] |
| time64[ns]:nullable | [12:30:00, None]@time64[ns] | [datetime.time(12, 30), None]@Series[object] | [datetime.time(12, 30), None]@Series[object] |
| time64[ns]:empty | []@time64[ns] | []@Series[object] | []@Series[object] |
+| timestamp[s]:overflow | [2500-01-01 00:00:00]@timestamp[s] | ERR@ArrowInvalid | ERR@ArrowInvalid |
+| duration[s]:overflow | [109500 days, 0:00:00]@duration[s] | [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]] | [Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]] |
| int64:standard | [0, 1, -1]@int64 | [0, 1, -1]@Series[int64] | [0, 1, -1]@Series[int64] |
| int64:nullable | [0, 1, None]@int64 | [0.0, 1.0, nan]@Series[float64] | [0.0, 1.0, nan]@Series[float64] |
| float64:standard | [0.0, 1.5, -1.5]@float64 | [0.0, 1.5, -1.5]@Series[float64] | [0.0, 1.5, -1.5]@Series[float64] |
diff --git a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
index e3a9a0c0f5bd..a7e2b6e056d1 100644
--- a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
+++ b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
@@ -27,6 +27,10 @@ arguments (no types_mapper, no self_destruct, etc.), tracking:
- How null values are handled (NaN, None, NaT, etc.)
- Whether values are preserved correctly after conversion
+The shared ``_PyArrowToPandasTestBase`` holds the conversion helper and the source-array
+inventory, split into type-family group methods that these and the non-default tests
+reuse whole or by group.
+
## Golden File Cell Format
Each cell uses the value@type format:
@@ -71,23 +75,28 @@ if have_pyarrow:
import pyarrow as pa
-@unittest.skipIf(
- not have_pyarrow or not have_pandas or not have_numpy,
- pyarrow_requirement_message or pandas_requirement_message or numpy_requirement_message,
-)
-class PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
+class _PyArrowToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
"""
- Tests pa.Array.to_pandas() with default arguments via golden file comparison.
+ Shared machinery for pa.Array.to_pandas() golden file tests.
- Covers all major Arrow types: integers, floats, bool, string, binary,
- decimal, date, timestamp, duration, time, null, and nested types.
- Each type is tested both without and with null values.
+ Holds the conversion helper and the source-array inventory, split into type-family
+ group methods reusable whole (via ``_build_source_arrays``) or one at a time.
+ Defines no ``test_*`` of its own.
"""
- def _build_source_arrays(self):
- """Build an ordered dict of named source PyArrow arrays for testing."""
- import pyarrow as pa
-
+ def _to_pandas_cell(self, arr, **to_pandas_kwargs) -> str:
+ """
+ Convert ``arr`` via ``to_pandas(**to_pandas_kwargs)`` and format the
+ result as a golden-file cell, returning ``ERR@<ExceptionClass>`` if the
+ conversion raises.
+ """
+ try:
+ return self.repr_value(arr.to_pandas(**to_pandas_kwargs), max_len=0)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+
+ def _numeric_sources(self):
+ """Integer, float, and boolean arrays."""
sources = {}
# =====================================================================
@@ -136,6 +145,12 @@ class PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
sources["bool:nullable"] = pa.array([True, False, None], pa.bool_())
sources["bool:empty"] = pa.array([], pa.bool_())
+ return sources
+
+ def _string_binary_sources(self):
+ """String, binary, and decimal arrays."""
+ sources = {}
+
# =====================================================================
# String types
# =====================================================================
@@ -168,6 +183,12 @@ class PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
)
sources["decimal128:empty"] = pa.array([], pa.decimal128(5, 2))
+ return sources
+
+ def _temporal_sources(self):
+ """Date, timestamp, duration, and time arrays."""
+ sources = {}
+
# =====================================================================
# Date types
# =====================================================================
@@ -226,6 +247,12 @@ class PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
sources["time64[ns]:nullable"] = pa.array([t1, None], pa.time64("ns"))
sources["time64[ns]:empty"] = pa.array([], pa.time64("ns"))
+ return sources
+
+ def _nested_sources(self):
+ """Null, list, struct, map, and their nested combinations."""
+ sources = {}
+
# =====================================================================
# Null type
# =====================================================================
@@ -307,6 +334,12 @@ class PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
pa.map_(pa.string(), pa.map_(pa.string(), pa.int64())),
)
+ return sources
+
+ def _dictionary_sources(self):
+ """Dictionary-encoded arrays."""
+ sources = {}
+
# =====================================================================
# Dictionary type
# =====================================================================
@@ -325,6 +358,33 @@ class PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
return sources
+ def _build_source_arrays(self):
+ """Build an ordered dict of named source PyArrow arrays for testing."""
+ sources = {}
+ for group in [
+ self._numeric_sources(),
+ self._string_binary_sources(),
+ self._temporal_sources(),
+ self._nested_sources(),
+ self._dictionary_sources(),
+ ]:
+ sources.update(group)
+ return sources
+
+
+@unittest.skipIf(
+ not have_pyarrow or not have_pandas or not have_numpy,
+ pyarrow_requirement_message or pandas_requirement_message or numpy_requirement_message,
+)
+class PyArrowArrayToPandasDefaultTests(_PyArrowToPandasTestBase):
+ """
+ Tests pa.Array.to_pandas() with default arguments via golden file comparison.
+
+ Covers all major Arrow types: integers, floats, bool, string, binary,
+ decimal, date, timestamp, duration, time, null, and nested types.
+ Each type is tested both without and with null values.
+ """
+
def test_to_pandas_default(self):
"""Test pa.Array.to_pandas() with default arguments against golden file."""
sources = self._build_source_arrays()
@@ -356,11 +416,7 @@ class PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
if col_name == "pyarrow array":
return self.repr_value(arr, max_len=0)
else:
- try:
- result = arr.to_pandas()
- return self.repr_value(result, max_len=0)
- except Exception as e:
- return f"ERR@{type(e).__name__}"
+ return self._to_pandas_cell(arr)
self.compare_or_generate_golden_matrix(
row_names=row_names,
diff --git a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
index 7e0dc3a0823a..b619a443912b 100644
--- a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
+++ b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
@@ -28,12 +28,10 @@ non-Arrow path. These tests record how each Arrow type behaves under those
arguments so CI fails loudly if the behavior drifts across pandas/PyArrow/NumPy
upgrades.
-The shared ``_PyArrowToPandasTestBase`` holds the golden-file matrix driver and
-the per-cell conversion helper. Each concrete test class supplies its own source
-arrays (the rows relevant to the argument under test) and one test method per
-argument combination (each producing its own golden file). New ``to_pandas``
-arguments (e.g. ``zero_copy_only``, ``integer_object_nulls``) can be added as
-additional classes without touching the existing ones.
+The shared ``_PyArrowToPandasTestBase`` (in ``test_pyarrow_arrow_to_pandas_default.py``)
+holds the conversion helper and the source-array inventory. Each class reuses the
+inventory whole or by group and adds a test method per argument combination (its own
+golden file), so a new argument is a new class rather than an edit to the existing ones.
## Golden File Cell Format
@@ -71,11 +69,12 @@ from pyspark.testing.utils import (
pandas_requirement_message,
numpy_requirement_message,
)
-from pyspark.testing.goldenutils import GoldenFileTestMixin
-# Imported as a module, not `from ... import PyArrowArrayToPandasDefaultTests`, so
-# that unittest does not collect and re-run the default test class from this module.
-from pyspark.tests.upstream.pyarrow import test_pyarrow_arrow_to_pandas_default
+# Import the shared base (which defines no test_* methods), not a concrete test class,
+# so that unittest does not collect and re-run the default file's tests here.
+from pyspark.tests.upstream.pyarrow.test_pyarrow_arrow_to_pandas_default import (
+ _PyArrowToPandasTestBase,
+)
if have_pandas:
import pandas as pd
@@ -85,107 +84,6 @@ if have_numpy:
import numpy as np
-class _PyArrowToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
- """
- Shared machinery for pa.Array.to_pandas() golden file tests.
-
- Concrete subclasses provide their own ``_build_source_arrays`` (the rows) and
- one or more ``test_*`` methods that call ``compare_or_generate_golden_matrix``.
- This base defines no ``test_*`` methods, so it contributes no tests itself.
- """
-
- def _to_pandas_cell(self, arr, **to_pandas_kwargs) -> str:
- """
- Convert ``arr`` via ``to_pandas(**to_pandas_kwargs)`` and format the
- result as a golden-file cell, returning ``ERR@<ExceptionClass>`` if the
- conversion raises.
- """
- try:
- return self.repr_value(arr.to_pandas(**to_pandas_kwargs), max_len=0)
- except Exception as e:
- return f"ERR@{type(e).__name__}"
-
- @staticmethod
- def _arrow_buffers(arrow_obj):
- """
- Every non-null buffer backing ``arrow_obj``.
-
- A ChunkedArray has no buffers of its own -- its data lives in its chunks --
- so it is expanded first. ``None`` entries (e.g. an absent validity bitmap)
- are skipped.
- """
- if isinstance(arrow_obj, pa.ChunkedArray):
- chunks = [arrow_obj.chunk(i) for i in range(arrow_obj.num_chunks)]
- else:
- chunks = [arrow_obj]
-
- buffers = []
- for chunk in chunks:
- for buffer in chunk.buffers():
- if buffer is not None:
- buffers.append(buffer)
- return buffers
-
- @classmethod
- def _verify_zero_copy(cls, arr, **to_pandas_kwargs) -> str:
- """
- Independently verify whether ``to_pandas`` reused ``arr``'s buffers,
- instead of trusting PyArrow's own ``zero_copy_only`` verdict.
-
- The check inspects whatever storage pandas returned rather than assuming a
- numpy-backed Series, so it stays correct as pandas moves more dtypes to
- Arrow-backed storage.
-
- Returns ``"zero-copy"``, ``"partial-copy"``, ``"copied"``, or
- ``"ERR@<ExceptionClass>"``.
- """
- try:
- series = arr.to_pandas(**to_pandas_kwargs)
- except Exception as e:
- return f"ERR@{type(e).__name__}"
-
- backing_array = series.array
-
- # Arrow-backed result: compare buffer addresses, reading the stored data
- # back through the public __arrow_array__ protocol. to_numpy() would
- # materialize a copy here and wrongly report no sharing. Keying on the
- # protocol rather than on ArrowDtype also covers dtypes that are
- # Arrow-backed without being ArrowDtype, such as pandas 3's string.
- #
- # The result has several buffers (validity, offsets, values) and can borrow
- # some while allocating others, so count how many it borrowed rather than
- # stopping at the first match. The result's buffers are the denominator
- # because the question is what pandas had to allocate.
- if hasattr(backing_array, "__arrow_array__"):
- source_addresses = {buffer.address for buffer in cls._arrow_buffers(arr)}
- stored_buffers = cls._arrow_buffers(pa.array(backing_array))
- borrowed = 0
- for buffer in stored_buffers:
- if buffer.address in source_addresses:
- borrowed += 1
- if borrowed == len(stored_buffers):
- return "zero-copy"
- return "partial-copy" if borrowed else "copied"
-
- # numpy-backed result: np.shares_memory accounts for slice offsets, so it
- # is robust where raw address equality is not.
- pandas_values = series.to_numpy()
- for buffer in cls._arrow_buffers(arr):
- if np.shares_memory(np.frombuffer(buffer, dtype=np.uint8), pandas_values):
- return "zero-copy"
-
- # Zero bytes cannot overlap, so an empty result is zero-copy if the numpy
- # array still borrows the Arrow array (numpy's .base is its memory owner).
- if len(arr) == 0:
- owner = pandas_values
- while (base := getattr(owner, "base", None)) is not None:
- if base is arr:
- return "zero-copy"
- owner = base
-
- return "copied"
-
-
@unittest.skipIf(
not have_pyarrow or not have_pandas or not have_numpy,
pyarrow_requirement_message or pandas_requirement_message or numpy_requirement_message,
@@ -206,41 +104,17 @@ class PyArrowArrayToPandasCoerceTemporalTests(_PyArrowToPandasTestBase):
"""
def _build_source_arrays(self):
- """Build an ordered dict of named source PyArrow arrays for testing."""
- sources = {}
-
- # =====================================================================
- # Timestamp types (the primary target of coerce_temporal_nanoseconds)
- # =====================================================================
- dt1 = datetime.datetime(2024, 1, 1, 12, 0, 0)
- dt2 = datetime.datetime(2024, 6, 15, 18, 30, 0)
- for unit in ["s", "ms", "us", "ns"]:
- sources[f"timestamp[{unit}]:standard"] = pa.array([dt1, dt2], pa.timestamp(unit))
- sources[f"timestamp[{unit}]:nullable"] = pa.array([dt1, None], pa.timestamp(unit))
- sources[f"timestamp[{unit}]:empty"] = pa.array([], pa.timestamp(unit))
- # Timestamp with timezone
- sources["timestamp[us,tz=UTC]:standard"] = pa.array(
- [dt1, dt2], pa.timestamp("us", tz="UTC")
- )
- sources["timestamp[us,tz=UTC]:nullable"] = pa.array(
- [dt1, None], pa.timestamp("us", tz="UTC")
- )
- sources["timestamp[us,tz=UTC]:empty"] = pa.array([], pa.timestamp("us", tz="UTC"))
+ """
+ Reuse the base's temporal group (the types this argument targets), then add
+ coercion-specific overflow rows and a few non-temporal controls.
+ """
+ sources = self._temporal_sources()
+
# Overflow: coercion to nanoseconds has a valid range (~1677-2262); a
# far-future second-resolution timestamp cannot fit and should error.
sources["timestamp[s]:overflow"] = pa.array(
[datetime.datetime(2500, 1, 1)], pa.timestamp("s")
)
-
- # =====================================================================
- # Duration types (also coerced to nanoseconds)
- # =====================================================================
- td1 = datetime.timedelta(days=1)
- td2 = datetime.timedelta(hours=2, minutes=30)
- for unit in ["s", "ms", "us", "ns"]:
- sources[f"duration[{unit}]:standard"] = pa.array([td1, td2], pa.duration(unit))
- sources[f"duration[{unit}]:nullable"] = pa.array([td1, None], pa.duration(unit))
- sources[f"duration[{unit}]:empty"] = pa.array([], pa.duration(unit))
# Overflow: a duration beyond ~292 years exceeds the int64 nanosecond
# range. Unlike timestamp overflow (which raises), coercing it silently
# wraps around to a bogus value; this row pins that behavior.
@@ -248,43 +122,7 @@ class PyArrowArrayToPandasCoerceTemporalTests(_PyArrowToPandasTestBase):
[datetime.timedelta(days=300 * 365)], pa.duration("s")
)
- # =====================================================================
- # Date types. With the default date_as_object=True, pandas yields an
- # object-dtype Series of datetime.date, so coerce_temporal_nanoseconds
- # has nothing to coerce; the "date_as_object=False" column forces the
- # numeric datetime64[ns] path where the argument actually takes effect.
- # =====================================================================
- d1 = datetime.date(2024, 1, 1)
- d2 = datetime.date(2024, 6, 15)
- sources["date32:standard"] = pa.array([d1, d2], pa.date32())
- sources["date32:nullable"] = pa.array([d1, None], pa.date32())
- sources["date32:empty"] = pa.array([], pa.date32())
- sources["date64:standard"] = pa.array([d1, d2], pa.date64())
- sources["date64:nullable"] = pa.array([d1, None], pa.date64())
- sources["date64:empty"] = pa.array([], pa.date64())
-
- # =====================================================================
- # Time types (control: pandas yields object-dtype datetime.time, no
- # native time-of-day dtype, so coerce_temporal_nanoseconds is a no-op)
- # =====================================================================
- t1 = datetime.time(12, 30, 0)
- t2 = datetime.time(18, 45, 30)
- sources["time32[s]:standard"] = pa.array([t1, t2], pa.time32("s"))
- sources["time32[s]:nullable"] = pa.array([t1, None], pa.time32("s"))
- sources["time32[s]:empty"] = pa.array([], pa.time32("s"))
- sources["time32[ms]:standard"] = pa.array([t1, t2], pa.time32("ms"))
- sources["time32[ms]:nullable"] = pa.array([t1, None], pa.time32("ms"))
- sources["time32[ms]:empty"] = pa.array([], pa.time32("ms"))
- sources["time64[us]:standard"] = pa.array([t1, t2], pa.time64("us"))
- sources["time64[us]:nullable"] = pa.array([t1, None], pa.time64("us"))
- sources["time64[us]:empty"] = pa.array([], pa.time64("us"))
- sources["time64[ns]:standard"] = pa.array([t1, t2], pa.time64("ns"))
- sources["time64[ns]:nullable"] = pa.array([t1, None], pa.time64("ns"))
- sources["time64[ns]:empty"] = pa.array([], pa.time64("ns"))
-
- # =====================================================================
- # Non-temporal controls (unaffected by coerce_temporal_nanoseconds)
- # =====================================================================
+ # Non-temporal controls (unaffected by coerce_temporal_nanoseconds).
sources["int64:standard"] = pa.array([0, 1, -1], pa.int64())
sources["int64:nullable"] = pa.array([0, 1, None], pa.int64())
sources["float64:standard"] = pa.array([0.0, 1.5, -1.5], pa.float64())
@@ -395,19 +233,18 @@ class PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
def _build_source_arrays(self):
"""
- Reuse every row of the sibling default test, then add the layout variants
- that only matter for zero-copy.
+ Reuse the base's full inventory, then add the layout variants that only
+ matter for zero-copy.
- Sharing the row set keeps both golden files pinning the same types, and
- means a type added to the default test is covered here automatically.
+ Sharing the row set keeps this and the default golden pinning the same types,
+ and a type added to the base is covered here automatically.
A slice still views a contiguous no-null region, so it stays zero-copy even
though its data starts partway into the parent buffer. A single chunk is
zero-copy, but multiple chunks must be concatenated into one contiguous
numpy buffer -- a copy. Multi-chunk is the common shape in real PySpark,
where each partition arrives as its own chunk.
"""
- default_tests = test_pyarrow_arrow_to_pandas_default.PyArrowArrayToPandasDefaultTests
- sources = default_tests._build_source_arrays(self)
+ sources = super()._build_source_arrays()
sources["int64:sliced"] = pa.array(list(range(10)), pa.int64()).slice(2, 3)
sources["int64:sliced-with-null"] = pa.array([1, 2, None, 4, 5], pa.int64()).slice(1, 3)
@@ -418,6 +255,86 @@ class PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
return sources
+ @staticmethod
+ def _arrow_buffers(arrow_obj):
+ """
+ Every non-null buffer backing ``arrow_obj``.
+
+ A ChunkedArray has no buffers of its own -- its data lives in its chunks --
+ so it is expanded first. ``None`` entries (e.g. an absent validity bitmap)
+ are skipped.
+ """
+ if isinstance(arrow_obj, pa.ChunkedArray):
+ chunks = [arrow_obj.chunk(i) for i in range(arrow_obj.num_chunks)]
+ else:
+ chunks = [arrow_obj]
+
+ buffers = []
+ for chunk in chunks:
+ for buffer in chunk.buffers():
+ if buffer is not None:
+ buffers.append(buffer)
+ return buffers
+
+ @classmethod
+ def _verify_zero_copy(cls, arr, **to_pandas_kwargs) -> str:
+ """
+ Independently verify whether ``to_pandas`` reused ``arr``'s buffers,
+ instead of trusting PyArrow's own ``zero_copy_only`` verdict.
+
+ The check inspects whatever storage pandas returned rather than assuming a
+ numpy-backed Series, so it stays correct as pandas moves more dtypes to
+ Arrow-backed storage.
+
+ Returns ``"zero-copy"``, ``"partial-copy"``, ``"copied"``, or
+ ``"ERR@<ExceptionClass>"``.
+ """
+ try:
+ series = arr.to_pandas(**to_pandas_kwargs)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+
+ backing_array = series.array
+
+ # Arrow-backed result: compare buffer addresses, reading the stored data
+ # back through the public __arrow_array__ protocol. to_numpy() would
+ # materialize a copy here and wrongly report no sharing. Keying on the
+ # protocol rather than on ArrowDtype also covers dtypes that are
+ # Arrow-backed without being ArrowDtype, such as pandas 3's string.
+ #
+ # The result has several buffers (validity, offsets, values) and can borrow
+ # some while allocating others, so count how many it borrowed rather than
+ # stopping at the first match. The result's buffers are the denominator
+ # because the question is what pandas had to allocate.
+ if hasattr(backing_array, "__arrow_array__"):
+ source_addresses = {buffer.address for buffer in cls._arrow_buffers(arr)}
+ stored_buffers = cls._arrow_buffers(pa.array(backing_array))
+ borrowed = 0
+ for buffer in stored_buffers:
+ if buffer.address in source_addresses:
+ borrowed += 1
+ if borrowed == len(stored_buffers):
+ return "zero-copy"
+ return "partial-copy" if borrowed else "copied"
+
+ # numpy-backed result: np.shares_memory accounts for slice offsets, so it
+ # is robust where raw address equality is not.
+ pandas_values = series.to_numpy()
+ for buffer in cls._arrow_buffers(arr):
+ if np.shares_memory(np.frombuffer(buffer, dtype=np.uint8), pandas_values):
+ return "zero-copy"
+
+ # Zero bytes cannot overlap, so an empty result is zero-copy if the numpy
+ # array still borrows the Arrow array (numpy's .base is its memory owner).
+ if len(arr) == 0:
+ owner = pandas_values
+ while (base := getattr(owner, "base", None)) is not None:
+ if base is arr:
+ return "zero-copy"
+ owner = base
+
+ return "copied"
+
# Output column for the default zero_copy_only=False: PyArrow silently copies
# when a view is not possible, so this always succeeds and records the dtype.
COL_ZERO_COPY_OFF = "zero_copy_only=False"
@@ -575,13 +492,11 @@ class PyArrowArrayToPandasIntegerObjectNullsTests(_PyArrowToPandasTestBase):
def _build_source_arrays(self):
"""
- Reuse every row of the sibling default test, then add the integer variants
- that only matter for this argument: the shared rows' nullable values are
- small enough to survive ``float64``, and their nested rows are missing whole
- sub-lists rather than a single integer inside one.
+ Reuse the base's full inventory, then add the integer variants it does not
+ reach: its nullable values are small enough to survive ``float64``, and its
+ nested rows are missing whole sub-lists rather than a single integer inside one.
"""
- default_tests = test_pyarrow_arrow_to_pandas_default.PyArrowArrayToPandasDefaultTests
- sources = default_tests._build_source_arrays(self)
+ sources = super()._build_source_arrays()
# Each width's min and max alongside a null. The shared rows use small values,
# which float64 represents exactly; only at 64 bits does the range exceed its
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org