You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by yi...@apache.org on 2026/08/14 07:34:34 UTC
(spark) branch branch-4.x updated: [SPARK-58720][PYTHON][TEST][FOLLOWUP] Move the pandas DataFrame golden repr into GoldenFileTestMixin
This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 6a87bcccd259 [SPARK-58720][PYTHON][TEST][FOLLOWUP] Move the pandas DataFrame golden repr into GoldenFileTestMixin
6a87bcccd259 is described below
commit 6a87bcccd259e94c90aa55dec04e2a3b5be5b603
Author: Spenser Sun <hs...@gmail.com>
AuthorDate: Fri Aug 14 07:33:58 2026 +0000
[SPARK-58720][PYTHON][TEST][FOLLOWUP] Move the pandas DataFrame golden repr into GoldenFileTestMixin
### What changes were proposed in this pull request?
Follow-up to #57939 (SPARK-58720). That PR added `pa.Table.to_pandas` golden-file tests with a local `_repr_dataframe` helper that formats a pandas DataFrame result via per-column `tolist()`, deliberately bypassing `GoldenFileTestMixin.repr_pandas_value` (which used `DataFrame.to_json`). This PR promotes that formatter into the shared mixin so there is a single DataFrame repr:
- `GoldenFileTestMixin.repr_pandas_value` now renders each column with `tolist()` (byte-identical to the old local `_repr_dataframe`) instead of `DataFrame.to_json`.
- `test_pyarrow_table_to_pandas.py` drops its local `_repr_dataframe` and formats results through `repr_value(pdf, max_len=0)`.
- `test_pandas_udf_return_type`'s golden files (`pandas_2` and `pandas_3`) are regenerated. It is the only other test that renders a pandas DataFrame through `repr_value` — the `pd.DataFrame({"_1": [1, 2]})` column header. The regeneration changes only that header; all coercion result cells are unchanged.
### Why are the changes needed?
`DataFrame.to_json` defaults to an epoch date format that serializes temporal values to an int64 nanosecond count. That overflows on out-of-nanosecond-range dates (e.g. year 9999 under the default `date_as_object=True`, which returns an object column of Python `datetime.date`) and misreads non-nanosecond units on pandas 2 (so the same cell drifts between pandas 2 and 3). `tolist()` bypasses the epoch machinery entirely, giving a stable, version-independent representation — exactly wha [...]
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
- `test_pyarrow_table_to_pandas` passes unchanged against its committed golden files, confirming the switch to `repr_value` is byte-identical (golden-inert).
- `test_pandas_udf_return_type` was regenerated and re-run in compare mode under both pandas 2 and pandas 3; the golden diff is the single DataFrame column header, with every data cell byte-identical.
- The full `python/pyspark/tests/upstream/pyarrow` suite and the `python/pyspark/sql/tests/coercion` suite pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
This pull request and its description were written by Isaac.
Closes #57999 from Spenserrrr/unify-dataframe-repr.
Authored-by: Spenser Sun <hs...@gmail.com>
Signed-off-by: Yicong-Huang <17...@users.noreply.github.com>
(cherry picked from commit de97fd4859948856ae0cb63045407758e0688ff0)
Signed-off-by: Yicong-Huang <17...@users.noreply.github.com>
---
...golden_pandas_udf_return_type_coercion_base.csv | 2 +-
.../golden_pandas_udf_return_type_coercion_base.md | 34 +++++++++++-----------
...golden_pandas_udf_return_type_coercion_base.csv | 2 +-
.../golden_pandas_udf_return_type_coercion_base.md | 34 +++++++++++-----------
python/pyspark/testing/goldenutils.py | 10 +++++--
.../pyarrow/test_pyarrow_table_to_pandas.py | 20 +------------
6 files changed, 45 insertions(+), 57 deletions(-)
diff --git a/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.csv b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.csv
index 4f5d13daaf10..ae47d24a24db 100644
--- a/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.csv
+++ b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.csv
@@ -1,4 +1,4 @@
-SQL Type \ Value@Type [None, None]@list [True, False]@list ['a', 'b']@list ['12', '34']@list [Decimal('1'), Decimal('2')]@list [{'a': 1}, {'b': 2}]@list [1 2]@ndarray[int8] [1 2]@ndarray[int16] [1 2]@ndarray[int32] [1 2]@ndarray[int64] [1 2]@ndarray[uint8] [1 2]@ndarray[uint16] [1 2]@ndarray[uint32] [1 2]@ndarray[uint64] [1. 2.]@ndarray[float16] [1. 2.]@ndarray[float32] [1. 2.]@ndarray[float64] [1.+0.j 2.+0.j]@ndarray[complex64] [1.+0.j 2.+0.j]@ndarray[complex128] [array([1, 2, 3], dtyp [...]
+SQL Type \ Value@Type [None, None]@list [True, False]@list ['a', 'b']@list ['12', '34']@list [Decimal('1'), Decimal('2')]@list [{'a': 1}, {'b': 2}]@list [1 2]@ndarray[int8] [1 2]@ndarray[int16] [1 2]@ndarray[int32] [1 2]@ndarray[int64] [1 2]@ndarray[uint8] [1 2]@ndarray[uint16] [1 2]@ndarray[uint32] [1 2]@ndarray[uint64] [1. 2.]@ndarray[float16] [1. 2.]@ndarray[float32] [1. 2.]@ndarray[float64] [1.+0.j 2.+0.j]@ndarray[complex64] [1.+0.j 2.+0.j]@ndarray[complex128] [array([1, 2, 3], dtyp [...]
boolean [None, None] [True, False] X X X X [True, True] [True, True] [True, True] [True, True] [True, True] [True, True] [True, True] [True, True] X [True, True] [True, True] X X X X X X X X
tinyint [None, None] [1, 0] X [12, 34] [1, 2] X [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] X X X X X X X X
smallint [None, None] [1, 0] X [12, 34] [1, 2] X [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] X X X X X X X X
diff --git a/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.md b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.md
index f8d7f11310fa..54a0d16389d2 100644
--- a/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.md
+++ b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.md
@@ -1,17 +1,17 @@
-| SQL Type \ Value@Type | [None, None]@list | [True, False]@list | ['a', 'b']@list | ['12', '34']@list | [Decimal('1'), Decimal('2')]@list | [{'a': 1}, {'b': 2}]@list | [1 2]@ndarray[int8] | [1 2]@ndarray[int16] | [1 2]@ndarray[int32] | [1 2]@ndarray[int64] | [1 2]@ndarray[uint8] | [1 2]@ndarray[uint16] | [1 2]@ndarray[uint32] | [1 2]@ndarray[uint64] | [1. 2.]@ndarray[float16] | [1. 2.]@ [...]
-|--------------------------|---------------------|----------------------|-------------------|---------------------|------------------------------------------|-----------------------------|------------------------------|------------------------------|------------------------------------------|-----------------------------------------|------------------------------|------------------------------|------------------------------|-------------------------|----------------------------|--------- [...]
-| boolean | [None, None] | [True, False] | X | X | X | X | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | X | [True, T [...]
-| tinyint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
-| smallint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
-| int | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
-| bigint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
-| string | [None, None] | X | ['a', 'b'] | ['12', '34'] | X | X | X | X | X | X | X | X | X | X | X | X [...]
-| date | [None, None] | X | X | X | [datetime.date(1970, 1, 2), datetime.dat | X | X | X | [datetime.date(1970, 1, 2), datetime.dat | X | X | X | X | X | X | X [...]
-| timestamp | [None, None] | X | X | X | [datetime.datetime(1969, 12, 31, 16, 0, | X | X | X | X | [datetime.datetime(1969, 12, 31, 16, 0, | X | X | X | X | X | X [...]
-| float | [None, None] | [1.0, 0.0] | X | [12.0, 34.0] | [1.0, 2.0] | X | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2. [...]
-| double | [None, None] | [1.0, 0.0] | X | [12.0, 34.0] | [1.0, 2.0] | X | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2. [...]
-| array<int> | [None, None] | X | X | [[1, 2], [3, 4]] | X | X | X | X | X | X | X | X | X | X | X | X [...]
-| binary | [None, None] | [b'\x01', b''] | [b'a', b'b'] | [b'12', b'34'] | X | X | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'', b''] | [b'', b' [...]
-| decimal(10,0) | [None, None] | X | X | X | [Decimal('1'), Decimal('2')] | X | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X | X | [Decimal [...]
-| map<string,int> | [None, None] | X | X | X | X | [{'a': 1}, {'b': 2}] | X | X | X | X | X | X | X | X | X | X [...]
-| struct<_1:int> | X | X | X | X | X | X | X | X | X | X | X | X | X | X | X | X [...]
\ No newline at end of file
+| SQL Type \ Value@Type | [None, None]@list | [True, False]@list | ['a', 'b']@list | ['12', '34']@list | [Decimal('1'), Decimal('2')]@list | [{'a': 1}, {'b': 2}]@list | [1 2]@ndarray[int8] | [1 2]@ndarray[int16] | [1 2]@ndarray[int32] | [1 2]@ndarray[int64] | [1 2]@ndarray[uint8] | [1 2]@ndarray[uint16] | [1 2]@ndarray[uint32] | [1 2]@ndarray[uint64] | [1. 2.]@ndarray[float16] | [1. 2.]@ [...]
+|--------------------------|---------------------|----------------------|-------------------|---------------------|------------------------------------------|-----------------------------|------------------------------|------------------------------|------------------------------------------|-----------------------------------------|------------------------------|------------------------------|------------------------------|-------------------------|----------------------------|--------- [...]
+| boolean | [None, None] | [True, False] | X | X | X | X | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | X | [True, T [...]
+| tinyint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
+| smallint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
+| int | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
+| bigint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
+| string | [None, None] | X | ['a', 'b'] | ['12', '34'] | X | X | X | X | X | X | X | X | X | X | X | X [...]
+| date | [None, None] | X | X | X | [datetime.date(1970, 1, 2), datetime.dat | X | X | X | [datetime.date(1970, 1, 2), datetime.dat | X | X | X | X | X | X | X [...]
+| timestamp | [None, None] | X | X | X | [datetime.datetime(1969, 12, 31, 16, 0, | X | X | X | X | [datetime.datetime(1969, 12, 31, 16, 0, | X | X | X | X | X | X [...]
+| float | [None, None] | [1.0, 0.0] | X | [12.0, 34.0] | [1.0, 2.0] | X | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2. [...]
+| double | [None, None] | [1.0, 0.0] | X | [12.0, 34.0] | [1.0, 2.0] | X | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2. [...]
+| array<int> | [None, None] | X | X | [[1, 2], [3, 4]] | X | X | X | X | X | X | X | X | X | X | X | X [...]
+| binary | [None, None] | [b'\x01', b''] | [b'a', b'b'] | [b'12', b'34'] | X | X | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'', b''] | [b'', b' [...]
+| decimal(10,0) | [None, None] | X | X | X | [Decimal('1'), Decimal('2')] | X | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X | X | [Decimal [...]
+| map<string,int> | [None, None] | X | X | X | X | [{'a': 1}, {'b': 2}] | X | X | X | X | X | X | X | X | X | X [...]
+| struct<_1:int> | X | X | X | X | X | X | X | X | X | X | X | X | X | X | X | X [...]
\ No newline at end of file
diff --git a/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv
index 5e12cc57c84a..0b955d06a552 100644
--- a/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv
+++ b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv
@@ -1,4 +1,4 @@
-SQL Type \ Value@Type [None, None]@list [True, False]@list ['a', 'b']@list ['12', '34']@list [Decimal('1'), Decimal('2')]@list [{'a': 1}, {'b': 2}]@list [1 2]@ndarray[int8] [1 2]@ndarray[int16] [1 2]@ndarray[int32] [1 2]@ndarray[int64] [1 2]@ndarray[uint8] [1 2]@ndarray[uint16] [1 2]@ndarray[uint32] [1 2]@ndarray[uint64] [1. 2.]@ndarray[float16] [1. 2.]@ndarray[float32] [1. 2.]@ndarray[float64] [1.+0.j 2.+0.j]@ndarray[complex64] [1.+0.j 2.+0.j]@ndarray[complex128] [array([1, 2, 3], dtyp [...]
+SQL Type \ Value@Type [None, None]@list [True, False]@list ['a', 'b']@list ['12', '34']@list [Decimal('1'), Decimal('2')]@list [{'a': 1}, {'b': 2}]@list [1 2]@ndarray[int8] [1 2]@ndarray[int16] [1 2]@ndarray[int32] [1 2]@ndarray[int64] [1 2]@ndarray[uint8] [1 2]@ndarray[uint16] [1 2]@ndarray[uint32] [1 2]@ndarray[uint64] [1. 2.]@ndarray[float16] [1. 2.]@ndarray[float32] [1. 2.]@ndarray[float64] [1.+0.j 2.+0.j]@ndarray[complex64] [1.+0.j 2.+0.j]@ndarray[complex128] [array([1, 2, 3], dtyp [...]
boolean [None, None] [True, False] X X X X [True, True] [True, True] [True, True] [True, True] [True, True] [True, True] [True, True] [True, True] X [True, True] [True, True] X X X X X X X X
tinyint [None, None] [1, 0] X [12, 34] [1, 2] X [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] X X X X X X X X
smallint [None, None] [1, 0] X [12, 34] [1, 2] X [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] X X X X X X X X
diff --git a/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md
index aa57ca6eec96..257ae0e7548c 100644
--- a/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md
+++ b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md
@@ -1,17 +1,17 @@
-| SQL Type \ Value@Type | [None, None]@list | [True, False]@list | ['a', 'b']@list | ['12', '34']@list | [Decimal('1'), Decimal('2')]@list | [{'a': 1}, {'b': 2}]@list | [1 2]@ndarray[int8] | [1 2]@ndarray[int16] | [1 2]@ndarray[int32] | [1 2]@ndarray[int64] | [1 2]@ndarray[uint8] | [1 2]@ndarray[uint16] | [1 2]@ndarray[uint32] | [1 2]@ndarray[uint64] | [1. 2.]@ndarray[float16] [...]
-|--------------------------|---------------------|----------------------|-------------------|--------------------------------|------------------------------------------|-----------------------------|------------------------------|------------------------------|------------------------------------------|-----------------------------------------|------------------------------|------------------------------|------------------------------|-------------------------|--------------------------- [...]
-| boolean | [None, None] | [True, False] | X | X | X | X | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | X [...]
-| tinyint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
-| smallint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
-| int | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
-| bigint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
-| string | [None, None] | X | ['a', 'b'] | ['12', '34'] | X | X | X | X | X | X | X | X | X | X | X [...]
-| date | [None, None] | X | X | X | [datetime.date(1970, 1, 2), datetime.dat | X | X | X | [datetime.date(1970, 1, 2), datetime.dat | X | X | X | X | X | X [...]
-| timestamp | [None, None] | X | X | X | [datetime.datetime(1969, 12, 31, 16, 0, | X | X | X | X | [datetime.datetime(1969, 12, 31, 16, 0, | X | X | X | X | X [...]
-| float | [None, None] | [1.0, 0.0] | X | [12.0, 34.0] | [1.0, 2.0] | X | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] [...]
-| double | [None, None] | [1.0, 0.0] | X | [12.0, 34.0] | [1.0, 2.0] | X | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] [...]
-| array<int> | [None, None] | X | X | [[1, 2], [3, 4]] | X | X | X | X | X | X | X | X | X | X | X [...]
-| binary | [None, None] | [b'\x01', b''] | [b'a', b'b'] | [b'12', b'34'] | X | X | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'', b''] [...]
-| decimal(10,0) | [None, None] | X | X | [Decimal('12'), Decimal('34')] | [Decimal('1'), Decimal('2')] | X | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X | X [...]
-| map<string,int> | [None, None] | X | X | X | X | [{'a': 1}, {'b': 2}] | X | X | X | X | X | X | X | X | X [...]
-| struct<_1:int> | X | X | X | X | X | X | X | X | X | X | X | X | X | X | X [...]
\ No newline at end of file
+| SQL Type \ Value@Type | [None, None]@list | [True, False]@list | ['a', 'b']@list | ['12', '34']@list | [Decimal('1'), Decimal('2')]@list | [{'a': 1}, {'b': 2}]@list | [1 2]@ndarray[int8] | [1 2]@ndarray[int16] | [1 2]@ndarray[int32] | [1 2]@ndarray[int64] | [1 2]@ndarray[uint8] | [1 2]@ndarray[uint16] | [1 2]@ndarray[uint32] | [1 2]@ndarray[uint64] | [1. 2.]@ndarray[float16] [...]
+|--------------------------|---------------------|----------------------|-------------------|--------------------------------|------------------------------------------|-----------------------------|------------------------------|------------------------------|------------------------------------------|-----------------------------------------|------------------------------|------------------------------|------------------------------|-------------------------|--------------------------- [...]
+| boolean | [None, None] | [True, False] | X | X | X | X | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | [True, True] | X [...]
+| tinyint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
+| smallint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
+| int | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
+| bigint | [None, None] | [1, 0] | X | [12, 34] | [1, 2] | X | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] | [1, 2] [...]
+| string | [None, None] | X | ['a', 'b'] | ['12', '34'] | X | X | X | X | X | X | X | X | X | X | X [...]
+| date | [None, None] | X | X | X | [datetime.date(1970, 1, 2), datetime.dat | X | X | X | [datetime.date(1970, 1, 2), datetime.dat | X | X | X | X | X | X [...]
+| timestamp | [None, None] | X | X | X | [datetime.datetime(1969, 12, 31, 16, 0, | X | X | X | X | [datetime.datetime(1969, 12, 31, 16, 0, | X | X | X | X | X [...]
+| float | [None, None] | [1.0, 0.0] | X | [12.0, 34.0] | [1.0, 2.0] | X | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] [...]
+| double | [None, None] | [1.0, 0.0] | X | [12.0, 34.0] | [1.0, 2.0] | X | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0] [...]
+| array<int> | [None, None] | X | X | [[1, 2], [3, 4]] | X | X | X | X | X | X | X | X | X | X | X [...]
+| binary | [None, None] | [b'\x01', b''] | [b'a', b'b'] | [b'12', b'34'] | X | X | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'', b''] [...]
+| decimal(10,0) | [None, None] | X | X | [Decimal('12'), Decimal('34')] | [Decimal('1'), Decimal('2')] | X | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X | X [...]
+| map<string,int> | [None, None] | X | X | X | X | [{'a': 1}, {'b': 2}] | X | X | X | X | X | X | X | X | X [...]
+| struct<_1:int> | X | X | X | X | X | X | X | X | X | X | X | X | X | X | X [...]
\ No newline at end of file
diff --git a/python/pyspark/testing/goldenutils.py b/python/pyspark/testing/goldenutils.py
index 95b7da956ff3..7f2c889954ed 100644
--- a/python/pyspark/testing/goldenutils.py
+++ b/python/pyspark/testing/goldenutils.py
@@ -366,6 +366,12 @@ class GoldenFileTestMixin:
"""
Format a pandas DataFrame for golden file.
+ Renders each column with tolist() (as ``repr_pandas_series_value`` does for a
+ Series), keyed by column name, plus the schema. tolist() gives a stable
+ Python-native representation and avoids ``DataFrame.to_json``'s epoch date
+ serialization, which overflows on out-of-nanosecond-range dates (year 9999
+ with the default date_as_object=True) and misreads non-ns units on pandas 2.
+
Parameters
----------
value : pd.DataFrame
@@ -376,9 +382,9 @@ class GoldenFileTestMixin:
Returns
-------
str
- "value@Dataframe[schema]"
+ "{col: [val1, val2], ...}@Dataframe[schema]"
"""
- v_str = value.to_json().replace("\n", " ")
+ v_str = str({name: col.tolist() for name, col in value.items()}).replace("\n", " ")
if max_len > 0:
v_str = v_str[:max_len]
simple_schema = ", ".join([f"{t} {d.name}" for t, d in value.dtypes.items()])
diff --git a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
index 42977bea5bf5..8c5175961184 100644
--- a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
+++ b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
@@ -70,24 +70,6 @@ class _PyArrowTableToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
its own.
"""
- @staticmethod
- def _repr_dataframe(df) -> str:
- """
- Format a pandas DataFrame result as a golden-file cell: per-column tolist()
- for a stable, Python-native representation, mirroring goldenutils'
- ``repr_pandas_series_value`` for the Series case.
-
- This deliberately does NOT go through ``repr_value``/``repr_pandas_value``
- (which use ``to_json``). With the default date_as_object=True a far-future
- date (year 9999) comes back as an object column of Python ``datetime.date``
- objects, and ``DataFrame.to_json`` overflows converting each to an epoch
- nanosecond int (OverflowError); tolist() returns the objects as-is, so
- nothing overflows.
- """
- body = str({name: col.tolist() for name, col in df.items()}).replace("\n", " ")
- schema = ", ".join(f"{t} {d.name}" for t, d in df.dtypes.items())
- return f"{body}@Dataframe[{schema}]"
-
def _to_pandas_cell(self, table, **to_pandas_kwargs) -> str:
"""
Convert ``table`` via ``to_pandas(**to_pandas_kwargs)`` and format the result
@@ -99,7 +81,7 @@ class _PyArrowTableToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
pdf = table.to_pandas(**to_pandas_kwargs)
except Exception as e:
return f"ERR@{type(e).__name__}"
- return self._repr_dataframe(pdf)
+ return self.repr_value(pdf, max_len=0)
def _structural_tables(self):
"""Assembly shapes and empty edges (no temporal coercion involved)."""
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org