You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Spenserrrr (via GitHub)" <gi...@apache.org> on 2026/08/13 23:47:31 UTC
[PR] [SPARK-58720][PYTHON][TESTS][FOLLOWUP] Move the pandas DataFrame golden repr into GoldenFileTestMixin [spark]
Spenserrrr opened a new pull request, #57999:
URL: https://github.com/apache/spark/pull/57999
### What changes were proposed in this pull request?
Follow-up to #57939 (SPARK-58720). That PR added `pa.Table.to_pandas` golden-file tests with a local `_repr_dataframe` helper that formats a pandas DataFrame result via per-column `tolist()`, deliberately bypassing `GoldenFileTestMixin.repr_pandas_value` (which used `DataFrame.to_json`). This PR promotes that formatter into the shared mixin so there is a single DataFrame repr:
- `GoldenFileTestMixin.repr_pandas_value` now renders each column with `tolist()` (byte-identical to the old local `_repr_dataframe`) instead of `DataFrame.to_json`.
- `test_pyarrow_table_to_pandas.py` drops its local `_repr_dataframe` and formats results through `repr_value(pdf, max_len=0)`.
- `test_pandas_udf_return_type`'s golden files (`pandas_2` and `pandas_3`) are regenerated. It is the only other test that renders a pandas DataFrame through `repr_value` — the `pd.DataFrame({"_1": [1, 2]})` column header. The regeneration changes only that header; all coercion result cells are unchanged.
### Why are the changes needed?
`DataFrame.to_json` defaults to an epoch date format that serializes temporal values to an int64 nanosecond count. That overflows on out-of-nanosecond-range dates (e.g. year 9999 under the default `date_as_object=True`, which returns an object column of Python `datetime.date`) and misreads non-nanosecond units on pandas 2 (so the same cell drifts between pandas 2 and 3). `tolist()` bypasses the epoch machinery entirely, giving a stable, version-independent representation — exactly what a drift-canary golden needs — and matches `repr_pandas_series_value`, which already uses `tolist()` for the Series case. Keeping the formatter local to one test file left the shared mixin with the fragile `to_json` behavior for any future DataFrame consumer.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
- `test_pyarrow_table_to_pandas` passes unchanged against its committed golden files, confirming the switch to `repr_value` is byte-identical (golden-inert).
- `test_pandas_udf_return_type` was regenerated and re-run in compare mode under both pandas 2 and pandas 3; the golden diff is the single DataFrame column header, with every data cell byte-identical.
- The full `python/pyspark/tests/upstream/pyarrow` suite and the `python/pyspark/sql/tests/coercion` suite pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58720][PYTHON][TESTS][FOLLOWUP] Move the pandas DataFrame golden repr into GoldenFileTestMixin [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #57999: [SPARK-58720][PYTHON][TESTS][FOLLOWUP] Move the pandas DataFrame golden repr into GoldenFileTestMixin
URL: https://github.com/apache/spark/pull/57999
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58720][PYTHON][TESTS][FOLLOWUP] Move the pandas DataFrame golden repr into GoldenFileTestMixin [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #57999:
URL: https://github.com/apache/spark/pull/57999#issuecomment-5290765354
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/de97fd4859948856ae0cb63045407758e0688ff0
- merged into branch-4.x https://github.com/apache/spark/commit/6a87bcccd259e94c90aa55dec04e2a3b5be5b603
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58720][PYTHON][TESTS][FOLLOWUP] Move the pandas DataFrame golden repr into GoldenFileTestMixin [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57999:
URL: https://github.com/apache/spark/pull/57999#issuecomment-5290683824
Thank you @Spenserrrr!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58720][PYTHON][TESTS][FOLLOWUP] Move the pandas DataFrame golden repr into GoldenFileTestMixin [spark]
Posted by "Spenserrrr (via GitHub)" <gi...@apache.org>.
Spenserrrr commented on PR #57999:
URL: https://github.com/apache/spark/pull/57999#issuecomment-5287715684
Hi @Yicong-Huang! This is a follow-up PR to move the pandas DataFrame golden repr into the utility file. Could you take a look when you have time? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org