You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2026/04/01 22:06:46 UTC
[PR] [SPARK-56327][PYTHON][TESTS] Fix grouped map pandas tests for pandas 3 [spark]
ueshin opened a new pull request, #55146:
URL: https://github.com/apache/spark/pull/55146
### What changes were proposed in this pull request?
This PR updates `python/pyspark/sql/tests/pandas/test_pandas_grouped_map.py` for pandas 3 behavior in grouped map pandas UDF tests.
The changes are:
- update the expected boolean inversion in `test_supported_types` to use `~pdf.bool` on pandas 3 while keeping the existing behavior on older pandas versions
- update several pandas-side expected-value paths to avoid grouping directly by the same in-DataFrame column on pandas 3
- use copied groupers such as `pdf.id.copy()` so the grouped pandas input still keeps the grouping columns while preserving the original grouping semantics
- add comments explaining why the pandas 3 branch uses copied groupers
### Why are the changes needed?
In pandas 3, `GroupBy.apply` drops grouping columns when the grouping key is the same DataFrame column. These tests build expected results by applying the Python function to pandas-grouped data, so the previous expectations no longer match the grouped map input shape seen by Spark in cases that rely on grouping columns remaining present.
The boolean expectation also needs a pandas-3-specific branch because the old scalar-style inversion logic does not match the pandas 3 object being operated on in these tests.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Updated the related tests.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56327][PYTHON][TESTS] Fix grouped map pandas tests for pandas 3 [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #55146:
URL: https://github.com/apache/spark/pull/55146#issuecomment-4173248436
cc @gaogaotiantian @HyukjinKwon @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56327][PYTHON][TESTS] Fix grouped map pandas tests for pandas 3 [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin closed pull request #55146: [SPARK-56327][PYTHON][TESTS] Fix grouped map pandas tests for pandas 3
URL: https://github.com/apache/spark/pull/55146
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56327][PYTHON][TESTS] Fix grouped map pandas tests for pandas 3 [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #55146:
URL: https://github.com/apache/spark/pull/55146#issuecomment-4180187560
Thanks! merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org