You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2026/03/25 22:47:59 UTC
[PR] [SPARK-56219][PS] Align groupby idxmax and idxmin skipna=False behavior with pandas 2/3 [spark]
ueshin opened a new pull request, #55021:
URL: https://github.com/apache/spark/pull/55021
### What changes were proposed in this pull request?
This PR updates pandas-on-Spark `GroupBy.idxmax` and `GroupBy.idxmin` so that `skipna=False` follows the version-specific pandas behavior more closely.
For pandas `< 3.0.0`, NA-containing groups return null results for the affected group/column outputs.
For pandas `>= 3.0.0`, NA-containing inputs raise for `skipna=False`, and no-NA inputs follow pandas 3’s all-missing-label behavior.
This PR also updates the related tests in `python/pyspark/pandas/tests/groupby/test_index.py` to cover both pandas 2 and pandas 3 behavior, including cases with non-default indexes.
### Why are the changes needed?
`GroupBy.idxmax(skipna=False)` and `GroupBy.idxmin(skipna=False)` differ between pandas 2 and pandas 3, and pandas-on-Spark was not matching those semantics.
As a result, pandas-on-Spark could return index labels where pandas would return null or raise, depending on the pandas version in use.
### Does this PR introduce _any_ user-facing change?
Yes.
For pandas-on-Spark groupby `idxmax(skipna=False)` and `idxmin(skipna=False)`, behavior now matches pandas more closely:
- with pandas 2, groups containing NA values return null results for the affected outputs
- with pandas 3, NA-containing inputs raise for `skipna=False`
- with pandas 3, no-NA `skipna=False` inputs follow pandas 3 behavior
### How was this patch tested?
Ran the related pandas-on-Spark groupby tests in both pandas 2 and pandas 3 environments, including:
- `GroupbyIndexTests.test_idxmax`
- `GroupbyIndexTests.test_idxmin`
- `GroupbyIndexTests.test_idxmax_idxmin_skipna_false_with_na`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56219][PS] Align groupby idxmax and idxmin skipna=False behavior with pandas 2/3 [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #55021:
URL: https://github.com/apache/spark/pull/55021#issuecomment-4130327981
cc @gaogaotiantian @HyukjinKwon @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56219][PS] Align groupby idxmax and idxmin skipna=False behavior with pandas 2/3 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #55021:
URL: https://github.com/apache/spark/pull/55021#issuecomment-4131928285
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56219][PS] Align groupby idxmax and idxmin skipna=False behavior with pandas 2/3 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #55021: [SPARK-56219][PS] Align groupby idxmax and idxmin skipna=False behavior with pandas 2/3
URL: https://github.com/apache/spark/pull/55021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org