You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "namviet157 (via GitHub)" <gi...@apache.org> on 2026/08/27 09:09:56 UTC
[PR] [SPARK-59011][PS] Fix descending rank first tie ordering [spark]
namviet157 opened a new pull request, #58337:
URL: https://github.com/apache/spark/pull/58337
### What changes were proposed in this pull request?
This pull request fixes the tie-breaking order of pandas-on-Spark `Series.rank(method="first", ascending=False)`.
The value column follows the requested ascending direction, while the natural-order column is always ordered ascending so that equal values are ranked according to their original occurrence order.
A regression test for descending ranking with `method="first"` is added.
### Why are the changes needed?
The existing implementation applies descending order to both the value column and the natural-order tie-breaker. This reverses the original order of equal values and produces results inconsistent with pandas.
For `pd.Series([1, 2, 3, 1])`, pandas returns:
`[3.0, 2.0, 1.0, 4.0]`
The previous pandas-on-Spark result was:
`[4.0, 2.0, 1.0, 3.0]`
### Does this PR introduce _any_ user-facing change?
Yes. `Series.rank(method="first", ascending=False)` now preserves the original occurrence order when breaking ties, matching pandas behavior.
### How was this patch tested?
A regression assertion was added to `SeriesStatTests.test_rank`.
The test failed before the fix with a 50% value difference and passed after the fix:
./python/run-tests --python-executables python3 \
--testnames 'pyspark.pandas.tests.series.test_stat SeriesStatTests.test_rank' \
-p 1
The Spark assembly build, Ruff checks, Ruff formatting check, and `git diff --check` also passed.
This contribution was prepared and reviewed by our university project team, including Phan Trung Nhut and Bui Nam Viet. We license this contribution to the project under the Apache License, Version 2.0.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59011][PS] Fix descending rank first tie ordering [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58337:
URL: https://github.com/apache/spark/pull/58337#issuecomment-5552098207
I assigned the jira ticket to `nhutphan` as it commented on the ticket. @namviet157 if you have a jira account to be assigned please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59011][PS] Fix descending rank first tie ordering [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58337:
URL: https://github.com/apache/spark/pull/58337#issuecomment-5471856420
Adding @zhengruifeng for PS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59011][PS] Fix descending rank first tie ordering [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #58337: [SPARK-59011][PS] Fix descending rank first tie ordering
URL: https://github.com/apache/spark/pull/58337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59011][PS] Fix descending rank first tie ordering [spark]
Posted by "namviet157 (via GitHub)" <gi...@apache.org>.
namviet157 commented on PR #58337:
URL: https://github.com/apache/spark/pull/58337#issuecomment-5476512802
Thank you for the thorough review and verification, @zhengruifeng!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59011][PS] Fix descending rank first tie ordering [spark]
Posted by "namviet157 (via GitHub)" <gi...@apache.org>.
namviet157 commented on PR #58337:
URL: https://github.com/apache/spark/pull/58337#issuecomment-5548302865
Hi @Yicong-Huang, gentle ping to see if you have any feedback on this PR when you have a moment. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59011][PS] Fix descending rank first tie ordering [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58337:
URL: https://github.com/apache/spark/pull/58337#issuecomment-5552067397
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/d7cb6592d67f92d11239e1cead153d6fc80fce18
- merged into branch-4.x https://github.com/apache/spark/commit/16b4b204a5e86aa5f56ecfc7c13aa4d56fabf7bf
- merged into branch-4.3 https://github.com/apache/spark/commit/9d66f580555ce3f8f16d6cec7762467fbbd35ad0
- merged into branch-4.2 https://github.com/apache/spark/commit/76ba3bb7c4c0c75db00d54fb31686668f01e6ae5
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59011][PS] Fix descending rank first tie ordering [spark]
Posted by "namviet157 (via GitHub)" <gi...@apache.org>.
namviet157 commented on PR #58337:
URL: https://github.com/apache/spark/pull/58337#issuecomment-5473473089
Thank you for reviewing the PR, @uros-b! We appreciate your time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org