You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by yi...@apache.org on 2026/09/05 13:15:03 UTC
(spark) branch branch-4.3 updated: [SPARK-59011][PS] Fix descending rank first tie ordering
This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 9d66f580555c [SPARK-59011][PS] Fix descending rank first tie ordering
9d66f580555c is described below
commit 9d66f580555ce3f8f16d6cec7762467fbbd35ad0
Author: NaVis <92...@users.noreply.github.com>
AuthorDate: Sat Sep 5 13:14:28 2026 +0000
[SPARK-59011][PS] Fix descending rank first tie ordering
### What changes were proposed in this pull request?
This pull request fixes the tie-breaking order of pandas-on-Spark `Series.rank(method="first", ascending=False)`.
The value column follows the requested ascending direction, while the natural-order column is always ordered ascending so that equal values are ranked according to their original occurrence order.
A regression test for descending ranking with `method="first"` is added.
### Why are the changes needed?
The existing implementation applies descending order to both the value column and the natural-order tie-breaker. This reverses the original order of equal values and produces results inconsistent with pandas.
For `pd.Series([1, 2, 3, 1])`, pandas returns:
`[3.0, 2.0, 1.0, 4.0]`
The previous pandas-on-Spark result was:
`[4.0, 2.0, 1.0, 3.0]`
### Does this PR introduce _any_ user-facing change?
Yes. `Series.rank(method="first", ascending=False)` now preserves the original occurrence order when breaking ties, matching pandas behavior.
### How was this patch tested?
A regression assertion was added to `SeriesStatTests.test_rank`.
The test failed before the fix with a 50% value difference and passed after the fix:
./python/run-tests --python-executables python3 \
--testnames 'pyspark.pandas.tests.series.test_stat SeriesStatTests.test_rank' \
-p 1
The Spark assembly build, Ruff checks, Ruff formatting check, and `git diff --check` also passed.
This contribution was prepared and reviewed by our university project team, including Phan Trung Nhut and Bui Nam Viet. We license this contribution to the project under the Apache License, Version 2.0.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
Closes #58337 from namviet157/fix-SPARK-59011.
Lead-authored-by: NaVis <92...@users.noreply.github.com>
Co-authored-by: Phan Trung Nhựt <17...@users.noreply.github.com>
Signed-off-by: Yicong-Huang <17...@users.noreply.github.com>
(cherry picked from commit d7cb6592d67f92d11239e1cead153d6fc80fce18)
Signed-off-by: Yicong-Huang <17...@users.noreply.github.com>
---
python/pyspark/pandas/series.py | 2 +-
python/pyspark/pandas/tests/series/test_stat.py | 4 ++++
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/series.py b/python/pyspark/pandas/series.py
index 159a7d01a135..9f9536799012 100644
--- a/python/pyspark/pandas/series.py
+++ b/python/pyspark/pandas/series.py
@@ -4316,7 +4316,7 @@ class Series(Frame, IndexOpsMixin, Generic[T]):
window = (
Window.orderBy(
asc_func(self.spark.column),
- asc_func(F.col(NATURAL_ORDER_COLUMN_NAME)),
+ F.col(NATURAL_ORDER_COLUMN_NAME).asc(),
)
.partitionBy(*part_cols)
.rowsBetween(Window.unboundedPreceding, Window.currentRow)
diff --git a/python/pyspark/pandas/tests/series/test_stat.py b/python/pyspark/pandas/tests/series/test_stat.py
index c169ed1dc507..10cf9ed9e3ef 100644
--- a/python/pyspark/pandas/tests/series/test_stat.py
+++ b/python/pyspark/pandas/tests/series/test_stat.py
@@ -319,6 +319,10 @@ class SeriesStatMixin:
self.assert_eq(pser.rank(method="min"), psser.rank(method="min").sort_index())
self.assert_eq(pser.rank(method="max"), psser.rank(method="max").sort_index())
self.assert_eq(pser.rank(method="first"), psser.rank(method="first").sort_index())
+ self.assert_eq(
+ pser.rank(method="first", ascending=False),
+ psser.rank(method="first", ascending=False).sort_index(),
+ )
self.assert_eq(pser.rank(method="dense"), psser.rank(method="dense").sort_index())
non_numeric_pser = pd.Series(["a", "c", "b", "d"], name="x", index=[10, 11, 12, 13])
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org