You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/08/27 09:10:00 UTC
[jira] [Updated] (SPARK-59011) Series.rank with method='first' and ascending=False gives wrong ranks for tied values
[ https://issues.apache.org/jira/browse/SPARK-59011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-59011:
-----------------------------------
Labels: pull-request-available (was: )
> Series.rank with method='first' and ascending=False gives wrong ranks for tied values
> -------------------------------------------------------------------------------------
>
> Key: SPARK-59011
> URL: https://issues.apache.org/jira/browse/SPARK-59011
> Project: Spark
> Issue Type: Task
> Components: Pandas API on Spark
> Affects Versions: 4.2.1
> Reporter: Jing Zhan
> Priority: Major
> Labels: pull-request-available
>
> Series.rank(method='first', ascending=False) produces incorrect ranks for tied values in the pandas API on Spark. The tiebreaker should always favor the element that appeared first in the original array (lower index), but Spark incorrectly reverses the tiebreaker direction when ascending=False.
> Reproduction:
> import pandas as pd
> import pyspark.pandas as ps
> pser = pd.Series([1, 2, 3, 1], name='x')
> psser = ps.from_pandas(pser)
> print(pser.rank(method='first', ascending=False).tolist()) # [3.0, 2.0, 1.0, 4.0]
> print(psser.rank(method='first', ascending=False).sort_index().tolist()) # [4.0, 2.0, 1.0, 3.0]
> Expected: [3.0, 2.0, 1.0, 4.0] (matches pandas — index 0 gets rank 3, index 3 gets rank 4)
> Actual: [4.0, 2.0, 1.0, 3.0] (wrong — index 0 gets rank 4, index 3 gets rank 3)
> Root cause: In Series._rank, the natural order tiebreaker column uses the same direction as the value sort (DESC when ascending=False). This reverses the tiebreaker for tied values. The fix is to always use ASC for the natural order tiebreaker since "first" always means first in the original array, regardless of rank direction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org