[jira] [Resolved] (SPARK-35239) Coalesce shuffle partition should handle empty input RDD

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-35239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan resolved SPARK-35239.
---------------------------------
    Fix Version/s: 3.2.0
       Resolution: Fixed

Issue resolved by pull request 32362
[https://github.com/apache/spark/pull/32362]

>  Coalesce shuffle partition should handle empty input RDD
> ---------------------------------------------------------
>
>                 Key: SPARK-35239
>                 URL: https://issues.apache.org/jira/browse/SPARK-35239
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.2.0
>            Reporter: ulysses you
>            Assignee: ulysses you
>            Priority: Minor
>             Fix For: 3.2.0
>
>
> If input RDD partition is empty then the map output statistics will be null. And if all shuffle stage's input RDD partition is empty, we will skip it and lose the chance to coalesce partition.
>  
> We can simply create a empty partition for these custom shuffle reader to reduce the partition number.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org