[jira] [Updated] (SPARK-47284) We should ensure enough parallelism when ShuffleExchangeLike join with specs without shuffle

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-47284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun updated SPARK-47284:
----------------------------------
    Target Version/s:   (was: 4.0.0)

> We should ensure enough parallelism when ShuffleExchangeLike join with specs without shuffle
> --------------------------------------------------------------------------------------------
>
>                 Key: SPARK-47284
>                 URL: https://issues.apache.org/jira/browse/SPARK-47284
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.0.0
>            Reporter: Qi Zhu
>            Priority: Major
>
> The following case is introduced by https://issues.apache.org/jira/browse/SPARK-35703
> // When choosing specs, we should consider those children with no `ShuffleExchangeLike` node
> // first. For instance, if we have:
> // A: (No_Exchange, 100) <---> B: (Exchange, 120)
> // it's better to pick A and change B to (Exchange, 100) instead of picking B and insert a
> // new shuffle for A.
> *But we'd better improve it in some cases, for example:*
> A: (No_Exchange, 2) <---> B: (Exchange, 100)
> The current logic will change to:
> A: (No_Exchange, 2) <---> B: (Exchange,2)
> It actually not ensure enough parallelism, it will reduce the performance i think.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org