[jira] [Commented] (SPARK-32299) Decide SMJ Join Orientation adaptively

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-32299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17157299#comment-17157299 ] 

Apache Spark commented on SPARK-32299:
--------------------------------------

User 'mayurdb' has created a pull request for this issue:
https://github.com/apache/spark/pull/29097

> Decide SMJ Join Orientation adaptively
> --------------------------------------
>
>                 Key: SPARK-32299
>                 URL: https://issues.apache.org/jira/browse/SPARK-32299
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.1.0
>            Reporter: Mayur Bhosale
>            Priority: Minor
>             Fix For: 3.1.0
>
>
> For SortMerge join of type EquiJoin, the left and right side of the joins are decided on the basis of the user order. In SMJ, the left side of the join is streamed and the right side is buffered (matching values). Because of this, B SMJ A would perform better than A SMJ B if, sizeOf(B) > sizeOf(A)
>  
> With adaptive query execution, once both ShuffleQueryStages corresponding to the join have completed and if none of them have sizes lesser than the broadcast threshold, join orientation can be changed at run time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org