You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Mayur Bhosale (Jira)" <ji...@apache.org> on 2020/07/14 08:04:00 UTC

[jira] [Created] (SPARK-32299) Decide SMJ Join Orientation adaptively

Mayur Bhosale created SPARK-32299:
-------------------------------------

             Summary: Decide SMJ Join Orientation adaptively
                 Key: SPARK-32299
                 URL: https://issues.apache.org/jira/browse/SPARK-32299
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.1.0
            Reporter: Mayur Bhosale
             Fix For: 3.1.0


For SortMerge join of type EquiJoin, the left and right side of the joins are decided on the basis of the user order. In SMJ, the left side of the join is streamed and the right side is buffered (matching values). Because of this, B SMJ A would perform better than A SMJ B if, sizeOf(B) > sizeOf(A)

 

With adaptive query execution, once both ShuffleQueryStages corresponding to the join have completed and if none of them have sizes lesser than the broadcast threshold, join orientation can be changed at run time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org