You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Eugene Koifman (Jira)" <ji...@apache.org> on 2021/11/02 03:34:00 UTC

[jira] [Created] (SPARK-37193) DynamicJoinSelection.shouldDemoteBroadcastHashJoin should not apply to outer joins

Eugene Koifman created SPARK-37193:
--------------------------------------

             Summary: DynamicJoinSelection.shouldDemoteBroadcastHashJoin should not apply to outer joins
                 Key: SPARK-37193
                 URL: https://issues.apache.org/jira/browse/SPARK-37193
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.1.1
            Reporter: Eugene Koifman


{{DynamicJoinSelection.shouldDemoteBroadcastHashJoin}} will prevent AQE from converting Sort merge join into a broadcast join because SMJ is faster when the side that would be broadcast has a lot of empty partitions.
 This makes sense for inner joins which can short circuit if one side is empty.
 For (left,right) outer join, the streaming side still has to be processed so demoting broadcast join doesn't have the same advantage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org