You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Cheng Su (Jira)" <ji...@apache.org> on 2021/04/21 22:53:00 UTC
[jira] [Commented] (SPARK-35179) Introduce hybrid join for sort
merge join and shuffled hash join in AQE
[ https://issues.apache.org/jira/browse/SPARK-35179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326991#comment-17326991 ]
Cheng Su commented on SPARK-35179:
----------------------------------
Thanks for [~cloud_fan] for the idea. Please comment or edit if this is not captured correctly, thanks.
> Introduce hybrid join for sort merge join and shuffled hash join in AQE
> -----------------------------------------------------------------------
>
> Key: SPARK-35179
> URL: https://issues.apache.org/jira/browse/SPARK-35179
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Cheng Su
> Priority: Minor
>
> Per discussion in [https://github.com/apache/spark/pull/32210#issuecomment-823503243] , we can introduce some kind of {{HybridJoin}} operator in AQE, and we can choose to do shuffled hash join vs sort merge join for each task independently, e.g. based on partition size, task1 can do shuffled hash join, and task2 can do sort merge join, etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org