You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/05/06 05:37:00 UTC
[jira] [Commented] (SPARK-35282) Support AQE side shuffled hash
join formula
[ https://issues.apache.org/jira/browse/SPARK-35282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17340004#comment-17340004 ]
Apache Spark commented on SPARK-35282:
--------------------------------------
User 'ulysses-you' has created a pull request for this issue:
https://github.com/apache/spark/pull/32450
> Support AQE side shuffled hash join formula
> -------------------------------------------
>
> Key: SPARK-35282
> URL: https://issues.apache.org/jira/browse/SPARK-35282
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: ulysses you
> Priority: Major
>
> Use AQE runtime statistics to decide if we can use shuffled hash join instead of sort merge join. Currently, the formula of shuffled hash join selection does not work due to the dymanic shuffle partition number.
>
> Add a new config `spark.sql.adaptive.shuffledHashJoinLocalMapThreshold` to decide if join can be converted to shuffled hash join safely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org