You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2019/10/23 17:21:00 UTC
[jira] [Assigned] (SPARK-29552) Fix the flaky test failed in
AdaptiveQueryExecSuite # multiple joins
[ https://issues.apache.org/jira/browse/SPARK-29552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-29552:
-----------------------------------
Assignee: Ke Jia
> Fix the flaky test failed in AdaptiveQueryExecSuite # multiple joins
> --------------------------------------------------------------------
>
> Key: SPARK-29552
> URL: https://issues.apache.org/jira/browse/SPARK-29552
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Ke Jia
> Assignee: Ke Jia
> Priority: Major
>
> AQE will optimize the logical plan once there is query stage finished. So for inner join, when two join side is all small to be the build side. The planner of converting logical plan to physical plan will select the build side as BuildRight if right side finished firstly or BuildLeft if left side finished firstly. In some case, when BuildRight or BuildLeft may introduce additional exchange to the parent node. The revert approach in OptimizeLocalShuffleReader rule may be too conservative, which revert all the local shuffle reader when introduce additional exchange not revert the local shuffle reader introduced shuffle. It may be expense to only revert the local shuffle reader introduced shuffle. The workaround is to apply the OptimizeLocalShuffleReader rule again when creating new query stage to further optimize the sub tree shuffle reader to local shuffle reader.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org