You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jiaan Geng (Jira)" <ji...@apache.org> on 2025/12/17 03:32:00 UTC
[jira] [Updated] (SPARK-54726) Improve a bit of performance for InsertAdaptiveSparkPlan
[ https://issues.apache.org/jira/browse/SPARK-54726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiaan Geng updated SPARK-54726:
-------------------------------
Description:
The implementation of shouldApplyAQE show below.
{code:java}
private def shouldApplyAQE(plan: SparkPlan, isSubquery: Boolean): Boolean = {
conf.getConf(SQLConf.ADAPTIVE_EXECUTION_FORCE_APPLY) || isSubquery || {
plan.exists {
case _: Exchange => true
case p if !p.requiredChildDistribution.forall(_ == UnspecifiedDistribution) => true
// AQE framework has a different way to update the query plan in the UI: it updates the plan
// at the end of execution, while non-AQE updates the plan before execution. If the cached
// plan is already AQEed, the current plan must be AQEed as well so that the UI can get plan
// update correctly.
case i: InMemoryTableScanExec
if i.relation.cachedPlan.isInstanceOf[AdaptiveSparkPlanExec] => true
case p => p.expressions.exists(_.exists {
case _: SubqueryExpression => true
case _ => false
})
}
}
}
{code}
We can improve the line.
{code:java}
case p if !p.requiredChildDistribution.forall(_ == UnspecifiedDistribution) => true
{code}
> Improve a bit of performance for InsertAdaptiveSparkPlan
> --------------------------------------------------------
>
> Key: SPARK-54726
> URL: https://issues.apache.org/jira/browse/SPARK-54726
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Jiaan Geng
> Assignee: Jiaan Geng
> Priority: Major
>
> The implementation of shouldApplyAQE show below.
> {code:java}
> private def shouldApplyAQE(plan: SparkPlan, isSubquery: Boolean): Boolean = {
> conf.getConf(SQLConf.ADAPTIVE_EXECUTION_FORCE_APPLY) || isSubquery || {
> plan.exists {
> case _: Exchange => true
> case p if !p.requiredChildDistribution.forall(_ == UnspecifiedDistribution) => true
> // AQE framework has a different way to update the query plan in the UI: it updates the plan
> // at the end of execution, while non-AQE updates the plan before execution. If the cached
> // plan is already AQEed, the current plan must be AQEed as well so that the UI can get plan
> // update correctly.
> case i: InMemoryTableScanExec
> if i.relation.cachedPlan.isInstanceOf[AdaptiveSparkPlanExec] => true
> case p => p.expressions.exists(_.exists {
> case _: SubqueryExpression => true
> case _ => false
> })
> }
> }
> }
> {code}
> We can improve the line.
> {code:java}
> case p if !p.requiredChildDistribution.forall(_ == UnspecifiedDistribution) => true
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org