You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/14 08:06:41 UTC
[GitHub] [spark] JkSelf commented on a change in pull request #32861: [SPARK-35710] [SQL] Support DPP + AQE when there is no reused broadcast exchange
JkSelf commented on a change in pull request #32861:
URL: https://github.com/apache/spark/pull/32861#discussion_r669386874
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
##########
@@ -62,8 +63,19 @@ case class PlanAdaptiveDynamicPruningFilters(
val broadcastValues = SubqueryBroadcastExec(
name, index, buildKeys, newAdaptivePlan)
DynamicPruningExpression(InSubqueryExec(value, broadcastValues, exprId))
- } else {
+ } else if (onlyInBroadcast) {
DynamicPruningExpression(Literal.TrueLiteral)
+ } else {
+ // we need to apply an aggregate on the buildPlan in order to be column pruned
+ val alias = Alias(buildKeys(index), buildKeys(index).toString)()
+ val aggregate = Aggregate(Seq(alias), Seq(alias), buildPlan)
+
+ val optimized = adaptivePlan.optimizer.execute(aggregate)
Review comment:
Yes. There is no need to call AQE optimizer here. Removed.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
##########
@@ -62,8 +63,19 @@ case class PlanAdaptiveDynamicPruningFilters(
val broadcastValues = SubqueryBroadcastExec(
name, index, buildKeys, newAdaptivePlan)
DynamicPruningExpression(InSubqueryExec(value, broadcastValues, exprId))
- } else {
+ } else if (onlyInBroadcast) {
DynamicPruningExpression(Literal.TrueLiteral)
+ } else {
+ // we need to apply an aggregate on the buildPlan in order to be column pruned
+ val alias = Alias(buildKeys(index), buildKeys(index).toString)()
+ val aggregate = Aggregate(Seq(alias), Seq(alias), buildPlan)
+
+ val optimized = adaptivePlan.optimizer.execute(aggregate)
+ val sparkPlan = adaptivePlan.context.session.sessionState.planner.
+ plan(ReturnAnswer(optimized)).next()
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org