[GitHub] [spark] leanken commented on a change in pull request #29614: [SPARK-32765][SQL] EliminateJoinToEmptyRelation should respect exchange behavior when canChangeNumPartitions == false

[GitBox <gi...@apache.org>]

leanken commented on a change in pull request #29614:
URL: https://github.com/apache/spark/pull/29614#discussion_r481735280



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/EliminateJoinToEmptyRelation.scala
##########
@@ -36,22 +36,31 @@ import org.apache.spark.sql.execution.joins.{EmptyHashedRelation, HashedRelation
  */
 object EliminateJoinToEmptyRelation extends Rule[LogicalPlan] {
 
-  private def canEliminate(plan: LogicalPlan, relation: HashedRelation): Boolean = plan match {
-    case LogicalQueryStage(_, stage: BroadcastQueryStageExec) if stage.resultOption.get().isDefined
-      && stage.broadcast.relationFuture.get().value == relation => true
-    case _ => false
+  private def canEliminate(streamedPlan: LogicalPlan,
+      buildPlan: LogicalPlan, relation: HashedRelation): Boolean = {
+    // If streamedSide of the Join is ShuffleQueryStageExec(canChangeNumPartitions== false)
+    // it can't be rewritten to EmptyRelation because the conversion will lost user specified
+    // number partition information.
+    (streamedPlan, buildPlan) match {
+      case (LogicalQueryStage(_, stage: ShuffleQueryStageExec), _)

Review comment:
       let me try and reply later.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org