[spark] branch master updated: [SPARK-28530][SQL] Cost-based join reorder optimizer batch should be FixedPoint(1)

[li...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

lixiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new d4e2466  [SPARK-28530][SQL] Cost-based join reorder optimizer batch should be FixedPoint(1)
d4e2466 is described below

commit d4e246658a308e2546d89e7ec9bbd9b6ca7517f7
Author: Yesheng Ma <ki...@gmail.com>
AuthorDate: Fri Jul 26 22:57:39 2019 -0700

    [SPARK-28530][SQL] Cost-based join reorder optimizer batch should be FixedPoint(1)
    
    ## What changes were proposed in this pull request?
    Since for AQP the cost for joins can change between multiple runs, there is no reason that we have an idempotence enforcement on this optimizer batch. We thus make it `FixedPoint(1)` instead of `Once`.
    
    ## How was this patch tested?
    Existing UTs.
    
    Closes #25266 from yeshengm/SPARK-28530.
    
    Lead-authored-by: Yesheng Ma <ki...@gmail.com>
    Co-authored-by: Xiao Li <ga...@gmail.com>
    Signed-off-by: gatorsmile <ga...@gmail.com>
---
 .../main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala | 3 +--
 .../org/apache/spark/sql/catalyst/optimizer/JoinReorderSuite.scala     | 2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index 3efc41d..670fc92 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -49,7 +49,6 @@ abstract class Optimizer(sessionCatalog: SessionCatalog)
 
   override protected val blacklistedOnceBatches: Set[String] =
     Set("Pullup Correlated Expressions",
-      "Join Reorder",
       "Extract Python UDFs"
     )
 
@@ -168,7 +167,7 @@ abstract class Optimizer(sessionCatalog: SessionCatalog)
       RemoveLiteralFromGroupExpressions,
       RemoveRepetitionFromGroupExpressions) :: Nil ++
     operatorOptimizationBatch) :+
-    Batch("Join Reorder", Once,
+    Batch("Join Reorder", FixedPoint(1),
       CostBasedJoinReorder) :+
     Batch("Remove Redundant Sorts", Once,
       RemoveRedundantSorts) :+
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/JoinReorderSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/JoinReorderSuite.scala
index 43e5bad..f775500 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/JoinReorderSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/JoinReorderSuite.scala
@@ -40,7 +40,7 @@ class JoinReorderSuite extends PlanTest with StatsEstimationTestBase {
         PushPredicateThroughJoin,
         ColumnPruning,
         CollapseProject) ::
-      Batch("Join Reorder", Once,
+      Batch("Join Reorder", FixedPoint(1),
         CostBasedJoinReorder) :: Nil
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org