[GitHub] [spark] Yikf commented on a diff in pull request #41064: [SPARK-43383][SQL] Add `rowCount` statistics to LocalRelation

["Yikf (via GitHub)" <gi...@apache.org>]

Yikf commented on code in PR #41064:
URL: https://github.com/apache/spark/pull/41064#discussion_r1187360291


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/joinReorder/JoinReorderSuite.scala:
##########
@@ -235,10 +235,10 @@ class JoinReorderSuite extends JoinReorderPlanTestBase with StatsEstimationTestB
     val tab3 = LocalRelation($"a".int, $"b".int)
     val original =
       tab1.join(tab2, Cross)
-          .join(tab3, Inner, Some($"a" === $"x" && $"b" === $"i"))
+          .join(tab3, Inner, Some($"a" === $"x"))
     val expected =
       tab1.join(tab3, Inner, Some($"a" === $"x"))
-          .join(tab2, Cross, Some($"b" === $"i"))

Review Comment:
   Now, this test case is affected by CBO because of this PR.  Since this PR adds rowCount statistics to LocalRelation, it will be affected by CBO according to [this code](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala#L64). 
   ```scala
   private def reorder(plan: LogicalPlan, output: Seq[Attribute]): LogicalPlan = {
       val (items, conditions) = extractInnerJoins(plan)
       val result =
         // Do reordering if the number of items is appropriate and join conditions exist.
         // We also need to check if costs of all items can be evaluated.
         if (items.size > 2 && items.size <= conf.joinReorderDPThreshold && conditions.nonEmpty &&
             items.forall(_.stats.rowCount.isDefined)) {
           JoinReorderDP.search(conf, items, conditions, output)
         } else {
           plan
         }
       // Set consecutive join nodes ordered.
       replaceWithOrderedJoin(result)
     }
   ``` 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org