[GitHub] [spark] allisonwang-db commented on a change in pull request #30093: [SPARK-33183][SQL] Fix EliminateSorts bug when removing global sorts

[GitBox <gi...@apache.org>]

allisonwang-db commented on a change in pull request #30093:
URL: https://github.com/apache/spark/pull/30093#discussion_r509898482



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1052,11 +1052,11 @@ object CombineFilters extends Rule[LogicalPlan] with PredicateHelper {
  *    function is order irrelevant
  */
 object EliminateSorts extends Rule[LogicalPlan] {
-  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {

Review comment:
       Yes. This case is tricky. Let's consider this example: `df.sortBy('a.asc).sortBy('a.asc).sortBy('a.asc)`
   ```
   Sort(a, false, _)  // A
     Sort(a, false, _)  // B
       Sort(a, false, _)  // C
         ...
   ```
   If we don't use `transformUp`here, the rule will keep both Sort B and C since it will not apply the rule to A's child (B) again. Plan after running this rule will be:
   ```
   Sort(a, false, _) // B
     Sort(a, false, _) // C
        ...
   ```
   And the `Once` strategy's idempotency will be broken since Sort B will be removed if we run this rule again.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org