[GitHub] [spark] tanelk commented on pull request #30093: [SPARK-33183][SQL] Fix EliminateSorts bug when removing global sorts

[GitBox <gi...@apache.org>]

tanelk commented on pull request #30093:
URL: https://github.com/apache/spark/pull/30093#issuecomment-713059186


   Shouldn't it be like this?
   
   ```diff
   diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
   index c37dac90c5..79610b4ab2 100644
   --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
   +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
   @@ -1058,15 +1058,9 @@ object EliminateSorts extends Rule[LogicalPlan] {
        case s @ Sort(orders, _, child) if orders.isEmpty || orders.exists(_.child.foldable) =>
          val newOrders = orders.filterNot(_.child.foldable)
          if (newOrders.isEmpty) child else s.copy(order = newOrders)
   -    case s @ Sort(orders, global, child)
   -        if SortOrder.orderingSatisfies(child.outputOrdering, orders) =>
   -      (global, child) match {
   -        case (false, _) => child
   -        case (true, r: Range) => r
   -        case (true, s @ Sort(_, true, _)) => s
   -        case (true, _) => s.copy(child = recursiveRemoveSort(child))
   -      }
   -    case s @ Sort(_, _, child) => s.copy(child = recursiveRemoveSort(child))
   +    case Sort(orders, false, child) if SortOrder.orderingSatisfies(child.outputOrdering, orders) =>
   +      child
   +    case s @ Sort(_, true, child) => s.copy(child = recursiveRemoveSort(child))
        case j @ Join(originLeft, originRight, _, cond, _) if cond.forall(_.deterministic) =>
          j.copy(left = recursiveRemoveSort(originLeft), right = recursiveRemoveSort(originRight))
        case g @ Aggregate(_, aggs, originChild) if isOrderIrrelevantAggs(aggs) =>
   ```
   
   Two simpler rules:
   * Remove local sort if child satisfies the required ordering
   * Keep global sorts, but remove all sorts from child
   
   This fails two tests:
   * Range test, but it is not correct because of what @viirya said
   * In the "remove two consecutive global sorts with same ordering" this keeps the parent sort, but in the test it keeps the child sort.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org