[GitHub] [spark] maropu commented on a change in pull request #31404: [SPARK-34283][SQL] Combines all adjacent 'Union' operators into a single 'Union' when using 'Dataset.union.distinct.union.distinct'

[GitBox <gi...@apache.org>]

maropu commented on a change in pull request #31404:
URL: https://github.com/apache/spark/pull/31404#discussion_r573400885



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1007,6 +1007,8 @@ object CombineUnions extends Rule[LogicalPlan] {
   def apply(plan: LogicalPlan): LogicalPlan = plan transformDown {
     case u: Union => flattenUnion(u, false)
     case Distinct(u: Union) => Distinct(flattenUnion(u, true))
+    case Deduplicate(keys: Seq[Attribute], u: Union) if keys == u.output =>

Review comment:
       The current approach (@cloud-fan suggested in his comment) looks fine, but does this condition work correctly? What if they have a different order, `u.output` has new `exprId`s, ...?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org