You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/01/13 13:33:45 UTC
[GitHub] [spark] chasingegg edited a comment on pull request #35168: [SPARK-37865][SQL]Spark should not dedup the grouping Expressions when the first child of union has duplicate columns
chasingegg edited a comment on pull request #35168:
URL: https://github.com/apache/spark/pull/35168#issuecomment-1012124307
> Yes this is a bug. Can you explain more about how this bug happens? Maybe adding alias is not the best way to fix it.
@cloud-fan The way of adding alias is to regard the same attribute as different exprId, Presto did the same way, so I use the solution. I have spent some time debugging the behavior why it would cause problems while the first child of union has duplicate attributes. And I provide a more concise example in description.
One part is in the optimization rule called PushProjectionThroughUnion,
In the buildReWrites, it will construct a map between the first child of union to the second child, for example, the first child has duplicate columns like **a, a**, it is completely the same, and the second child fetch **c and d** column, it will generate the map with **a -> d**，making the second child's outputs are all the same as **d column**.
But After I remove the rule, the behavior is that the second child's outputs are all the same as **c column**... I would need to take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org