[GitHub] [spark] cloud-fan commented on pull request #35290: [SPARK-37865][SQL][3.0]Fix union bug when the first child of union has duplicate columns

[GitBox <gi...@apache.org>]

cloud-fan commented on pull request #35290:
URL: https://github.com/apache/spark/pull/35290#issuecomment-1019849857


   Is it a breaking change? Seems it will break `val df = ...; df.select("a", "a").select(df("a"));`
   
   I'd like to have a surgical fix for this bug. Ideally, when a plan has conflicting output attr ids, the columns with the same attr id should output the same value, so it doesn't matter which column you bind to at the end. However, `Union` breaks this assumption. We don't want to fix `Union` in old branches as it's a breaking change, and we should fix the attribute binding logic for `Union` to minimize the impact.
   
   My proposal is:
   1. For SQL API, the plan is `Distinct(Union(A, B))`. In the rule `ReplaceDistinctWithAggregate`, we replace `Distinct` with a group-only `Aggregate`, and take all the output attrs of `Union` as the grouping columns. We can add a special metadata property to `AttributeReference`, to indicate the actual column it wants to bind. e.g. `__index_of_attributes_with_same_id=1` means it should bind to the second column that has this attribute id, and we update `BindReference` accordingly to implement this special binding logic.
   2. For dataframe API (`df.distinct`), the plan is `Deduplicate(attrs, Union(A, B))`. We can use the same idea to add special metadata property in `df.distinct`.
   3. For `PushProjectionThroughUnion`, we skip the optimization if Union has conflicting attr ids in its output.
   
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org