You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2022/08/17 06:59:01 UTC
[spark] branch branch-3.2 updated: [SPARK-39887][SQL][FOLLOW-UP] Do not exclude Union's first child attributes when traversing other children in RemoveRedundantAliases
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new f96fc06ca4a [SPARK-39887][SQL][FOLLOW-UP] Do not exclude Union's first child attributes when traversing other children in RemoveRedundantAliases
f96fc06ca4a is described below
commit f96fc06ca4a5a1bbc154aca1c7a9b96e7fc42c7e
Author: Peter Toth <pt...@cloudera.com>
AuthorDate: Wed Aug 17 14:57:35 2022 +0800
[SPARK-39887][SQL][FOLLOW-UP] Do not exclude Union's first child attributes when traversing other children in RemoveRedundantAliases
### What changes were proposed in this pull request?
Do not exclude `Union`'s first child attributes when traversing other children in `RemoveRedundantAliases`.
### Why are the changes needed?
We don't need to exclude those attributes that `Union` inherits from its first child. See discussion here: https://github.com/apache/spark/pull/37496#discussion_r944509115
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing UTs.
Closes #37534 from peter-toth/SPARK-39887-keep-attributes-of-unions-first-child-follow-up.
Authored-by: Peter Toth <pt...@cloudera.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit e732232dac420826af269d8cf5efacb52933f59a)
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
.../scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index 7993829f1f6..6797bbfc714 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -536,7 +536,7 @@ object RemoveRedundantAliases extends Rule[LogicalPlan] {
})
Join(newLeft, newRight, joinType, newCondition, hint)
- case _: Union =>
+ case u: Union =>
var first = true
plan.mapChildren { child =>
if (first) {
@@ -547,7 +547,8 @@ object RemoveRedundantAliases extends Rule[LogicalPlan] {
// output attributes could return incorrect result.
removeRedundantAliases(child, excluded ++ child.outputSet)
} else {
- removeRedundantAliases(child, excluded)
+ // We don't need to exclude those attributes that `Union` inherits from its first child.
+ removeRedundantAliases(child, excluded -- u.children.head.outputSet)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org