You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "mihailotim-db (via GitHub)" <gi...@apache.org> on 2025/07/04 12:45:17 UTC
[PR] ] [spark]
mihailotim-db opened a new pull request, #51376:
URL: https://github.com/apache/spark/pull/51376
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192228881
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -895,6 +897,24 @@ object LimitPushDown extends Rule[LogicalPlan] {
*/
object PushProjectionThroughUnion extends Rule[LogicalPlan] {
+ /**
+ * When pushing a [[Project]] through [[Union]] we need to maintain the invariant that
+ * [[Union]] children must have unique [[ExprId]]s per branch.
Review Comment:
It does and that is the issue. We are going to deduplicate ids there, but then get to optimizer and add duplicate ids again. For example if we are pushing down `Project(col1#1, col1#1)` we are going to add it to both branches of Union, making the Union unresolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2206524603
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -4963,6 +4963,45 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
)
}
+ test("SPARK-52686: Union should be resolved only if there are no duplicates") {
Review Comment:
The newly added test has been causing failures in the Non-Ansi daily test since 5 days ago:
- https://github.com/apache/spark/actions/runs/16209775973/job/45767565421
<img width="1644" height="554" alt="image" src="https://github.com/user-attachments/assets/0538a97e-f0da-4a6d-9242-28b25fc870ab" />
We've reproduced it locally using the following command:
```
SPARK_ANSI_SQL_MODE=false build/sbt clean "sql/testOnly org.apache.spark.sql.SQLQuerySuite"
```
```
[info] - SPARK-52686: Union should be resolved only if there are no duplicates *** FAILED *** (143 milliseconds)
[info] 9 did not equal 8 (SQLQuerySuite.scala:4998)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info] at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info] at org.apache.spark.sql.SQLQuerySuite.$anonfun$new$1023(SQLQuerySuite.scala:4998)
[info] at org.apache.spark.sql.catalyst.SQLConfHelper.withSQLConf(SQLConfHelper.scala:56)
[info] at org.apache.spark.sql.catalyst.SQLConfHelper.withSQLConf$(SQLConfHelper.scala:38)
[info] at org.apache.spark.sql.SQLQuerySuite.org$apache$spark$sql$test$SQLTestUtilsBase$$super$withSQLConf(SQLQuerySuite.scala:63)
[info] at org.apache.spark.sql.test.SQLTestUtilsBase.withSQLConf(SQLTestUtils.scala:253)
[info] at org.apache.spark.sql.test.SQLTestUtilsBase.withSQLConf$(SQLTestUtils.scala:251)
[info] at org.apache.spark.sql.SQLQuerySuite.withSQLConf(SQLQuerySuite.scala:63)
[info] at org.apache.spark.sql.SQLQuerySuite.$anonfun$new$1022(SQLQuerySuite.scala:4975)
[info] at org.apache.spark.sql.SQLQuerySuite.$anonfun$new$1022$adapted(SQLQuerySuite.scala:4972)
[info] at scala.collection.immutable.List.foreach(List.scala:334)
[info] at org.apache.spark.sql.SQLQuerySuite.$anonfun$new$1021(SQLQuerySuite.scala:4972)
```
Could you take a look at this issue when you have time? thanks @mihailotim-db
also cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "vladimirg-db (via GitHub)" <gi...@apache.org>.
vladimirg-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192168003
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -895,6 +897,24 @@ object LimitPushDown extends Rule[LogicalPlan] {
*/
object PushProjectionThroughUnion extends Rule[LogicalPlan] {
+ /**
+ * When pushing a [[Project]] through [[Union]] we need to maintain the invariant that
+ * [[Union]] children must have unique [[ExprId]]s per branch.
+ */
+ private def deduplicateProjectList(projectList: Seq[NamedExpression]) = {
+ val existingExprIds = new HashSet[ExprId]
+ projectList.map(attr => if (existingExprIds.contains(attr.exprId)) {
+ val newMetadata = new MetadataBuilder()
+ .withMetadata(attr.metadata)
+ .putNull("__is_duplicate")
Review Comment:
Yeah, @mihailotim-db since you are stripping it away in your other PR, we should not add it in the Optimizer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2190179081
##########
sql/connect/common/src/test/resources/query-tests/explain-results/union.explain:
##########
@@ -1,3 +1,3 @@
-Union false, false
+'Union false, false
Review Comment:
Yes, not sure how to change the input plan but seems this is common across all `ProtoToParsedPlanSuite` inputs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192881482
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -241,6 +241,16 @@ object SQLConf {
}
}
+ val UNION_IS_RESOLVED_WHEN_DUPLICATES_PER_CHILD_RESOLVED =
+ buildConf("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved")
+ .internal()
+ .doc(
+ "When true, union should only be resolved once there are no duplicate attributes in " +
+ "each branch."
+ )
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192874067
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -241,6 +241,16 @@ object SQLConf {
}
}
+ val UNION_IS_RESOLVED_WHEN_DUPLICATES_PER_CHILD_RESOLVED =
+ buildConf("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved")
+ .internal()
+ .doc(
+ "When true, union should only be resolved once there are no duplicate attributes in " +
+ "each branch."
+ )
Review Comment:
nit. Please move this `)` into the above line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192480796
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -923,10 +943,15 @@ object PushProjectionThroughUnion extends Rule[LogicalPlan] {
}
def pushProjectionThroughUnion(projectList: Seq[NamedExpression], u: Union): Seq[LogicalPlan] = {
- val newFirstChild = Project(projectList, u.children.head)
+ val deduplicatedProjectList = if (conf.unionIsResolvedWhenDuplicatesPerChildResolved) {
+ deduplicateProjectList(projectList)
Review Comment:
yea column pruning will work. And we can't remove them if the `Project` is the root node, as it will change the final result schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2185327148
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:
##########
@@ -620,7 +620,11 @@ case class Union(
}
override lazy val resolved: Boolean = {
- children.length > 1 && !(byName || allowMissingCol) && childrenResolved && allChildrenCompatible
+ children.length > 1 &&
+ !(byName || allowMissingCol) &&
+ childrenResolved &&
+ allChildrenCompatible &&
+ duplicateResolved
Review Comment:
This looks fine to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2185305597
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:
##########
@@ -620,7 +620,11 @@ case class Union(
}
override lazy val resolved: Boolean = {
- children.length > 1 && !(byName || allowMissingCol) && childrenResolved && allChildrenCompatible
+ children.length > 1 &&
+ !(byName || allowMissingCol) &&
+ childrenResolved &&
+ allChildrenCompatible &&
+ duplicateResolved
Review Comment:
We can, but I am not sure if this is needed. This behavior is already present for `Except` and `Intersect` but we are missing that here for some reason. @cloud-fan What do you think? Do we need a flag here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192481534
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -923,10 +943,15 @@ object PushProjectionThroughUnion extends Rule[LogicalPlan] {
}
def pushProjectionThroughUnion(projectList: Seq[NamedExpression], u: Union): Seq[LogicalPlan] = {
- val newFirstChild = Project(projectList, u.children.head)
+ val deduplicatedProjectList = if (conf.unionIsResolvedWhenDuplicatesPerChildResolved) {
+ deduplicateProjectList(projectList)
Review Comment:
let's add some code comments to explain this tricky case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192502431
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -923,10 +943,15 @@ object PushProjectionThroughUnion extends Rule[LogicalPlan] {
}
def pushProjectionThroughUnion(projectList: Seq[NamedExpression], u: Union): Seq[LogicalPlan] = {
- val newFirstChild = Project(projectList, u.children.head)
+ val deduplicatedProjectList = if (conf.unionIsResolvedWhenDuplicatesPerChildResolved) {
+ deduplicateProjectList(projectList)
Review Comment:
We have a scaladoc for `deduplicateProjectList`. Do we need to expand on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2206559683
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -4963,6 +4963,45 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
)
}
+ test("SPARK-52686: Union should be resolved only if there are no duplicates") {
Review Comment:
Followup to fix the test: https://github.com/apache/spark/pull/51490
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2206536842
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -4963,6 +4963,45 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
)
}
+ test("SPARK-52686: Union should be resolved only if there are no duplicates") {
Review Comment:
Ah yes, the test is dependent on `WidenSetOperationTypes` triggering and I guess that happens only with ANSI false. Should we just set ANSI to true for this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192164646
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -895,6 +897,24 @@ object LimitPushDown extends Rule[LogicalPlan] {
*/
object PushProjectionThroughUnion extends Rule[LogicalPlan] {
+ /**
+ * When pushing a [[Project]] through [[Union]] we need to maintain the invariant that
+ * [[Union]] children must have unique [[ExprId]]s per branch.
+ */
+ private def deduplicateProjectList(projectList: Seq[NamedExpression]) = {
+ val existingExprIds = new HashSet[ExprId]
+ projectList.map(attr => if (existingExprIds.contains(attr.exprId)) {
+ val newMetadata = new MetadataBuilder()
+ .withMetadata(attr.metadata)
+ .putNull("__is_duplicate")
Review Comment:
why do we need this metadata?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192225617
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -923,10 +943,15 @@ object PushProjectionThroughUnion extends Rule[LogicalPlan] {
}
def pushProjectionThroughUnion(projectList: Seq[NamedExpression], u: Union): Seq[LogicalPlan] = {
- val newFirstChild = Project(projectList, u.children.head)
+ val deduplicatedProjectList = if (conf.unionIsResolvedWhenDuplicatesPerChildResolved) {
+ deduplicateProjectList(projectList)
Review Comment:
We don't because these new attributes are going to be unused actually. For example if we have `col1#1, col1#1`, we can safely resinstantiate to `col1#1, col1 as col1#2`, because we would still have reference to `col1#1` available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192502431
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -923,10 +943,15 @@ object PushProjectionThroughUnion extends Rule[LogicalPlan] {
}
def pushProjectionThroughUnion(projectList: Seq[NamedExpression], u: Union): Seq[LogicalPlan] = {
- val newFirstChild = Project(projectList, u.children.head)
+ val deduplicatedProjectList = if (conf.unionIsResolvedWhenDuplicatesPerChildResolved) {
+ deduplicateProjectList(projectList)
Review Comment:
We have a scaladoc for `deduplicateProjectList` and an extensive PR description with examples. Do we need to expand on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192683834
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -923,10 +943,15 @@ object PushProjectionThroughUnion extends Rule[LogicalPlan] {
}
def pushProjectionThroughUnion(projectList: Seq[NamedExpression], u: Union): Seq[LogicalPlan] = {
- val newFirstChild = Project(projectList, u.children.head)
+ val deduplicatedProjectList = if (conf.unionIsResolvedWhenDuplicatesPerChildResolved) {
+ deduplicateProjectList(projectList)
Review Comment:
Expanded the scaladoc for `deduplicateProjectList`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192235337
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -895,6 +897,24 @@ object LimitPushDown extends Rule[LogicalPlan] {
*/
object PushProjectionThroughUnion extends Rule[LogicalPlan] {
+ /**
+ * When pushing a [[Project]] through [[Union]] we need to maintain the invariant that
+ * [[Union]] children must have unique [[ExprId]]s per branch.
+ */
+ private def deduplicateProjectList(projectList: Seq[NamedExpression]) = {
+ val existingExprIds = new HashSet[ExprId]
+ projectList.map(attr => if (existingExprIds.contains(attr.exprId)) {
+ val newMetadata = new MetadataBuilder()
+ .withMetadata(attr.metadata)
+ .putNull("__is_duplicate")
Review Comment:
Good point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192244603
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -923,10 +943,15 @@ object PushProjectionThroughUnion extends Rule[LogicalPlan] {
}
def pushProjectionThroughUnion(projectList: Seq[NamedExpression], u: Union): Seq[LogicalPlan] = {
- val newFirstChild = Project(projectList, u.children.head)
+ val deduplicatedProjectList = if (conf.unionIsResolvedWhenDuplicatesPerChildResolved) {
+ deduplicateProjectList(projectList)
Review Comment:
so the new aliased attribute is not referenced at all, unless this Project is the root node?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #51376:
URL: https://github.com/apache/spark/pull/51376#issuecomment-3047536308
@mihailotim-db can you push a commit with the proposed change so that we can review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "vladimirg-db (via GitHub)" <gi...@apache.org>.
vladimirg-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2185301004
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:
##########
@@ -620,7 +620,11 @@ case class Union(
}
override lazy val resolved: Boolean = {
- children.length > 1 && !(byName || allowMissingCol) && childrenResolved && allChildrenCompatible
+ children.length > 1 &&
+ !(byName || allowMissingCol) &&
+ childrenResolved &&
+ allChildrenCompatible &&
+ duplicateResolved
Review Comment:
Any chance we can put this under a flag? This could probably change the results of some queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2190133765
##########
sql/connect/common/src/test/resources/query-tests/explain-results/union.explain:
##########
@@ -1,3 +1,3 @@
-Union false, false
+'Union false, false
Review Comment:
it's unresolved now because the normalized plan uses 0 expr id for all attributes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #51376: [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates
URL: https://github.com/apache/spark/pull/51376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2206524603
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -4963,6 +4963,45 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
)
}
+ test("SPARK-52686: Union should be resolved only if there are no duplicates") {
Review Comment:
The newly added test has been causing failures in the Non-Ansi daily test since 5 days ago:
- https://github.com/apache/spark/actions/runs/16209775973/job/45767565421
<img width="1644" height="554" alt="image" src="https://github.com/user-attachments/assets/0538a97e-f0da-4a6d-9242-28b25fc870ab" />
We can reproduce it locally using the following command:
```
SPARK_ANSI_SQL_MODE=false build/sbt clean "sql/testOnly org.apache.spark.sql.SQLQuerySuite"
```
```
[info] - SPARK-52686: Union should be resolved only if there are no duplicates *** FAILED *** (143 milliseconds)
[info] 9 did not equal 8 (SQLQuerySuite.scala:4998)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info] at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info] at org.apache.spark.sql.SQLQuerySuite.$anonfun$new$1023(SQLQuerySuite.scala:4998)
[info] at org.apache.spark.sql.catalyst.SQLConfHelper.withSQLConf(SQLConfHelper.scala:56)
[info] at org.apache.spark.sql.catalyst.SQLConfHelper.withSQLConf$(SQLConfHelper.scala:38)
[info] at org.apache.spark.sql.SQLQuerySuite.org$apache$spark$sql$test$SQLTestUtilsBase$$super$withSQLConf(SQLQuerySuite.scala:63)
[info] at org.apache.spark.sql.test.SQLTestUtilsBase.withSQLConf(SQLTestUtils.scala:253)
[info] at org.apache.spark.sql.test.SQLTestUtilsBase.withSQLConf$(SQLTestUtils.scala:251)
[info] at org.apache.spark.sql.SQLQuerySuite.withSQLConf(SQLQuerySuite.scala:63)
[info] at org.apache.spark.sql.SQLQuerySuite.$anonfun$new$1022(SQLQuerySuite.scala:4975)
[info] at org.apache.spark.sql.SQLQuerySuite.$anonfun$new$1022$adapted(SQLQuerySuite.scala:4972)
[info] at scala.collection.immutable.List.foreach(List.scala:334)
[info] at org.apache.spark.sql.SQLQuerySuite.$anonfun$new$1021(SQLQuerySuite.scala:4972)
```
Could you take a look at this issue when you have time? thanks @mihailotim-db
also cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on PR #51376:
URL: https://github.com/apache/spark/pull/51376#issuecomment-3045910315
There are some optimizer tests failing because `PushProjectionThroughUnion` doesn't take into account that project list must contain unique `ExprId`s even though we have an Analyzer rule that does deduplication. I think we can simply alias these duplicate IDs without in place without adding an extra project. What do you think @cloud-fan @dongjoon-hyun ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on PR #51376:
URL: https://github.com/apache/spark/pull/51376#issuecomment-3047745957
@cloud-fan Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192170814
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -923,10 +943,15 @@ object PushProjectionThroughUnion extends Rule[LogicalPlan] {
}
def pushProjectionThroughUnion(projectList: Seq[NamedExpression], u: Union): Seq[LogicalPlan] = {
- val newFirstChild = Project(projectList, u.children.head)
+ val deduplicatedProjectList = if (conf.unionIsResolvedWhenDuplicatesPerChildResolved) {
+ deduplicateProjectList(projectList)
Review Comment:
This changes the output attribute ids, don't we need to update all downstream operators to use the new attribute ids?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192198697
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -895,6 +897,24 @@ object LimitPushDown extends Rule[LogicalPlan] {
*/
object PushProjectionThroughUnion extends Rule[LogicalPlan] {
+ /**
+ * When pushing a [[Project]] through [[Union]] we need to maintain the invariant that
+ * [[Union]] children must have unique [[ExprId]]s per branch.
Review Comment:
Doesn't `DeduplicateRelations` handle the conflicting attributes within a Project?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on code in PR #51376:
URL: https://github.com/apache/spark/pull/51376#discussion_r2192273194
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -923,10 +943,15 @@ object PushProjectionThroughUnion extends Rule[LogicalPlan] {
}
def pushProjectionThroughUnion(projectList: Seq[NamedExpression], u: Union): Seq[LogicalPlan] = {
- val newFirstChild = Project(projectList, u.children.head)
+ val deduplicatedProjectList = if (conf.unionIsResolvedWhenDuplicatesPerChildResolved) {
+ deduplicateProjectList(projectList)
Review Comment:
Exactly. I was thinking of puning them away entirely, but maybe best not to change plan shape (they might get removed later anyways)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "mihailotim-db (via GitHub)" <gi...@apache.org>.
mihailotim-db commented on PR #51376:
URL: https://github.com/apache/spark/pull/51376#issuecomment-3049429570
@peter-toth Config is a safety net in case there are other cases in Optimizer that can make Union unresolved (similar to `PushProjectionThroughUnion`). I am not aware of any other rules, but better be safe in this case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #51376:
URL: https://github.com/apache/spark/pull/51376#issuecomment-3050904305
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org