You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/18 07:54:25 UTC
[GitHub] [spark] viirya opened a new pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
viirya opened a new pull request #34038:
URL: https://github.com/apache/spark/pull/34038
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This patch proposes to generalize the resolving-by-position behavior to nested columns for Union.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Union, by the API definition, resolves columns by position. Currently we only follow this behavior at top-level columns, but not nested columns.
As we are making nested columns as first-class citizen, the nested-column-only limitation and the difference between top-level column and nested column do not make sense. We should also resolve nested columns like top-level columns for Union.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes. After this change, Union also resolves nested columns by position.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926289915
**[Test build #143580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143580/testReport)** for PR 34038 at commit [`a0af93c`](https://github.com/apache/spark/commit/a0af93cb5b8e319ef8dd87d1131fb91979e51e30).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926928227
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r715476863
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
##########
@@ -273,29 +274,37 @@ abstract class TypeCoercionBase {
@tailrec private def getWidestTypes(
children: Seq[LogicalPlan],
attrIndex: Int,
- castedTypes: mutable.Queue[DataType]): Seq[DataType] = {
+ castedTypes: mutable.Queue[Option[DataType]]): Seq[Option[DataType]] = {
// Return the result after the widen data types have been found for all the children
if (attrIndex >= children.head.output.length) return castedTypes.toSeq
// For the attrIndex-th attribute, find the widest type
findWiderCommonType(children.map(_.output(attrIndex).dataType)) match {
// If unable to find an appropriate widen type for this column, return an empty Seq
- case None => Seq.empty[DataType]
+ case None =>
+ castedTypes.enqueue(None)
+ getWidestTypes(children, attrIndex + 1, castedTypes)
// Otherwise, record the result in the queue and find the type for the next column
case Some(widenType) =>
- castedTypes.enqueue(widenType)
+ castedTypes.enqueue(Some(widenType))
getWidestTypes(children, attrIndex + 1, castedTypes)
Review comment:
the code can be simplified
```
findWiderCommonType(children.map(_.output(attrIndex).dataType)).map { widenTypeOpt =>
castedTypes.enqueue(widenTypeOpt)
getWidestTypes(children, attrIndex + 1, castedTypes)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927099090
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143620/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926927002
**[Test build #143607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143607/testReport)** for PR 34038 at commit [`80bb6e1`](https://github.com/apache/spark/commit/80bb6e135dddaa75aee1658e05681b992be91896).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927232746
**[Test build #143628 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143628/testReport)** for PR 34038 at commit [`f382cf2`](https://github.com/apache/spark/commit/f382cf27d1b9eb640129e08da3c2811af04cdc5f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926769095
**[Test build #143607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143607/testReport)** for PR 34038 at commit [`80bb6e1`](https://github.com/apache/spark/commit/80bb6e135dddaa75aee1658e05681b992be91896).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r716009611
##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/union.sql.out
##########
@@ -684,8 +684,8 @@ SELECT cast('3.4' as decimal(38, 18)) UNION SELECT 'foo'
-- !query schema
struct<>
-- !query output
-org.apache.spark.SparkException
-Failed to merge incompatible data types decimal(38,18) and string
+org.apache.spark.sql.AnalysisException
+Union can only be performed on tables with the compatible column types. string <> decimal(38,18) at the first column of the second table
Review comment:
Updated the error message. Please let me know if it looks good to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925471709
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48034/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922356065
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47944/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922241502
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47942/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922378826
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47946/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922236467
**[Test build #143434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143434/testReport)** for PR 34038 at commit [`f9c133c`](https://github.com/apache/spark/commit/f9c133c57a27ae9789d6c4382c11f369717b15e7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922236467
**[Test build #143434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143434/testReport)** for PR 34038 at commit [`f9c133c`](https://github.com/apache/spark/commit/f9c133c57a27ae9789d6c4382c11f369717b15e7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925365774
**[Test build #143520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143520/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922243518
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47942/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922401714
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143438/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r712593090
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,16 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
// Check if the data types match.
- dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
- // SPARK-18058: we shall not care about the nullability of columns
- if (TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty) {
- failAnalysis(
- s"""
- |${operator.nodeName} can only be performed on tables with the compatible
- |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
- |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
- """.stripMargin.replace("\n", " ").trim())
+ if (!isUnion) {
Review comment:
I think these set operations work basically the same. But at the API doc, we don't have document it for all set operations except for union. The by-position resolution for union, I think, is to follow SQL. It only requires the columns to union have the same data types in same order, but not column names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922365573
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925490991
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48034/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926289915
**[Test build #143580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143580/testReport)** for PR 34038 at commit [`a0af93c`](https://github.com/apache/spark/commit/a0af93cb5b8e319ef8dd87d1131fb91979e51e30).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925449012
**[Test build #143519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143519/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714479944
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
It is not always able to cast the types between children of union. For incompatible types, we need to find it out and throw analysis error here. Do I misunderstand it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #34038:
URL: https://github.com/apache/spark/pull/34038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926349536
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143580/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927619431
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927095678
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48132/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926305432
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48089/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927205558
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48139/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925472747
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143520/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925472747
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143520/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r715750877
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
##########
@@ -273,29 +274,37 @@ abstract class TypeCoercionBase {
@tailrec private def getWidestTypes(
children: Seq[LogicalPlan],
attrIndex: Int,
- castedTypes: mutable.Queue[DataType]): Seq[DataType] = {
+ castedTypes: mutable.Queue[Option[DataType]]): Seq[Option[DataType]] = {
// Return the result after the widen data types have been found for all the children
if (attrIndex >= children.head.output.length) return castedTypes.toSeq
// For the attrIndex-th attribute, find the widest type
findWiderCommonType(children.map(_.output(attrIndex).dataType)) match {
// If unable to find an appropriate widen type for this column, return an empty Seq
- case None => Seq.empty[DataType]
+ case None =>
+ castedTypes.enqueue(None)
+ getWidestTypes(children, attrIndex + 1, castedTypes)
// Otherwise, record the result in the queue and find the type for the next column
case Some(widenType) =>
- castedTypes.enqueue(widenType)
+ castedTypes.enqueue(Some(widenType))
getWidestTypes(children, attrIndex + 1, castedTypes)
Review comment:
Just simplified to
```scala
val widenTypeOpt = findWiderCommonType(children.map(_.output(attrIndex).dataType))
castedTypes.enqueue(widenTypeOpt)
getWidestTypes(children, attrIndex + 1, castedTypes)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926285172
**[Test build #143576 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143576/testReport)** for PR 34038 at commit [`a061497`](https://github.com/apache/spark/commit/a0614975564d0dca09077f7a8b50541dd5202e9c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r715749405
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
##########
@@ -273,29 +274,37 @@ abstract class TypeCoercionBase {
@tailrec private def getWidestTypes(
children: Seq[LogicalPlan],
attrIndex: Int,
- castedTypes: mutable.Queue[DataType]): Seq[DataType] = {
+ castedTypes: mutable.Queue[Option[DataType]]): Seq[Option[DataType]] = {
// Return the result after the widen data types have been found for all the children
if (attrIndex >= children.head.output.length) return castedTypes.toSeq
// For the attrIndex-th attribute, find the widest type
findWiderCommonType(children.map(_.output(attrIndex).dataType)) match {
// If unable to find an appropriate widen type for this column, return an empty Seq
- case None => Seq.empty[DataType]
+ case None =>
+ castedTypes.enqueue(None)
+ getWidestTypes(children, attrIndex + 1, castedTypes)
// Otherwise, record the result in the queue and find the type for the next column
case Some(widenType) =>
- castedTypes.enqueue(widenType)
+ castedTypes.enqueue(Some(widenType))
getWidestTypes(children, attrIndex + 1, castedTypes)
Review comment:
`findWiderCommonType` returns `Opion[DataType]`. `map` can iterate over the `DataType` if any, but we still need to enqueue the `None`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927164556
**[Test build #143624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143624/testReport)** for PR 34038 at commit [`87cf3e1`](https://github.com/apache/spark/commit/87cf3e118cf05356644ac74a762ffc63f4d9018d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925321066
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48022/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925553084
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143526/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926507141
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143589/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926399922
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48100/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926325590
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48089/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925348100
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927233299
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143628/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r712593090
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,16 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
// Check if the data types match.
- dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
- // SPARK-18058: we shall not care about the nullability of columns
- if (TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty) {
- failAnalysis(
- s"""
- |${operator.nodeName} can only be performed on tables with the compatible
- |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
- |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
- """.stripMargin.replace("\n", " ").trim())
+ if (!isUnion) {
Review comment:
I think these set operations work basically the same. The by-position resolution, I think, is to follow SQL. It only requires the columns to union have the same data types in same order, but not column names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922401123
**[Test build #143438 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143438/testReport)** for PR 34038 at commit [`47bec5d`](https://github.com/apache/spark/commit/47bec5d1449d2aafe27d8ed7febc7d0fbe0e9add).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922245150
**[Test build #143434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143434/testReport)** for PR 34038 at commit [`f9c133c`](https://github.com/apache/spark/commit/f9c133c57a27ae9789d6c4382c11f369717b15e7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r712557058
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,16 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
// Check if the data types match.
- dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
- // SPARK-18058: we shall not care about the nullability of columns
- if (TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty) {
- failAnalysis(
- s"""
- |${operator.nodeName} can only be performed on tables with the compatible
- |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
- |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
- """.stripMargin.replace("\n", " ").trim())
+ if (!isUnion) {
Review comment:
How are top-level columns handled for other set operations? In general I feel Spark SQL is built around column names by default, not positions, so I would expect it to be by-name. I was surprised to realize recently that `union` is by-position.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,16 +401,35 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
// Check if the data types match.
- dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
- // SPARK-18058: we shall not care about the nullability of columns
- if (TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty) {
- failAnalysis(
- s"""
- |${operator.nodeName} can only be performed on tables with the compatible
- |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
- |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
- """.stripMargin.replace("\n", " ").trim())
+ if (!isUnion) {
+ dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
+ // SPARK-18058: we shall not care about the nullability of columns
+ if (TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty) {
+ failAnalysis(
+ s"""
+ |${operator.nodeName} can only be performed on tables with the compatible
+ |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
+ |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
+ """.stripMargin.replace("\n", " ").trim())
+ }
+ }
+ } else {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
+ }
+ dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
+ if (!DataType.equalsStructurally(dt1, dt2, true, typeChecker)) {
+ failAnalysis(
+ s"""
+ |${operator.nodeName} can only be performed on tables with the compatible
+ |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
+ |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
+ """.stripMargin.replace("\n", " ").trim())
+ }
Review comment:
Maybe we can simplify like:
```scala
val dataTypesAreCompatibleFn = if (isUnion) {
// `TypeCoercion` takes care of type coercion already. If any columns or nested
// columns are not compatible, we detect it here and throw analysis exception.
val typeChecker = (dt1: DataType, dt2: DataType) => {
!TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
}
(dt1: DataType, dt2: DataType) => !DataType.equalsStructurally(dt1, dt2, true, typeChecker)
} else {
// SPARK-18058: we shall not care about the nullability of columns
(dt1: DataType, dt2: DataType) => TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
}
dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
if (dataTypesAreCompatibleFn(dt1, dt2)) {
failAnalysis(
s"""
|${operator.nodeName} can only be performed on tables with the compatible
|column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
|${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
""".stripMargin.replace("\n", " ").trim())
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927211237
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48139/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927177096
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48136/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925343895
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143512/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926252559
**[Test build #143576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143576/testReport)** for PR 34038 at commit [`a061497`](https://github.com/apache/spark/commit/a0614975564d0dca09077f7a8b50541dd5202e9c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925385113
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48029/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714001994
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,16 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
// Check if the data types match.
- dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
- // SPARK-18058: we shall not care about the nullability of columns
- if (TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty) {
- failAnalysis(
- s"""
- |${operator.nodeName} can only be performed on tables with the compatible
- |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
- |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
- """.stripMargin.replace("\n", " ").trim())
+ if (!isUnion) {
Review comment:
I think it's better to make top-level and nested columns consistent in other set operations as well, which is, do by-position resolution. We can't go with the other direction as that will be a breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925343895
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143512/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922245222
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143434/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922245222
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143434/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926349536
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143580/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925490991
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48034/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926505487
**[Test build #143589 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143589/testReport)** for PR 34038 at commit [`be31929`](https://github.com/apache/spark/commit/be31929417fb240c098eb12ff79cb3a8e364e973).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714527623
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
You mean to add cast for compatible ones, and ignore these cannot? let me try if it's possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927233299
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143628/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922365573
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143436/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925548459
**[Test build #143526 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143526/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714490029
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
Oh, I spent a little time to recall why I keep original check logic.
It is because if `TypeCoercion` fails to find compatible types for all columns, it won't add cast for all. It is all or nothing logic here.
So if we only check `dt1 == dt2` here, we compare the original data types even some of them are compatible.
`AnalysisErrorSuite` has one example. One relation has `short, string, double, decimal`, another one has `string, string, string, map`.
The first three columns are compatible, only the fourth isn't. So `TypeCoercion` fails to add casts for all.
If we compare `dt1 == dt2`, the error will be like "short is not compatible with string". But currently we get like "decimal is not compatible with map".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r715229676
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
Ok, it works. I changed `TypeCoercion` to follow the discussed behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926319931
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48089/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925454693
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143519/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926831758
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48119/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926795315
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48119/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925327429
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48022/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922356047
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47944/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922348986
**[Test build #143436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143436/testReport)** for PR 34038 at commit [`b91c08e`](https://github.com/apache/spark/commit/b91c08e6dda6d7f9b1e74d6b54146aa86b6b35a5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r712593090
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,16 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
// Check if the data types match.
- dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
- // SPARK-18058: we shall not care about the nullability of columns
- if (TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty) {
- failAnalysis(
- s"""
- |${operator.nodeName} can only be performed on tables with the compatible
- |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
- |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
- """.stripMargin.replace("\n", " ").trim())
+ if (!isUnion) {
Review comment:
I think these set operations work basically the same. The by-position resolution for union, I think, is to follow SQL. It only requires the columns to union have the same data types in same order, but not column names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714504782
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
I see, thanks for the explanation!
I think it's not a good practice to rely on the code logic that is far away from the current code. Is it possible to update the type coercion rule and add implicit casts using best efforts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922372717
**[Test build #143438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143438/testReport)** for PR 34038 at commit [`47bec5d`](https://github.com/apache/spark/commit/47bec5d1449d2aafe27d8ed7febc7d0fbe0e9add).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925327429
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48022/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925457360
**[Test build #143526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143526/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927177096
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48136/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r715477607
##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/union.sql.out
##########
@@ -684,8 +684,8 @@ SELECT cast('3.4' as decimal(38, 18)) UNION SELECT 'foo'
-- !query schema
struct<>
-- !query output
-org.apache.spark.SparkException
-Failed to merge incompatible data types decimal(38,18) and string
+org.apache.spark.sql.AnalysisException
+Union can only be performed on tables with the compatible column types. string <> decimal(38,18) at the first column of the second table
Review comment:
`string <> decimal(38,18) at the first column of the second table` is it valid English syntax?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927099090
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143620/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927164556
**[Test build #143624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143624/testReport)** for PR 34038 at commit [`87cf3e1`](https://github.com/apache/spark/commit/87cf3e118cf05356644ac74a762ffc63f4d9018d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927098741
**[Test build #143620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143620/testReport)** for PR 34038 at commit [`5e17567`](https://github.com/apache/spark/commit/5e17567e19e863021aa4ee5e2e9c11b576f3c940).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926325590
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48089/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926288170
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926831758
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48119/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714490029
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
Oh, I spent a little time to recall why I keep original check logic.
It is because if `TypeCoercion` fails to find compatible types for any column, it won't add cast for all. It is all or nothing logic here.
So if we only check `dt1 == dt2` here, we compare the original data types even some of them are compatible.
`AnalysisErrorSuite` has one example. One relation has `short, string, double, decimal`, another one has `string, string, string, map`.
The first three columns are compatible, only the fourth isn't. So `TypeCoercion` fails to add casts for all.
If we compare `dt1 == dt2`, the error will be like "short is not compatible with string". But currently we get like "decimal is not compatible with map".
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
Oh, I spent a little time to recall why I keep original check logic.
It is because if `TypeCoercion` fails to find compatible types for any column, it won't add cast for all. It is all or nothing logic there.
So if we only check `dt1 == dt2` here, we compare the original data types even some of them are compatible.
`AnalysisErrorSuite` has one example. One relation has `short, string, double, decimal`, another one has `string, string, string, map`.
The first three columns are compatible, only the fourth isn't. So `TypeCoercion` fails to add casts for all.
If we compare `dt1 == dt2`, the error will be like "short is not compatible with string". But currently we get like "decimal is not compatible with map".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925485874
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48034/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926399893
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48100/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r715745567
##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/union.sql.out
##########
@@ -684,8 +684,8 @@ SELECT cast('3.4' as decimal(38, 18)) UNION SELECT 'foo'
-- !query schema
struct<>
-- !query output
-org.apache.spark.SparkException
-Failed to merge incompatible data types decimal(38,18) and string
+org.apache.spark.sql.AnalysisException
+Union can only be performed on tables with the compatible column types. string <> decimal(38,18) at the first column of the second table
Review comment:
Ha, this comes from `CheckAnalysis`'s original error message. We can improve it, although there are some more tests relying on the error message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926769095
**[Test build #143607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143607/testReport)** for PR 34038 at commit [`80bb6e1`](https://github.com/apache/spark/commit/80bb6e135dddaa75aee1658e05681b992be91896).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922356065
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47944/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714203517
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,16 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
// Check if the data types match.
- dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
- // SPARK-18058: we shall not care about the nullability of columns
- if (TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty) {
- failAnalysis(
- s"""
- |${operator.nodeName} can only be performed on tables with the compatible
- |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
- |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
- """.stripMargin.replace("\n", " ").trim())
+ if (!isUnion) {
Review comment:
Ok. I think it makes more sense. I will make other set operations as by-position too at nested column level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714476343
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
I know it's from the old code. But is it necessary? The analyzer can add implicit casts to make the types the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925457360
**[Test build #143526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143526/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925553084
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143526/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926252559
**[Test build #143576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143576/testReport)** for PR 34038 at commit [`a061497`](https://github.com/apache/spark/commit/a0614975564d0dca09077f7a8b50541dd5202e9c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922379052
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47946/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922243518
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47942/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714000029
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##########
@@ -327,7 +327,7 @@ case class Union(
child.output.length == children.head.output.length &&
// compare the data types with the first child
child.output.zip(children.head.output).forall {
- case (l, r) => l.dataType.sameType(r.dataType)
+ case (l, r) => DataType.equalsStructurally(l.dataType, r.dataType, true)
Review comment:
Why do we skip data type equality check here? I think analyzer will guarantee it, by adding implicit cast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927211243
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48139/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927095678
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48132/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927169657
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48136/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927196616
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143624/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r711537729
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,16 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
// Check if the data types match.
- dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
- // SPARK-18058: we shall not care about the nullability of columns
- if (TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty) {
- failAnalysis(
- s"""
- |${operator.nodeName} can only be performed on tables with the compatible
- |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
- |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
- """.stripMargin.replace("\n", " ").trim())
+ if (!isUnion) {
Review comment:
Not sure if we should also generalize to all set operations? Although it looks reasonable, but by their API definition seems we don't have the by-position definition as Union.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714485017
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
For compatible types, spark will add implicit cast and the types should be equal. So I think a simple `dt1 == dt2` should be sufficient here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927196616
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143624/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926399922
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48100/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927202534
**[Test build #143628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143628/testReport)** for PR 34038 at commit [`f382cf2`](https://github.com/apache/spark/commit/f382cf27d1b9eb640129e08da3c2811af04cdc5f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926831732
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48119/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925413307
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48029/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925405036
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48029/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925413307
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48029/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925225642
**[Test build #143512 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143512/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925278282
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48022/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927176896
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48136/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926336277
**[Test build #143580 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143580/testReport)** for PR 34038 at commit [`a0af93c`](https://github.com/apache/spark/commit/a0af93cb5b8e319ef8dd87d1131fb91979e51e30).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926928227
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926285031
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48084/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927194299
**[Test build #143624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143624/testReport)** for PR 34038 at commit [`87cf3e1`](https://github.com/apache/spark/commit/87cf3e118cf05356644ac74a762ffc63f4d9018d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927090772
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48132/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922354692
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47944/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922372717
**[Test build #143438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143438/testReport)** for PR 34038 at commit [`47bec5d`](https://github.com/apache/spark/commit/47bec5d1449d2aafe27d8ed7febc7d0fbe0e9add).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714202853
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##########
@@ -327,7 +327,7 @@ case class Union(
child.output.length == children.head.output.length &&
// compare the data types with the first child
child.output.zip(children.head.output).forall {
- case (l, r) => l.dataType.sameType(r.dataType)
+ case (l, r) => DataType.equalsStructurally(l.dataType, r.dataType, true)
Review comment:
`DataType.equalsStructurally` still check data type equality, it just ignores field name difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922348986
**[Test build #143436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143436/testReport)** for PR 34038 at commit [`b91c08e`](https://github.com/apache/spark/commit/b91c08e6dda6d7f9b1e74d6b54146aa86b6b35a5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714490029
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
Oh, I spent a little time to recall why I keep original check logic.
It is because if `TypeCoercion` fails to find compatible types for any column, it won't add cast for all. It is all or nothing logic there.
So if we only check `dt1 == dt2` here, we compare the original data types even some of them are compatible.
`AnalysisErrorSuite` has one example. One relation has `short, string, double, decimal`, another one has `string, string, string, map`.
The first three columns are compatible, only the fourth isn't. So `TypeCoercion` doesn't add casts for all.
If we compare `dt1 == dt2`, the error will be like "short is not compatible with string". But currently we get like "decimal is not compatible with map".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925343050
**[Test build #143512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143512/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927211243
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48139/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-923368509
Thank you for pinging me, @viirya .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-923364319
cc @cloud-fan @dongjoon-hyun @xkrogen @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922379052
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47946/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926507141
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143589/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925454693
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143519/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925362566
**[Test build #143519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143519/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927095675
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48132/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927086631
**[Test build #143620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143620/testReport)** for PR 34038 at commit [`5e17567`](https://github.com/apache/spark/commit/5e17567e19e863021aa4ee5e2e9c11b576f3c940).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922365522
**[Test build #143436 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143436/testReport)** for PR 34038 at commit [`b91c08e`](https://github.com/apache/spark/commit/b91c08e6dda6d7f9b1e74d6b54146aa86b6b35a5).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927202534
**[Test build #143628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143628/testReport)** for PR 34038 at commit [`f382cf2`](https://github.com/apache/spark/commit/f382cf27d1b9eb640129e08da3c2811af04cdc5f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-927086631
**[Test build #143620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143620/testReport)** for PR 34038 at commit [`5e17567`](https://github.com/apache/spark/commit/5e17567e19e863021aa4ee5e2e9c11b576f3c940).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922242583
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47942/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922377859
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47946/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-922401714
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143438/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925468415
**[Test build #143520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143520/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926349944
**[Test build #143589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143589/testReport)** for PR 34038 at commit [`be31929`](https://github.com/apache/spark/commit/be31929417fb240c098eb12ff79cb3a8e364e973).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926288170
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926371144
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48100/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926266036
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48084/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-926349944
**[Test build #143589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143589/testReport)** for PR 34038 at commit [`be31929`](https://github.com/apache/spark/commit/be31929417fb240c098eb12ff79cb3a8e364e973).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714932061
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,15 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
+ val dataTypesAreCompatibleFn = if (isUnion) {
+ // `TypeCoercion` takes care of type coercion already. If any columns or nested
+ // columns are not compatible, we detect it here and throw analysis exception.
+ val typeChecker = (dt1: DataType, dt2: DataType) => {
+ !TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
Review comment:
> You mean to add cast for compatible ones, and ignore these cannot?
Yes, by doing that we can just do `dt1 == dt2` here, and makes the code less coupled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925456070
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925225642
**[Test build #143512 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143512/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925365774
**[Test build #143520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143520/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34038:
URL: https://github.com/apache/spark/pull/34038#discussion_r714204628
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -401,16 +401,30 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
|the ${ordinalNumber(ti + 1)} table has ${child.output.length} columns
""".stripMargin.replace("\n", " ").trim())
}
+ val isUnion = operator.isInstanceOf[Union]
// Check if the data types match.
- dataTypes(child).zip(ref).zipWithIndex.foreach { case ((dt1, dt2), ci) =>
- // SPARK-18058: we shall not care about the nullability of columns
- if (TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty) {
- failAnalysis(
- s"""
- |${operator.nodeName} can only be performed on tables with the compatible
- |column types. ${dt1.catalogString} <> ${dt2.catalogString} at the
- |${ordinalNumber(ci)} column of the ${ordinalNumber(ti + 1)} table
- """.stripMargin.replace("\n", " ").trim())
+ if (!isUnion) {
Review comment:
BTW I will make the change for other set operations in another PR (JIRA). It might require more change (doc, test..).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34038:
URL: https://github.com/apache/spark/pull/34038#issuecomment-925362566
**[Test build #143519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143519/testReport)** for PR 34038 at commit [`8db8b50`](https://github.com/apache/spark/commit/8db8b50e0621b46e6572ec72c1cc9aeca2a66807).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org